[jira] [Updated] (HIVE-14221) set SQLStdHiveAuthorizerFactoryForTest as default HIVE_AUTHORIZATION_MANAGER
[ https://issues.apache.org/jira/browse/HIVE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14221: --- Status: Open (was: Patch Available) > set SQLStdHiveAuthorizerFactoryForTest as default HIVE_AUTHORIZATION_MANAGER > > > Key: HIVE-14221 > URL: https://issues.apache.org/jira/browse/HIVE-14221 > Project: Hive > Issue Type: Sub-task > Components: Security >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-14221.01.patch, HIVE-14221.02.patch, > HIVE-14221.03.patch, HIVE-14221.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14221) set SQLStdHiveAuthorizerFactoryForTest as default HIVE_AUTHORIZATION_MANAGER
[ https://issues.apache.org/jira/browse/HIVE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14221: --- Attachment: HIVE-14221.04.patch > set SQLStdHiveAuthorizerFactoryForTest as default HIVE_AUTHORIZATION_MANAGER > > > Key: HIVE-14221 > URL: https://issues.apache.org/jira/browse/HIVE-14221 > Project: Hive > Issue Type: Sub-task > Components: Security >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-14221.01.patch, HIVE-14221.02.patch, > HIVE-14221.03.patch, HIVE-14221.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14221) set SQLStdHiveAuthorizerFactoryForTest as default HIVE_AUTHORIZATION_MANAGER
[ https://issues.apache.org/jira/browse/HIVE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14221: --- Status: Patch Available (was: Open) > set SQLStdHiveAuthorizerFactoryForTest as default HIVE_AUTHORIZATION_MANAGER > > > Key: HIVE-14221 > URL: https://issues.apache.org/jira/browse/HIVE-14221 > Project: Hive > Issue Type: Sub-task > Components: Security >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.1.0 > > Attachments: HIVE-14221.01.patch, HIVE-14221.02.patch, > HIVE-14221.03.patch, HIVE-14221.04.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[ https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381711#comment-15381711 ] Wei Zheng commented on HIVE-13934: -- Test failures not related. [~hagleitn] Can you take another look? > Configure Tez to make nocondiional task size memory available for the > Processor > --- > > Key: HIVE-13934 > URL: https://issues.apache.org/jira/browse/HIVE-13934 > Project: Hive > Issue Type: Bug >Reporter: Wei Zheng >Assignee: Wei Zheng > Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch, > HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch, > HIVE-13934.7.patch, HIVE-13934.8.patch, HIVE-13934.9.patch > > > Currently, noconditionaltasksize is not validated against the container size, > the reservations made in the container by Tez for Inputs / Outputs etc. > Check this at compile time to see if enough memory is available, or set up > the vertex to reserve additional memory for the Processor. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9478) Beeline CLI Creating and Selecting On Tables Without Passing in -n Parameter
[ https://issues.apache.org/jira/browse/HIVE-9478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381710#comment-15381710 ] Vihang Karajgaonkar commented on HIVE-9478: --- I am not able to reproduce this issue on the latest beeline version > Beeline CLI Creating and Selecting On Tables Without Passing in -n Parameter > > > Key: HIVE-9478 > URL: https://issues.apache.org/jira/browse/HIVE-9478 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 0.13.1 >Reporter: Mac Noland >Priority: Minor > > When I first got started with beeline, since I was already the cloud user on > my edge node (not using Kerbose), I was logging into beeline without passing > in a user via –n. Not sure if that was right or not, but seemed to let me > create tables. However, when doing this and trying to do a select, I would > get the error below. See Appendix #1 for me creating a table and then doing > a select to get the error. > I spent some time messing around with different things and stumbled upon > where if a passed in the –n parameter, the select command was completed > successfully. See Appendix #2 for that output. > > I’m not sure why I could create tables but not select on them when not > passing in –n? But then select on the table when passing in my username via > –n? I’m not using Kerberos authentication so my expert contact might have > suggested that setup might give me a unusual experience. > Anyway, I’m off and running using the –n parameter but wanted to share my > experience as I cut over to beeline. Thanks again in advance for everyone's > help and great work on Hive and Beeline. > Appendix 1 > cloud@c-10-206-76-8:~> beeline -u > jdbc:hive2://c-10-206-76-8.int.cis.trcloud:1/default --verbose=true > issuing: !connect jdbc:hive2://c-10-206-76-8.int.cis.trcloud:1/default '' > '' > scan complete in 4ms > Connecting to jdbc:hive2://c-10-206-76-8.int.cis.trcloud:1/default > Connected to: Apache Hive (version 0.13.1-cdh5.2.0) > Driver: Hive JDBC (version 0.13.1-cdh5.2.0) > Transaction isolation: TRANSACTION_REPEATABLE_READ > Beeline version 0.13.1-cdh5.2.0 by Apache Hive > 0: jdbc:hive2://c-10-206-76-8.int.cis.trcloud> create table test123 (test123 > int); > No rows affected (0.217 seconds) > 0: jdbc:hive2://c-10-206-76-8.int.cis.trcloud> select avg(test123) from > test123; > Error: Error while processing statement: FAILED: Execution Error, return code > 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask (state=08S01,code=1) > java.sql.SQLException: Error while processing statement: FAILED: Execution > Error, return code 1 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask > at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:277) > at org.apache.hive.beeline.Commands.execute(Commands.java:736) > at org.apache.hive.beeline.Commands.sql(Commands.java:657) > at org.apache.hive.beeline.BeeLine.dispatch(BeeLine.java:908) > at org.apache.hive.beeline.BeeLine.execute(BeeLine.java:770) > at org.apache.hive.beeline.BeeLine.begin(BeeLine.java:732) > at > org.apache.hive.beeline.BeeLine.mainWithInputRedirection(BeeLine.java:467) > at org.apache.hive.beeline.BeeLine.main(BeeLine.java:450) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > 0: jdbc:hive2://c-10-206-76-8.int.cis.trcloud> !quit > Appendix #2 > Closing: 0: jdbc:hive2://c-10-206-76-8.int.cis.trcloud:1/default > cloud@c-10-206-76-8:~> beeline -u > jdbc:hive2://c-10-206-76-8.int.cis.trcloud:1/default --verbose=true -n > cloud > issuing: !connect jdbc:hive2://c-10-206-76-8.int.cis.trcloud:1/default > cloud '' > scan complete in 3ms > Connecting to jdbc:hive2://c-10-206-76-8.int.cis.trcloud:1/default > Connected to: Apache Hive (version 0.13.1-cdh5.2.0) > Driver: Hive JDBC (version 0.13.1-cdh5.2.0) > Transaction isolation: TRANSACTION_REPEATABLE_READ > Beeline version 0.13.1-cdh5.2.0 by Apache Hive > 0: jdbc:hive2://c-10-206-76-8.int.cis.trcloud> select avg(test123) from > test123; > +---+--+ > | _c0 | > +---+--+ > | NULL | > +---+--+ > 1 row selected (34.084 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14167) Use work directories provided by Tez instead of directly using YARN local dirs
[ https://issues.apache.org/jira/browse/HIVE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381708#comment-15381708 ] Wei Zheng commented on HIVE-14167: -- Test failures unrelated. [~sseth] Can you take another look? > Use work directories provided by Tez instead of directly using YARN local dirs > -- > > Key: HIVE-14167 > URL: https://issues.apache.org/jira/browse/HIVE-14167 > Project: Hive > Issue Type: Improvement >Affects Versions: 2.1.0 >Reporter: Siddharth Seth >Assignee: Wei Zheng > Attachments: HIVE-14167.1.patch, HIVE-14167.2.patch, > HIVE-14167.3.patch > > > HIVE-13303 fixed things to use multiple directories instead of a single tmp > directory. However it's using yarn-local-dirs directly. > I'm not sure how well using the yarn-local-dir will work on a secure cluster. > Would be better to use Tez*Context.getWorkDirs. This provides an app specific > directory - writable by the user. > cc [~sershe] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[ https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381688#comment-15381688 ] Eugene Koifman commented on HIVE-13369: --- all test failures have age > 1 > AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing > the "best" base file > -- > > Key: HIVE-13369 > URL: https://issues.apache.org/jira/browse/HIVE-13369 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13369.1.patch, HIVE-13369.2.patch, > HIVE-13369.3.patch, HIVE-13369.4.patch, HIVE-13369.5.patch, HIVE-13369.6.patch > > > The JavaDoc on getAcidState() reads, in part: > "Note that because major compactions don't >preserve the history, we can't use a base directory that includes a >transaction id that we must exclude." > which is correct but there is nothing in the code that does this. > And if we detect a situation where txn X must be excluded but and there are > deltas that contain X, we'll have to abort the txn. This can't (reasonably) > happen with auto commit mode, but with multi statement txns it's possible. > Suppose some long running txn starts and lock in snapshot at 17 (HWM). An > hour later it decides to access some partition for which all txns < 20 (for > example) have already been compacted (i.e. GC'd). > == > Here is a more concrete example. Let's say the file for table A are as > follows and created in the order listed. > delta_4_4 > delta_5_5 > delta_4_5 > base_5 > delta_16_16 > delta_17_17 > base_17 (for example user ran major compaction) > let's say getAcidState() is called with ValidTxnList(20:16), i.e. with HWM=20 > and ExceptionList=<16> > Assume that all txns <= 20 commit. > Reader can't use base_17 because it has result of txn16. So it should chose > base_5 "TxnBase bestBase" in _getChildState()_. > Then the reset of the logic in _getAcidState()_ should choose delta_16_16 and > delta_17_17 in _Directory_ object. This would represent acceptable snapshot > for such reader. > The issue is if at the same time the Cleaner process is running. It will see > everything with txnid<17 as obsolete. Then it will check lock manger state > and decide to delete (as there may not be any locks in LM for table A). The > order in which the files are deleted is undefined right now. It may delete > delta_16_16 and delta_17_17 first and right at this moment the read request > with ValidTxnList(20:16) arrives (such snapshot may have bee locked in by > some multi-stmt txn that started some time ago. It acquires locks after the > Cleaner checks LM state and calls getAcidState(). This request will choose > base_5 but it won't see delta_16_16 and delta_17_17 and thus return the > snapshot w/o modifications made by those txns. > [This is not possible currently since we only support autoCommit=true. The > reason is the a query (0) opens txn (if appropriate), (1) acquires locks, (2) > locks in the snapshot. The cleaner won't delete anything for a given > compaction (partition) if there are locks on it. Thus for duration of the > transaction, nothing will be deleted so it's safe to use base_5] > This is a subtle race condition but possible. > 1. So the safest thing to do to ensure correctness is to use the latest > base_x as the "best" and check against exceptions in ValidTxnList and throw > an exception if there is an exception <=x. > 2. A better option is to keep 2 exception lists: aborted and open and only > throw if there is an open txn <=x. Compaction throws away data from aborted > txns and thus there is no harm using base with aborted txns in its range. > 3. You could make each txn record the lowest open txn id at its start and > prevent the cleaner from cleaning anything delta with id range that includes > this open txn id for any txn that is still running. This has a drawback of > potentially delaying GC of old files for arbitrarily long periods. So this > should be a user config choice. The implementation is not trivial. > I would go with 1 now and do 2/3 together with multi-statement txn work. > Side note: if 2 deltas have overlapping ID range, then 1 must be a subset of > the other -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14263) Log message when HS2 query is waiting on compile lock
[ https://issues.apache.org/jira/browse/HIVE-14263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14263: - Component/s: HiveServer2 > Log message when HS2 query is waiting on compile lock > - > > Key: HIVE-14263 > URL: https://issues.apache.org/jira/browse/HIVE-14263 > Project: Hive > Issue Type: Bug > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Thejas M Nair > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-14263) Log message when HS2 query is waiting on compile lock
[ https://issues.apache.org/jira/browse/HIVE-14263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair reassigned HIVE-14263: Assignee: Thejas M Nair > Log message when HS2 query is waiting on compile lock > - > > Key: HIVE-14263 > URL: https://issues.apache.org/jira/browse/HIVE-14263 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14205) Hive doesn't support union type with AVRO file format
[
https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381670#comment-15381670
]
Yibing Shi commented on HIVE-14205:
---
[~ctang.ma], could you please helpl check whether you can apply the patch? I
can apply it on my laptop
> Hive doesn't support union type with AVRO file format
> -
>
> Key: HIVE-14205
> URL: https://issues.apache.org/jira/browse/HIVE-14205
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-14205.1.patch, HIVE-14205.2.patch,
> HIVE-14205.3.patch
>
>
> Reproduce steps:
> {noformat}
> hive> CREATE TABLE avro_union_test
> > PARTITIONED BY (p int)
> > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> > STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> > OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> > TBLPROPERTIES ('avro.schema.literal'='{
> >"type":"record",
> >"name":"nullUnionTest",
> >"fields":[
> > {
> > "name":"value",
> > "type":[
> > "null",
> > "int",
> > "long"
> > ],
> > "default":null
> > }
> >]
> > }');
> OK
> Time taken: 0.105 seconds
> hive> alter table avro_union_test add partition (p=1);
> OK
> Time taken: 0.093 seconds
> hive> select * from avro_union_test;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1187)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1220)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1200)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:581)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:482)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Another test case to show this problem is:
> {noformat}
> hive> create table avro_union_test2 (value uniontype) stored as
> avro;
> OK
> Time taken: 0.053 seconds
> hive> show create table avro_union_test2;
> OK
> CREATE TABLE `avro_union_test2`(
> `value` uniontype COMMENT '')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT
>
[jira] [Commented] (HIVE-14205) Hive doesn't support union type with AVRO file format
[
https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381661#comment-15381661
]
Hive QA commented on HIVE-14205:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818491/HIVE-14205.3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/561/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/561/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-561/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-561/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at b53794b HIVE-14236: CTAS with UNION ALL puts the wrong stats in
Tez (Pengcheng Xiong, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at b53794b HIVE-14236: CTAS with UNION ALL puts the wrong stats in
Tez (Pengcheng Xiong, reviewed by Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818491 - PreCommit-HIVE-MASTER-Build
> Hive doesn't support union type with AVRO file format
> -
>
> Key: HIVE-14205
> URL: https://issues.apache.org/jira/browse/HIVE-14205
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-14205.1.patch, HIVE-14205.2.patch,
> HIVE-14205.3.patch
>
>
> Reproduce steps:
> {noformat}
> hive> CREATE TABLE avro_union_test
> > PARTITIONED BY (p int)
> > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> > STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> > OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> > TBLPROPERTIES ('avro.schema.literal'='{
> >"type":"record",
> >"name":"nullUnionTest",
> >"fields":[
> > {
> > "name":"value",
> > "type":[
> > "null",
> > "int",
> > "long"
> > ],
> > "default":null
> > }
> >]
> > }');
> OK
> Time taken: 0.105 seconds
> hive> alter table avro_union_test add partition (p=1);
> OK
> Time taken: 0.093 seconds
> hive> select * from avro_union_test;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149)
> at
>
[jira] [Commented] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[
https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381659#comment-15381659
]
Hive QA commented on HIVE-13369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818480/HIVE-13369.6.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10333 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/560/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/560/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-560/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818480 - PreCommit-HIVE-MASTER-Build
> AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing
> the "best" base file
> --
>
> Key: HIVE-13369
> URL: https://issues.apache.org/jira/browse/HIVE-13369
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-13369.1.patch, HIVE-13369.2.patch,
> HIVE-13369.3.patch, HIVE-13369.4.patch, HIVE-13369.5.patch, HIVE-13369.6.patch
>
>
> The JavaDoc on getAcidState() reads, in part:
> "Note that because major compactions don't
>preserve the history, we can't use a base directory that includes a
>transaction id that we must exclude."
> which is correct but there is nothing in the code that does this.
> And if we detect a situation where txn X must be excluded but and there are
> deltas that contain X, we'll have to abort the txn. This can't (reasonably)
> happen with auto commit mode, but with multi statement txns it's possible.
> Suppose some long running txn starts and lock in snapshot at 17 (HWM). An
> hour later it decides to access some partition for which all txns < 20 (for
> example) have already been compacted (i.e. GC'd).
> ==
> Here is a more concrete example. Let's say the file for table A are as
> follows and created in the order listed.
> delta_4_4
> delta_5_5
> delta_4_5
> base_5
> delta_16_16
> delta_17_17
> base_17 (for example user ran major compaction)
> let's say getAcidState() is called with ValidTxnList(20:16), i.e. with HWM=20
> and ExceptionList=<16>
> Assume that all txns <= 20 commit.
> Reader can't use base_17 because it has result of txn16. So it should chose
> base_5 "TxnBase bestBase" in _getChildState()_.
> Then the reset of the logic in _getAcidState()_ should choose delta_16_16 and
> delta_17_17 in _Directory_ object. This would represent acceptable snapshot
> for such reader.
> The issue is if at the same time the Cleaner process is running. It will see
> everything with txnid<17 as obsolete. Then it will check lock manger state
> and decide to delete (as there may not be any locks in LM for table A). The
> order in which the files are deleted is undefined right now. It may delete
> delta_16_16 and delta_17_17 first and right at this moment the read request
> with ValidTxnList(20:16) arrives (such snapshot may have bee locked in by
> some multi-stmt txn that started some time ago. It acquires locks after the
> Cleaner checks LM state and calls getAcidState(). This request will choose
> base_5 but it won't see delta_16_16 and delta_17_17 and thus return the
> snapshot w/o modifications made by those txns.
> [This is not possible currently since we only support autoCommit=true. The
> reason is the a query (0) opens txn (if appropriate), (1) acquires locks, (2)
> locks in the snapshot. The cleaner won't delete anything
[jira] [Updated] (HIVE-14205) Hive doesn't support union type with AVRO file format
[
https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated HIVE-14205:
--
Attachment: HIVE-14205.3.patch
I created this patch with command:
{noformat}
git diff --no-prefix --binary HEAD~1 HEAD > ~/Downloads/HIVE-14205.3.patch
{noformat}
> Hive doesn't support union type with AVRO file format
> -
>
> Key: HIVE-14205
> URL: https://issues.apache.org/jira/browse/HIVE-14205
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-14205.1.patch, HIVE-14205.2.patch,
> HIVE-14205.3.patch
>
>
> Reproduce steps:
> {noformat}
> hive> CREATE TABLE avro_union_test
> > PARTITIONED BY (p int)
> > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> > STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> > OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> > TBLPROPERTIES ('avro.schema.literal'='{
> >"type":"record",
> >"name":"nullUnionTest",
> >"fields":[
> > {
> > "name":"value",
> > "type":[
> > "null",
> > "int",
> > "long"
> > ],
> > "default":null
> > }
> >]
> > }');
> OK
> Time taken: 0.105 seconds
> hive> alter table avro_union_test add partition (p=1);
> OK
> Time taken: 0.093 seconds
> hive> select * from avro_union_test;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1187)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1220)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1200)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:581)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:482)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Another test case to show this problem is:
> {noformat}
> hive> create table avro_union_test2 (value uniontype) stored as
> avro;
> OK
> Time taken: 0.053 seconds
> hive> show create table avro_union_test2;
> OK
> CREATE TABLE `avro_union_test2`(
> `value` uniontype COMMENT '')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS
[jira] [Updated] (HIVE-14205) Hive doesn't support union type with AVRO file format
[
https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated HIVE-14205:
--
Attachment: (was: HIVE-14205.3.patch)
> Hive doesn't support union type with AVRO file format
> -
>
> Key: HIVE-14205
> URL: https://issues.apache.org/jira/browse/HIVE-14205
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-14205.1.patch, HIVE-14205.2.patch
>
>
> Reproduce steps:
> {noformat}
> hive> CREATE TABLE avro_union_test
> > PARTITIONED BY (p int)
> > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> > STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> > OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> > TBLPROPERTIES ('avro.schema.literal'='{
> >"type":"record",
> >"name":"nullUnionTest",
> >"fields":[
> > {
> > "name":"value",
> > "type":[
> > "null",
> > "int",
> > "long"
> > ],
> > "default":null
> > }
> >]
> > }');
> OK
> Time taken: 0.105 seconds
> hive> alter table avro_union_test add partition (p=1);
> OK
> Time taken: 0.093 seconds
> hive> select * from avro_union_test;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1187)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1220)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1200)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:581)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:482)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Another test case to show this problem is:
> {noformat}
> hive> create table avro_union_test2 (value uniontype) stored as
> avro;
> OK
> Time taken: 0.053 seconds
> hive> show create table avro_union_test2;
> OK
> CREATE TABLE `avro_union_test2`(
> `value` uniontype COMMENT '')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
>
[jira] [Updated] (HIVE-14205) Hive doesn't support union type with AVRO file format
[
https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated HIVE-14205:
--
Attachment: HIVE-14205.3.patch
> Hive doesn't support union type with AVRO file format
> -
>
> Key: HIVE-14205
> URL: https://issues.apache.org/jira/browse/HIVE-14205
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-14205.1.patch, HIVE-14205.2.patch,
> HIVE-14205.3.patch
>
>
> Reproduce steps:
> {noformat}
> hive> CREATE TABLE avro_union_test
> > PARTITIONED BY (p int)
> > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> > STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> > OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> > TBLPROPERTIES ('avro.schema.literal'='{
> >"type":"record",
> >"name":"nullUnionTest",
> >"fields":[
> > {
> > "name":"value",
> > "type":[
> > "null",
> > "int",
> > "long"
> > ],
> > "default":null
> > }
> >]
> > }');
> OK
> Time taken: 0.105 seconds
> hive> alter table avro_union_test add partition (p=1);
> OK
> Time taken: 0.093 seconds
> hive> select * from avro_union_test;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1187)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1220)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1200)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:581)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:482)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Another test case to show this problem is:
> {noformat}
> hive> create table avro_union_test2 (value uniontype) stored as
> avro;
> OK
> Time taken: 0.053 seconds
> hive> show create table avro_union_test2;
> OK
> CREATE TABLE `avro_union_test2`(
> `value` uniontype COMMENT '')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
>
[jira] [Commented] (HIVE-14205) Hive doesn't support union type with AVRO file format
[
https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381651#comment-15381651
]
Yibing Shi commented on HIVE-14205:
---
[~ctang.ma], these 2 files are binary AVRO files. Looks like they are causing
trouble to git apply.
Let me recreate the patch file with the command described
[here|http://stackoverflow.com/questions/17152171/git-cannot-apply-binary-patch-without-full-index-line]
> Hive doesn't support union type with AVRO file format
> -
>
> Key: HIVE-14205
> URL: https://issues.apache.org/jira/browse/HIVE-14205
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-14205.1.patch, HIVE-14205.2.patch
>
>
> Reproduce steps:
> {noformat}
> hive> CREATE TABLE avro_union_test
> > PARTITIONED BY (p int)
> > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> > STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> > OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> > TBLPROPERTIES ('avro.schema.literal'='{
> >"type":"record",
> >"name":"nullUnionTest",
> >"fields":[
> > {
> > "name":"value",
> > "type":[
> > "null",
> > "int",
> > "long"
> > ],
> > "default":null
> > }
> >]
> > }');
> OK
> Time taken: 0.105 seconds
> hive> alter table avro_union_test add partition (p=1);
> OK
> Time taken: 0.093 seconds
> hive> select * from avro_union_test;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1187)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1220)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1200)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:581)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:482)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Another test case to show this problem is:
> {noformat}
> hive> create table avro_union_test2 (value uniontype) stored as
> avro;
> OK
> Time taken: 0.053 seconds
> hive> show create table avro_union_test2;
> OK
> CREATE TABLE `avro_union_test2`(
> `value`
[jira] [Updated] (HIVE-14260) show WriteEntity writetype in explain output
[ https://issues.apache.org/jira/browse/HIVE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14260: - Attachment: HIVE-14262.2.patch > show WriteEntity writetype in explain output > > > Key: HIVE-14260 > URL: https://issues.apache.org/jira/browse/HIVE-14260 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14260.1.patch, HIVE-14262.2.patch > > > It is useful to see the WriteEntity writeType in explain output, specially > for 'explain authorization'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14260) show WriteEntity writetype in explain output
[ https://issues.apache.org/jira/browse/HIVE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14260: - Attachment: (was: HIVE-14262.2.patch) > show WriteEntity writetype in explain output > > > Key: HIVE-14260 > URL: https://issues.apache.org/jira/browse/HIVE-14260 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14260.1.patch, HIVE-14262.2.patch > > > It is useful to see the WriteEntity writeType in explain output, specially > for 'explain authorization'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14260) show WriteEntity writetype in explain output
[ https://issues.apache.org/jira/browse/HIVE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14260: - Attachment: HIVE-14262.2.patch > show WriteEntity writetype in explain output > > > Key: HIVE-14260 > URL: https://issues.apache.org/jira/browse/HIVE-14260 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14260.1.patch, HIVE-14262.2.patch > > > It is useful to see the WriteEntity writeType in explain output, specially > for 'explain authorization'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14205) Hive doesn't support union type with AVRO file format
[
https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381641#comment-15381641
]
Chaoyu Tang commented on HIVE-14205:
[~Yibing] I am not able to git apply the union_non_nullable.avro and
union_nullable.avro data file, and got the error msg like "cannot apply binary
patch to 'data/files/union_non_nullable.avro' without full index line". Any
idea? Thanks
> Hive doesn't support union type with AVRO file format
> -
>
> Key: HIVE-14205
> URL: https://issues.apache.org/jira/browse/HIVE-14205
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-14205.1.patch, HIVE-14205.2.patch
>
>
> Reproduce steps:
> {noformat}
> hive> CREATE TABLE avro_union_test
> > PARTITIONED BY (p int)
> > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> > STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> > OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> > TBLPROPERTIES ('avro.schema.literal'='{
> >"type":"record",
> >"name":"nullUnionTest",
> >"fields":[
> > {
> > "name":"value",
> > "type":[
> > "null",
> > "int",
> > "long"
> > ],
> > "default":null
> > }
> >]
> > }');
> OK
> Time taken: 0.105 seconds
> hive> alter table avro_union_test add partition (p=1);
> OK
> Time taken: 0.093 seconds
> hive> select * from avro_union_test;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1187)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1220)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1200)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:581)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:482)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Another test case to show this problem is:
> {noformat}
> hive> create table avro_union_test2 (value uniontype) stored as
> avro;
> OK
> Time taken: 0.053 seconds
> hive> show create table avro_union_test2;
> OK
> CREATE TABLE `avro_union_test2`(
> `value` uniontype
[jira] [Commented] (HIVE-14262) Inherit writetype from partition WriteEntity for table WriteEntity
[
https://issues.apache.org/jira/browse/HIVE-14262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381612#comment-15381612
]
Hive QA commented on HIVE-14262:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818477/HIVE-14262.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 871 failed/errored test(s), 10334 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_add_part_exist
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter5
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_char1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_char2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_file_format
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_2_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table2_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_numbuckets_partitioned_table_h23
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_change_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_clusterby_sortby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_coltype
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_partition_format_loc
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_rename_partition_authorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_skewed_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_cascade
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_invalidate_column_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_location
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_not_sorted
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_partition_drop
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_serde
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_table_serde2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_varchar1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_varchar2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_view_as_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_alter_view_rename
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_analyze_table_null_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_filter
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_groupby2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_join_pkfk
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_limit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_select
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_annotate_stats_union
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_archive_excludeHadoop20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_1_sql_std
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_6
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_7
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_admin_almighty1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_admin_almighty2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_cli_nonsql
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_authorization_create_temp_table
[jira] [Commented] (HIVE-14261) Support set/unset partition parameters
[
https://issues.apache.org/jira/browse/HIVE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381576#comment-15381576
]
Hive QA commented on HIVE-14261:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818468/HIVE-14261.01.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10319 tests
executed
*Failed tests:*
{noformat}
TestMiniTezCliDriver-insert_values_non_partitioned.q-update_after_multiple_inserts.q-tez_union_dynamic_partition.q-and-12-more
- did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/558/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/558/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-558/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818468 - PreCommit-HIVE-MASTER-Build
> Support set/unset partition parameters
> --
>
> Key: HIVE-14261
> URL: https://issues.apache.org/jira/browse/HIVE-14261
> Project: Hive
> Issue Type: New Feature
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14261.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[ https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13369: -- Attachment: HIVE-13369.6.patch > AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing > the "best" base file > -- > > Key: HIVE-13369 > URL: https://issues.apache.org/jira/browse/HIVE-13369 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13369.1.patch, HIVE-13369.2.patch, > HIVE-13369.3.patch, HIVE-13369.4.patch, HIVE-13369.5.patch, HIVE-13369.6.patch > > > The JavaDoc on getAcidState() reads, in part: > "Note that because major compactions don't >preserve the history, we can't use a base directory that includes a >transaction id that we must exclude." > which is correct but there is nothing in the code that does this. > And if we detect a situation where txn X must be excluded but and there are > deltas that contain X, we'll have to abort the txn. This can't (reasonably) > happen with auto commit mode, but with multi statement txns it's possible. > Suppose some long running txn starts and lock in snapshot at 17 (HWM). An > hour later it decides to access some partition for which all txns < 20 (for > example) have already been compacted (i.e. GC'd). > == > Here is a more concrete example. Let's say the file for table A are as > follows and created in the order listed. > delta_4_4 > delta_5_5 > delta_4_5 > base_5 > delta_16_16 > delta_17_17 > base_17 (for example user ran major compaction) > let's say getAcidState() is called with ValidTxnList(20:16), i.e. with HWM=20 > and ExceptionList=<16> > Assume that all txns <= 20 commit. > Reader can't use base_17 because it has result of txn16. So it should chose > base_5 "TxnBase bestBase" in _getChildState()_. > Then the reset of the logic in _getAcidState()_ should choose delta_16_16 and > delta_17_17 in _Directory_ object. This would represent acceptable snapshot > for such reader. > The issue is if at the same time the Cleaner process is running. It will see > everything with txnid<17 as obsolete. Then it will check lock manger state > and decide to delete (as there may not be any locks in LM for table A). The > order in which the files are deleted is undefined right now. It may delete > delta_16_16 and delta_17_17 first and right at this moment the read request > with ValidTxnList(20:16) arrives (such snapshot may have bee locked in by > some multi-stmt txn that started some time ago. It acquires locks after the > Cleaner checks LM state and calls getAcidState(). This request will choose > base_5 but it won't see delta_16_16 and delta_17_17 and thus return the > snapshot w/o modifications made by those txns. > [This is not possible currently since we only support autoCommit=true. The > reason is the a query (0) opens txn (if appropriate), (1) acquires locks, (2) > locks in the snapshot. The cleaner won't delete anything for a given > compaction (partition) if there are locks on it. Thus for duration of the > transaction, nothing will be deleted so it's safe to use base_5] > This is a subtle race condition but possible. > 1. So the safest thing to do to ensure correctness is to use the latest > base_x as the "best" and check against exceptions in ValidTxnList and throw > an exception if there is an exception <=x. > 2. A better option is to keep 2 exception lists: aborted and open and only > throw if there is an open txn <=x. Compaction throws away data from aborted > txns and thus there is no harm using base with aborted txns in its range. > 3. You could make each txn record the lowest open txn id at its start and > prevent the cleaner from cleaning anything delta with id range that includes > this open txn id for any txn that is still running. This has a drawback of > potentially delaying GC of old files for arbitrarily long periods. So this > should be a user config choice. The implementation is not trivial. > I would go with 1 now and do 2/3 together with multi-statement txn work. > Side note: if 2 deltas have overlapping ID range, then 1 must be a subset of > the other -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[ https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13369: -- Status: Patch Available (was: Open) > AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing > the "best" base file > -- > > Key: HIVE-13369 > URL: https://issues.apache.org/jira/browse/HIVE-13369 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13369.1.patch, HIVE-13369.2.patch, > HIVE-13369.3.patch, HIVE-13369.4.patch, HIVE-13369.5.patch, HIVE-13369.6.patch > > > The JavaDoc on getAcidState() reads, in part: > "Note that because major compactions don't >preserve the history, we can't use a base directory that includes a >transaction id that we must exclude." > which is correct but there is nothing in the code that does this. > And if we detect a situation where txn X must be excluded but and there are > deltas that contain X, we'll have to abort the txn. This can't (reasonably) > happen with auto commit mode, but with multi statement txns it's possible. > Suppose some long running txn starts and lock in snapshot at 17 (HWM). An > hour later it decides to access some partition for which all txns < 20 (for > example) have already been compacted (i.e. GC'd). > == > Here is a more concrete example. Let's say the file for table A are as > follows and created in the order listed. > delta_4_4 > delta_5_5 > delta_4_5 > base_5 > delta_16_16 > delta_17_17 > base_17 (for example user ran major compaction) > let's say getAcidState() is called with ValidTxnList(20:16), i.e. with HWM=20 > and ExceptionList=<16> > Assume that all txns <= 20 commit. > Reader can't use base_17 because it has result of txn16. So it should chose > base_5 "TxnBase bestBase" in _getChildState()_. > Then the reset of the logic in _getAcidState()_ should choose delta_16_16 and > delta_17_17 in _Directory_ object. This would represent acceptable snapshot > for such reader. > The issue is if at the same time the Cleaner process is running. It will see > everything with txnid<17 as obsolete. Then it will check lock manger state > and decide to delete (as there may not be any locks in LM for table A). The > order in which the files are deleted is undefined right now. It may delete > delta_16_16 and delta_17_17 first and right at this moment the read request > with ValidTxnList(20:16) arrives (such snapshot may have bee locked in by > some multi-stmt txn that started some time ago. It acquires locks after the > Cleaner checks LM state and calls getAcidState(). This request will choose > base_5 but it won't see delta_16_16 and delta_17_17 and thus return the > snapshot w/o modifications made by those txns. > [This is not possible currently since we only support autoCommit=true. The > reason is the a query (0) opens txn (if appropriate), (1) acquires locks, (2) > locks in the snapshot. The cleaner won't delete anything for a given > compaction (partition) if there are locks on it. Thus for duration of the > transaction, nothing will be deleted so it's safe to use base_5] > This is a subtle race condition but possible. > 1. So the safest thing to do to ensure correctness is to use the latest > base_x as the "best" and check against exceptions in ValidTxnList and throw > an exception if there is an exception <=x. > 2. A better option is to keep 2 exception lists: aborted and open and only > throw if there is an open txn <=x. Compaction throws away data from aborted > txns and thus there is no harm using base with aborted txns in its range. > 3. You could make each txn record the lowest open txn id at its start and > prevent the cleaner from cleaning anything delta with id range that includes > this open txn id for any txn that is still running. This has a drawback of > potentially delaying GC of old files for arbitrarily long periods. So this > should be a user config choice. The implementation is not trivial. > I would go with 1 now and do 2/3 together with multi-statement txn work. > Side note: if 2 deltas have overlapping ID range, then 1 must be a subset of > the other -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[ https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13369: -- Status: Open (was: Patch Available) > AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing > the "best" base file > -- > > Key: HIVE-13369 > URL: https://issues.apache.org/jira/browse/HIVE-13369 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13369.1.patch, HIVE-13369.2.patch, > HIVE-13369.3.patch, HIVE-13369.4.patch, HIVE-13369.5.patch > > > The JavaDoc on getAcidState() reads, in part: > "Note that because major compactions don't >preserve the history, we can't use a base directory that includes a >transaction id that we must exclude." > which is correct but there is nothing in the code that does this. > And if we detect a situation where txn X must be excluded but and there are > deltas that contain X, we'll have to abort the txn. This can't (reasonably) > happen with auto commit mode, but with multi statement txns it's possible. > Suppose some long running txn starts and lock in snapshot at 17 (HWM). An > hour later it decides to access some partition for which all txns < 20 (for > example) have already been compacted (i.e. GC'd). > == > Here is a more concrete example. Let's say the file for table A are as > follows and created in the order listed. > delta_4_4 > delta_5_5 > delta_4_5 > base_5 > delta_16_16 > delta_17_17 > base_17 (for example user ran major compaction) > let's say getAcidState() is called with ValidTxnList(20:16), i.e. with HWM=20 > and ExceptionList=<16> > Assume that all txns <= 20 commit. > Reader can't use base_17 because it has result of txn16. So it should chose > base_5 "TxnBase bestBase" in _getChildState()_. > Then the reset of the logic in _getAcidState()_ should choose delta_16_16 and > delta_17_17 in _Directory_ object. This would represent acceptable snapshot > for such reader. > The issue is if at the same time the Cleaner process is running. It will see > everything with txnid<17 as obsolete. Then it will check lock manger state > and decide to delete (as there may not be any locks in LM for table A). The > order in which the files are deleted is undefined right now. It may delete > delta_16_16 and delta_17_17 first and right at this moment the read request > with ValidTxnList(20:16) arrives (such snapshot may have bee locked in by > some multi-stmt txn that started some time ago. It acquires locks after the > Cleaner checks LM state and calls getAcidState(). This request will choose > base_5 but it won't see delta_16_16 and delta_17_17 and thus return the > snapshot w/o modifications made by those txns. > [This is not possible currently since we only support autoCommit=true. The > reason is the a query (0) opens txn (if appropriate), (1) acquires locks, (2) > locks in the snapshot. The cleaner won't delete anything for a given > compaction (partition) if there are locks on it. Thus for duration of the > transaction, nothing will be deleted so it's safe to use base_5] > This is a subtle race condition but possible. > 1. So the safest thing to do to ensure correctness is to use the latest > base_x as the "best" and check against exceptions in ValidTxnList and throw > an exception if there is an exception <=x. > 2. A better option is to keep 2 exception lists: aborted and open and only > throw if there is an open txn <=x. Compaction throws away data from aborted > txns and thus there is no harm using base with aborted txns in its range. > 3. You could make each txn record the lowest open txn id at its start and > prevent the cleaner from cleaning anything delta with id range that includes > this open txn id for any txn that is still running. This has a drawback of > potentially delaying GC of old files for arbitrarily long periods. So this > should be a user config choice. The implementation is not trivial. > I would go with 1 now and do 2/3 together with multi-statement txn work. > Side note: if 2 deltas have overlapping ID range, then 1 must be a subset of > the other -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[ https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-13369: -- Attachment: HIVE-13369.5.patch patch 5 fixes test failure caused by patch 4 > AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing > the "best" base file > -- > > Key: HIVE-13369 > URL: https://issues.apache.org/jira/browse/HIVE-13369 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 1.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Attachments: HIVE-13369.1.patch, HIVE-13369.2.patch, > HIVE-13369.3.patch, HIVE-13369.4.patch, HIVE-13369.5.patch > > > The JavaDoc on getAcidState() reads, in part: > "Note that because major compactions don't >preserve the history, we can't use a base directory that includes a >transaction id that we must exclude." > which is correct but there is nothing in the code that does this. > And if we detect a situation where txn X must be excluded but and there are > deltas that contain X, we'll have to abort the txn. This can't (reasonably) > happen with auto commit mode, but with multi statement txns it's possible. > Suppose some long running txn starts and lock in snapshot at 17 (HWM). An > hour later it decides to access some partition for which all txns < 20 (for > example) have already been compacted (i.e. GC'd). > == > Here is a more concrete example. Let's say the file for table A are as > follows and created in the order listed. > delta_4_4 > delta_5_5 > delta_4_5 > base_5 > delta_16_16 > delta_17_17 > base_17 (for example user ran major compaction) > let's say getAcidState() is called with ValidTxnList(20:16), i.e. with HWM=20 > and ExceptionList=<16> > Assume that all txns <= 20 commit. > Reader can't use base_17 because it has result of txn16. So it should chose > base_5 "TxnBase bestBase" in _getChildState()_. > Then the reset of the logic in _getAcidState()_ should choose delta_16_16 and > delta_17_17 in _Directory_ object. This would represent acceptable snapshot > for such reader. > The issue is if at the same time the Cleaner process is running. It will see > everything with txnid<17 as obsolete. Then it will check lock manger state > and decide to delete (as there may not be any locks in LM for table A). The > order in which the files are deleted is undefined right now. It may delete > delta_16_16 and delta_17_17 first and right at this moment the read request > with ValidTxnList(20:16) arrives (such snapshot may have bee locked in by > some multi-stmt txn that started some time ago. It acquires locks after the > Cleaner checks LM state and calls getAcidState(). This request will choose > base_5 but it won't see delta_16_16 and delta_17_17 and thus return the > snapshot w/o modifications made by those txns. > [This is not possible currently since we only support autoCommit=true. The > reason is the a query (0) opens txn (if appropriate), (1) acquires locks, (2) > locks in the snapshot. The cleaner won't delete anything for a given > compaction (partition) if there are locks on it. Thus for duration of the > transaction, nothing will be deleted so it's safe to use base_5] > This is a subtle race condition but possible. > 1. So the safest thing to do to ensure correctness is to use the latest > base_x as the "best" and check against exceptions in ValidTxnList and throw > an exception if there is an exception <=x. > 2. A better option is to keep 2 exception lists: aborted and open and only > throw if there is an open txn <=x. Compaction throws away data from aborted > txns and thus there is no harm using base with aborted txns in its range. > 3. You could make each txn record the lowest open txn id at its start and > prevent the cleaner from cleaning anything delta with id range that includes > this open txn id for any txn that is still running. This has a drawback of > potentially delaying GC of old files for arbitrarily long periods. So this > should be a user config choice. The implementation is not trivial. > I would go with 1 now and do 2/3 together with multi-statement txn work. > Side note: if 2 deltas have overlapping ID range, then 1 must be a subset of > the other -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14262) Inherit writetype from partition WriteEntity for table WriteEntity
[ https://issues.apache.org/jira/browse/HIVE-14262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14262: - Attachment: HIVE-14262.1.patch > Inherit writetype from partition WriteEntity for table WriteEntity > -- > > Key: HIVE-14262 > URL: https://issues.apache.org/jira/browse/HIVE-14262 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14262.1.patch > > > For partitioned table operations, a Table WriteEntity is being added to the > list to be authorized if there is a partition in the output list from > semantic analyzer. > However, it is being added with a default WriteType of DDL_NO_TASK. > The new Table WriteEntity should be created with the WriteType of the > partition WriteEntity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14262) Inherit writetype from partition WriteEntity for table WriteEntity
[ https://issues.apache.org/jira/browse/HIVE-14262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14262: - Attachment: (was: HIVE-14262.1.patch) > Inherit writetype from partition WriteEntity for table WriteEntity > -- > > Key: HIVE-14262 > URL: https://issues.apache.org/jira/browse/HIVE-14262 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > > For partitioned table operations, a Table WriteEntity is being added to the > list to be authorized if there is a partition in the output list from > semantic analyzer. > However, it is being added with a default WriteType of DDL_NO_TASK. > The new Table WriteEntity should be created with the WriteType of the > partition WriteEntity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14262) Inherit writetype from partition WriteEntity for table WriteEntity
[ https://issues.apache.org/jira/browse/HIVE-14262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14262: - Status: Patch Available (was: Open) > Inherit writetype from partition WriteEntity for table WriteEntity > -- > > Key: HIVE-14262 > URL: https://issues.apache.org/jira/browse/HIVE-14262 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14262.1.patch > > > For partitioned table operations, a Table WriteEntity is being added to the > list to be authorized if there is a partition in the output list from > semantic analyzer. > However, it is being added with a default WriteType of DDL_NO_TASK. > The new Table WriteEntity should be created with the WriteType of the > partition WriteEntity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14262) Inherit writetype from partition WriteEntity for table WriteEntity
[ https://issues.apache.org/jira/browse/HIVE-14262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14262: - Attachment: HIVE-14262.1.patch > Inherit writetype from partition WriteEntity for table WriteEntity > -- > > Key: HIVE-14262 > URL: https://issues.apache.org/jira/browse/HIVE-14262 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14262.1.patch > > > For partitioned table operations, a Table WriteEntity is being added to the > list to be authorized if there is a partition in the output list from > semantic analyzer. > However, it is being added with a default WriteType of DDL_NO_TASK. > The new Table WriteEntity should be created with the WriteType of the > partition WriteEntity. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[
https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381526#comment-15381526
]
Hive QA commented on HIVE-13974:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818467/HIVE-13974.095.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 139 failed/errored test(s), 10333 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_table_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_vectorization_project
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_autoColumnStats_4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_all_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_orig_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_no_match
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_delete_whole_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_dynpart_sort_optimization_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_acid_dynamic_partition
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_nonacid_from_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_orig_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_dynamic_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_orig_table_use_metadata
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_insert_values_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_llap_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_orc_ppd_exception
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acid_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_acidvec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_part
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_schema_evol_orc_vec_mapwork_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_transform_acid
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_after_multiple_inserts
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_after_multiple_inserts_special_characters
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_all_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_all_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_all_types
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_orig_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_tmp_table
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_two_cols
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_where_no_match
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_where_non_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_update_where_partitioned
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_acid3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_partition_diff_num_cols
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_dynamic
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver_encryption_insert_partition_static
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_join_part_col_char
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14117: Status: Open (was: Patch Available) > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14117) HS2 UI: List of recent queries shows most recent query last
[ https://issues.apache.org/jira/browse/HIVE-14117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14117: Status: Patch Available (was: Open) > HS2 UI: List of recent queries shows most recent query last > --- > > Key: HIVE-14117 > URL: https://issues.apache.org/jira/browse/HIVE-14117 > Project: Hive > Issue Type: Bug >Reporter: Gunther Hagleitner >Assignee: Gunther Hagleitner > Attachments: HIVE-14117.1.patch > > > It's more useful to see the latest one first in your "last n queries" view. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14155) Vectorization: Custom UDF Vectorization annotations are ignored
[
https://issues.apache.org/jira/browse/HIVE-14155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381523#comment-15381523
]
Ashutosh Chauhan commented on HIVE-14155:
-
+1
> Vectorization: Custom UDF Vectorization annotations are ignored
> ---
>
> Key: HIVE-14155
> URL: https://issues.apache.org/jira/browse/HIVE-14155
> Project: Hive
> Issue Type: Bug
> Components: UDF, Vectorization
>Affects Versions: 2.2.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-14155.1.patch, HIVE-14155.2.patch
>
>
> {code}
> @VectorizedExpressions(value = { VectorStringRot13.class })
> {code}
> in a custom UDF Is ignored because the check for annotations happens after
> custom UDF detection.
> The custom UDF codepath is on the fail-over track of annotation lookups, so
> the detection during validation of SEL is sufficient, instead of during
> expression creation.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14204) Optimize loading dynamic partitions
[
https://issues.apache.org/jira/browse/HIVE-14204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381521#comment-15381521
]
Ashutosh Chauhan commented on HIVE-14204:
-
Are test failures related?
> Optimize loading dynamic partitions
>
>
> Key: HIVE-14204
> URL: https://issues.apache.org/jira/browse/HIVE-14204
> Project: Hive
> Issue Type: Improvement
>Reporter: Rajesh Balamohan
>Assignee: Rajesh Balamohan
>Priority: Minor
> Attachments: HIVE-14204.1.patch
>
>
> Lots of time is spent in sequential fashion to load dynamic partitioned
> dataset in driver side. E.g simple dynamic partitioned load as follows takes
> 300+ seconds
> {noformat}
> INSERT INTO web_sales_test partition(ws_sold_date_sk) select * from
> tpcds_bin_partitioned_orc_200.web_sales;
> Time taken to load dynamic partitions: 309.22 seconds
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14257) CBO: Push Join through Groupby to trigger shuffle reductions
[
https://issues.apache.org/jira/browse/HIVE-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381512#comment-15381512
]
Gopal V commented on HIVE-14257:
Yes, that does help - the stats based PPD does rewrite the store_sales scan
into a "Predicate: false" & then null-scan detection reduces it to 1 split + 0
rows.
This is sort of a meaningless query. The Query59 d_month_seq case doesn't
filter to "false" (btw, it is one of the queries for which store_sales CTE
*should* not be merged between sub-queries, since they scan different
partitions without overlap).
{code}
with wss as
(select d_week_seq,
ss_store_sk,
sum(case when (d_day_name='Sunday') then ss_sales_price else null end)
sun_sales,
sum(case when (d_day_name='Monday') then ss_sales_price else null end)
mon_sales,
sum(case when (d_day_name='Tuesday') then ss_sales_price else null
end) tue_sales,
sum(case when (d_day_name='Wednesday') then ss_sales_price else null
end) wed_sales,
sum(case when (d_day_name='Thursday') then ss_sales_price else null
end) thu_sales,
sum(case when (d_day_name='Friday') then ss_sales_price else null end)
fri_sales,
sum(case when (d_day_name='Saturday') then ss_sales_price else null
end) sat_sales
from store_sales,date_dim
where d_date_sk = ss_sold_date_sk
group by d_week_seq,ss_store_sk
)
select s_store_name1,s_store_id1,d_week_seq1
,sun_sales1/sun_sales2,mon_sales1/mon_sales2
,tue_sales1/tue_sales1,wed_sales1/wed_sales2,thu_sales1/thu_sales2
,fri_sales1/fri_sales2,sat_sales1/sat_sales2
from
(select s_store_name s_store_name1,wss.d_week_seq d_week_seq1
,s_store_id s_store_id1,sun_sales sun_sales1
,mon_sales mon_sales1,tue_sales tue_sales1
,wed_sales wed_sales1,thu_sales thu_sales1
,fri_sales fri_sales1,sat_sales sat_sales1
from wss,store,date_dim d
where d.d_week_seq = wss.d_week_seq and
ss_store_sk = s_store_sk and
d_month_seq between 1185 and 1185 + 11) y,
(select s_store_name s_store_name2,wss.d_week_seq d_week_seq2
,s_store_id s_store_id2,sun_sales sun_sales2
,mon_sales mon_sales2,tue_sales tue_sales2
,wed_sales wed_sales2,thu_sales thu_sales2
,fri_sales fri_sales2,sat_sales sat_sales2
from wss,store,date_dim d
where d.d_week_seq = wss.d_week_seq and
ss_store_sk = s_store_sk and
d_month_seq between 1185+ 12 and 1185 + 23) x
where s_store_id1=s_store_id2
and d_week_seq1=d_week_seq2-52
order by s_store_name1,s_store_id1,d_week_seq1
limit 100;
{code}
> CBO: Push Join through Groupby to trigger shuffle reductions
>
>
> Key: HIVE-14257
> URL: https://issues.apache.org/jira/browse/HIVE-14257
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Gopal V
>
> Similar to the optimizations in hive, already which push aggregates through a
> join (hive.transpose.aggr.join=true).
> {code}
> select count(v) from (select d_year, count(ss_item_sk) as v from store_sales,
> date_dim where ss_sold_date_sk=d_Date_sk group by d_year) w, date_dim d where
> d.d_year = w.d_year and d_date_sk = 1;
> {code}
> currently produces an entire aggregate of all years before discarding all of
> that (because obviously, there's no data for d_date_sk=1;
> This particular example is a simplified version of TPC-DS Query59's join
> condition, which can have a reduction in scans by applying the d_month_seq
> between 1185 and 1185 + 11 into the wss alias.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[
https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381481#comment-15381481
]
Hive QA commented on HIVE-14214:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818466/HIVE-14214.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 10333 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_join
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_distinct_gby
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testExternalFooterCache
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testExternalFooterCachePpd
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/556/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/556/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-556/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818466 - PreCommit-HIVE-MASTER-Build
> ORC Schema Evolution and Predicate Push Down do not work together (no rows
> returned)
>
>
> Key: HIVE-14214
> URL: https://issues.apache.org/jira/browse/HIVE-14214
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14214.01.patch, HIVE-14214.02.patch,
> HIVE-14214.03.patch, HIVE-14214.WIP.patch
>
>
> In Schema Evolution, the reader schema is different than the file schema
> which is used to evaluate predicate push down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14236) CTAS with UNION ALL puts the wrong stats in Tez
[ https://issues.apache.org/jira/browse/HIVE-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14236: --- Resolution: Fixed Status: Resolved (was: Patch Available) > CTAS with UNION ALL puts the wrong stats in Tez > --- > > Key: HIVE-14236 > URL: https://issues.apache.org/jira/browse/HIVE-14236 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14236.01.patch, HIVE-14236.02.patch > > > to repo. in Tez, create table t as select * from src union all select * from > src; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14236) CTAS with UNION ALL puts the wrong stats in Tez
[ https://issues.apache.org/jira/browse/HIVE-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14236: --- Affects Version/s: 2.0.0 2.1.0 > CTAS with UNION ALL puts the wrong stats in Tez > --- > > Key: HIVE-14236 > URL: https://issues.apache.org/jira/browse/HIVE-14236 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14236.01.patch, HIVE-14236.02.patch > > > to repo. in Tez, create table t as select * from src union all select * from > src; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14236) CTAS with UNION ALL puts the wrong stats in Tez
[ https://issues.apache.org/jira/browse/HIVE-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14236: --- Fix Version/s: 2.1.1 2.2.0 > CTAS with UNION ALL puts the wrong stats in Tez > --- > > Key: HIVE-14236 > URL: https://issues.apache.org/jira/browse/HIVE-14236 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14236.01.patch, HIVE-14236.02.patch > > > to repo. in Tez, create table t as select * from src union all select * from > src; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14236) CTAS with UNION ALL puts the wrong stats in Tez
[ https://issues.apache.org/jira/browse/HIVE-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381478#comment-15381478 ] Pengcheng Xiong commented on HIVE-14236: The test failures are unrelated and can not be repro. pushed to master and 2.1. Thanks [~ashutoshc] for the review. > CTAS with UNION ALL puts the wrong stats in Tez > --- > > Key: HIVE-14236 > URL: https://issues.apache.org/jira/browse/HIVE-14236 > Project: Hive > Issue Type: Bug >Affects Versions: 2.0.0, 2.1.0 >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Fix For: 2.2.0, 2.1.1 > > Attachments: HIVE-14236.01.patch, HIVE-14236.02.patch > > > to repo. in Tez, create table t as select * from src union all select * from > src; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381473#comment-15381473 ] Lefty Leverenz edited comment on HIVE-13269 at 7/17/16 7:23 PM: [~jcamachorodriguez] documented *hive.optimize.filter.stats.reduction* in the wiki, so I removed the TODOC2.1 label. * [Configuration Properties -- hive.optimize.filter.stats.reduction | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.optimize.filter.stats.reduction] Thanks, Jesús. was (Author: le...@hortonworks.com): [~jcamachorodriguez] documented *hive.optimize.filter.stats.reduction* in the wiki, so I removed the TODOC2.1 label. * [Configuration Properties -- hive.optimize.filter.stats.reduction | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.optimize.filter.stats.reduction] Thanks, Jesus. > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381473#comment-15381473 ] Lefty Leverenz commented on HIVE-13269: --- [~jcamachorodriguez] documented *hive.optimize.filter.stats.reduction* in the wiki, so I removed the TODOC2.1 label. * [Configuration Properties -- hive.optimize.filter.stats.reduction | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.optimize.filter.stats.reduction] Thanks, Jesus. > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13269) Simplify comparison expressions using column stats
[ https://issues.apache.org/jira/browse/HIVE-13269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-13269: -- Labels: (was: TODOC2.1) > Simplify comparison expressions using column stats > -- > > Key: HIVE-13269 > URL: https://issues.apache.org/jira/browse/HIVE-13269 > Project: Hive > Issue Type: Bug > Components: CBO >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez > Fix For: 2.1.0 > > Attachments: HIVE-13269.01.patch, HIVE-13269.02.patch, > HIVE-13269.03.patch, HIVE-13269.04.patch, HIVE-13269.patch, HIVE-13269.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14261) Support set/unset partition parameters
[ https://issues.apache.org/jira/browse/HIVE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381444#comment-15381444 ] Pengcheng Xiong commented on HIVE-14261: [~ashutoshc], coud u take a look? Thanks. > Support set/unset partition parameters > -- > > Key: HIVE-14261 > URL: https://issues.apache.org/jira/browse/HIVE-14261 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14261.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14261) Support set/unset partition parameters
[ https://issues.apache.org/jira/browse/HIVE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14261: --- Status: Patch Available (was: Open) > Support set/unset partition parameters > -- > > Key: HIVE-14261 > URL: https://issues.apache.org/jira/browse/HIVE-14261 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14261.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14261) Support set/unset partition parameters
[ https://issues.apache.org/jira/browse/HIVE-14261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-14261: --- Attachment: HIVE-14261.01.patch > Support set/unset partition parameters > -- > > Key: HIVE-14261 > URL: https://issues.apache.org/jira/browse/HIVE-14261 > Project: Hive > Issue Type: New Feature >Reporter: Pengcheng Xiong >Assignee: Pengcheng Xiong > Attachments: HIVE-14261.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Attachment: HIVE-13974.095.patch > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 1.3.0, 2.1.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch, > HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch, > HIVE-13974.09.patch, HIVE-13974.091.patch, HIVE-13974.092.patch, > HIVE-13974.093.patch, HIVE-13974.094.patch, HIVE-13974.095.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: In Progress (was: Patch Available) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 2.1.0, 1.3.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch, > HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch, > HIVE-13974.09.patch, HIVE-13974.091.patch, HIVE-13974.092.patch, > HIVE-13974.093.patch, HIVE-13974.094.patch, HIVE-13974.095.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13974) ORC Schema Evolution doesn't support add columns to non-last STRUCT columns
[ https://issues.apache.org/jira/browse/HIVE-13974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-13974: Status: Patch Available (was: In Progress) > ORC Schema Evolution doesn't support add columns to non-last STRUCT columns > --- > > Key: HIVE-13974 > URL: https://issues.apache.org/jira/browse/HIVE-13974 > Project: Hive > Issue Type: Bug > Components: Hive, ORC, Transactions >Affects Versions: 2.1.0, 1.3.0, 2.2.0 >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Blocker > Attachments: HIVE-13974.01.patch, HIVE-13974.02.patch, > HIVE-13974.03.patch, HIVE-13974.04.patch, HIVE-13974.05.WIP.patch, > HIVE-13974.06.patch, HIVE-13974.07.patch, HIVE-13974.08.patch, > HIVE-13974.09.patch, HIVE-13974.091.patch, HIVE-13974.092.patch, > HIVE-13974.093.patch, HIVE-13974.094.patch, HIVE-13974.095.patch > > > Currently, the included columns are based on the fileSchema and not the > readerSchema which doesn't work for adding columns to non-last STRUCT data > type columns. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13191:
---
Resolution: Fixed
Fix Version/s: 2.1.1
2.2.0
1.3.0
Status: Resolved (was: Patch Available)
Pushed to master, branch-2.1, branch-1. Thanks for the review [~ashutoshc]!
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Fix For: 1.3.0, 2.2.0, 2.1.1
>
> Attachments: HIVE-13191.01.patch, HIVE-13191.02.patch,
> HIVE-13191.03.patch, HIVE-13191.patch, tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[ https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14214: Status: Patch Available (was: In Progress) > ORC Schema Evolution and Predicate Push Down do not work together (no rows > returned) > > > Key: HIVE-14214 > URL: https://issues.apache.org/jira/browse/HIVE-14214 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14214.01.patch, HIVE-14214.02.patch, > HIVE-14214.03.patch, HIVE-14214.WIP.patch > > > In Schema Evolution, the reader schema is different than the file schema > which is used to evaluate predicate push down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13995) Hive generates inefficient metastore queries for TPCDS tables with 1800+ partitions leading to higher compile time
[
https://issues.apache.org/jira/browse/HIVE-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381424#comment-15381424
]
Ashutosh Chauhan commented on HIVE-13995:
-
[~ndembla] Any comments for above? Do you think if there is a better to write
this query?
> Hive generates inefficient metastore queries for TPCDS tables with 1800+
> partitions leading to higher compile time
> --
>
> Key: HIVE-13995
> URL: https://issues.apache.org/jira/browse/HIVE-13995
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 2.2.0
>Reporter: Nita Dembla
>Assignee: Hari Sankar Sivarama Subramaniyan
> Attachments: HIVE-13995.1.patch, HIVE-13995.2.patch
>
>
> TPCDS fact tables (store_sales, catalog_sales) have 1800+ partitions and when
> the query does not a filter on the partition column, metastore queries
> generated have a large IN clause listing all the partition names. Most RDBMS
> systems have issues optimizing large IN clause and even when a good index
> plan is chosen , comparing to 1800+ string values will not lead to best
> execution time.
> When all partitions are chosen, not specifying the partition list and having
> filters only on table and column name will generate the same result set as
> long as there are no concurrent modifications to partition list of the hive
> table (adding/dropping partitions).
> For eg: For TPCDS query18, the metastore query gathering partition column
> statistics runs in 0.5 secs in Mysql. Following is output from mysql log
> {noformat}
> -- Query_time: 0.482063 Lock_time: 0.003037 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales'
> and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" in
> ('cs_sold_date_sk=2450815','cs_sold_date_sk=2450816','cs_sold_date_sk=2450817','cs_sold_date_sk=2450818','cs_sold_date_sk=2450819','cs_sold_date_sk=2450820','cs_sold_date_sk=2450821','cs_sold_date_sk=2450822','cs_sold_date_sk=2450823','cs_sold_date_sk=2450824','cs_sold_date_sk=2450825','cs_sold_date_sk=2450826','cs_sold_date_sk=2450827','cs_sold_date_sk=2450828','cs_sold_date_sk=2450829','cs_sold_date_sk=2450830','cs_sold_date_sk=2450831','cs_sold_date_sk=2450832','cs_sold_date_sk=2450833','cs_sold_date_sk=2450834','cs_sold_date_sk=2450835','cs_sold_date_sk=2450836','cs_sold_date_sk=2450837','cs_sold_date_sk=2450838','cs_sold_date_sk=2450839','cs_sold_date_sk=2450840','cs_sold_date_sk=2450841','cs_sold_date_sk=2450842','cs_sold_date_sk=2450843','cs_sold_date_sk=2450844','cs_sold_date_sk=2450845','cs_sold_date_sk=2450846','cs_sold_date_sk=2450847','cs_sold_date_sk=2450848','cs_sold_date_sk=2450849','cs_sold_date_sk=2450850','cs_sold_date_sk=2450851','cs_sold_date_sk=2450852','cs_sold_date_sk=2450853','cs_sold_date_sk=2450854','cs_sold_date_sk=2450855','cs_sold_date_sk=2450856',...,'cs_sold_date_sk=2452654')
> group by "PARTITION_NAME";
> {noformat}
> Functionally equivalent query runs in 0.1 seconds
> {noformat}
> --Query_time: 0.121296 Lock_time: 0.000156 Rows_sent: 1836 Rows_examined:
> 18360
> select count("COLUMN_NAME") from "PART_COL_STATS"
> where "DB_NAME" = 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" =
> 'catalog_sales' and "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> group by "PARTITION_NAME";
> {noformat}
> If removing the partition list seems drastic, its also possible to simply
> list the range since hive gets a ordered list of partition names. This
> performs equally well as earlier query
> {noformat}
> # Query_time: 0.143874 Lock_time: 0.000154 Rows_sent: 1836 Rows_examined:
> 18360
> SET timestamp=1464014881;
> select count("COLUMN_NAME") from "PART_COL_STATS" where "DB_NAME" =
> 'tpcds_bin_partitioned_orc_3' and "TABLE_NAME" = 'catalog_sales' and
> "COLUMN_NAME" in
> ('cs_bill_customer_sk','cs_bill_cdemo_sk','cs_item_sk','cs_quantity','cs_list_price','cs_sales_price','cs_coupon_amt','cs_net_profit')
> and "PARTITION_NAME" >= 'cs_sold_date_sk=2450815' and "PARTITION_NAME" <=
> 'cs_sold_date_sk=2452654'
> group by "PARTITION_NAME";
> {noformat}
> Another thing to check is the IN clause of column names. Columns in
> projection list of hive query are mentioned here. Not sure if statistics of
> these columns are required for hive query optimization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[ https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14214: Status: In Progress (was: Patch Available) > ORC Schema Evolution and Predicate Push Down do not work together (no rows > returned) > > > Key: HIVE-14214 > URL: https://issues.apache.org/jira/browse/HIVE-14214 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14214.01.patch, HIVE-14214.02.patch, > HIVE-14214.WIP.patch > > > In Schema Evolution, the reader schema is different than the file schema > which is used to evaluate predicate push down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14257) CBO: Push Join through Groupby to trigger shuffle reductions
[
https://issues.apache.org/jira/browse/HIVE-14257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381417#comment-15381417
]
Jesus Camacho Rodriguez commented on HIVE-14257:
[~gopalv], does {{hive.optimize.filter.stats.reduction=true}} help?
> CBO: Push Join through Groupby to trigger shuffle reductions
>
>
> Key: HIVE-14257
> URL: https://issues.apache.org/jira/browse/HIVE-14257
> Project: Hive
> Issue Type: Improvement
> Components: CBO
>Reporter: Gopal V
>
> Similar to the optimizations in hive, already which push aggregates through a
> join (hive.transpose.aggr.join=true).
> {code}
> select count(v) from (select d_year, count(ss_item_sk) as v from store_sales,
> date_dim where ss_sold_date_sk=d_Date_sk group by d_year) w, date_dim d where
> d.d_year = w.d_year and d_date_sk = 1;
> {code}
> currently produces an entire aggregate of all years before discarding all of
> that (because obviously, there's no data for d_date_sk=1;
> This particular example is a simplified version of TPC-DS Query59's join
> condition, which can have a reduction in scans by applying the d_month_seq
> between 1185 and 1185 + 11 into the wss alias.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381415#comment-15381415
]
Ashutosh Chauhan commented on HIVE-13191:
-
To detect, if column in value of RS is also in key of RS we need isSame()
+1 for latest patch.
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13191.01.patch, HIVE-13191.02.patch,
> HIVE-13191.03.patch, HIVE-13191.patch, tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381406#comment-15381406
]
Hive QA commented on HIVE-13191:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818450/HIVE-13191.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10331 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/555/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/555/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-555/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818450 - PreCommit-HIVE-MASTER-Build
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13191.01.patch, HIVE-13191.02.patch,
> HIVE-13191.03.patch, HIVE-13191.patch, tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[
https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381358#comment-15381358
]
Hive QA commented on HIVE-14214:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818441/HIVE-14214.02.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/554/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/554/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-554/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-554/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9764966 HIVE-14226 : Invalid check on an ASTNode#toStringTree in
CalcitePlanner (Barna Zsombor Klara via Jesus Camacho Rodriguez)
+ git clean -f -d
Removing common/src/test/org/apache/hadoop/hive/common/TestFileUtils.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9764966 HIVE-14226 : Invalid check on an ASTNode#toStringTree in
CalcitePlanner (Barna Zsombor Klara via Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818441 - PreCommit-HIVE-MASTER-Build
> ORC Schema Evolution and Predicate Push Down do not work together (no rows
> returned)
>
>
> Key: HIVE-14214
> URL: https://issues.apache.org/jira/browse/HIVE-14214
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Matt McCline
>Assignee: Matt McCline
>Priority: Critical
> Attachments: HIVE-14214.01.patch, HIVE-14214.02.patch,
> HIVE-14214.WIP.patch
>
>
> In Schema Evolution, the reader schema is different than the file schema
> which is used to evaluate predicate push down.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[
https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381357#comment-15381357
]
Hive QA commented on HIVE-14259:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818355/HIVE-14259.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10340 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/553/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/553/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-553/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818355 - PreCommit-HIVE-MASTER-Build
> FileUtils.isSubDir may return incorrect result
> --
>
> Key: HIVE-14259
> URL: https://issues.apache.org/jira/browse/HIVE-14259
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14259.1.patch
>
>
> while I was working on HIVE-12244 i've looked around for utility
> methods...i've found this method; but it considers path: `/dir12` inside
> `/dir1`
> which is not true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14067) Rename pendingCount to activeCalls in HiveSessionImpl for easier understanding.
[
https://issues.apache.org/jira/browse/HIVE-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381313#comment-15381313
]
Hive QA commented on HIVE-14067:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818317/HIVE-14067.001.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10330 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/552/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/552/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-552/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818317 - PreCommit-HIVE-MASTER-Build
> Rename pendingCount to activeCalls in HiveSessionImpl for easier
> understanding.
>
>
> Key: HIVE-14067
> URL: https://issues.apache.org/jira/browse/HIVE-14067
> Project: Hive
> Issue Type: Improvement
> Components: HiveServer2
>Reporter: zhihai xu
>Assignee: zhihai xu
>Priority: Trivial
> Attachments: HIVE-14067.000.patch, HIVE-14067.000.patch,
> HIVE-14067.001.patch
>
>
> Rename pendingCount to activeCalls in HiveSessionImpl for easier
> understanding.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381307#comment-15381307 ] Rui Li commented on HIVE-14259: --- [~kgyrtkirk] - Yeah I think it's better to put the utility methods together, or even consolidate them a bit - the two isSubDir methods are essentially trying to achieve the same purpose right? Hive::isSubDir solves the issue here by appending '/' at the end of each path, so we can still use String::startsWith. Intuitively, I think this may save some comparisons for long paths. > FileUtils.isSubDir may return incorrect result > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[
https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381281#comment-15381281
]
Zoltan Haindrich commented on HIVE-14259:
-
[~xuefuz] i've fixed another intresting subdirectory related method during
HIVE-12244; that one i think matched '/backup/foo/bar' as a parent of '/foo'
https://github.com/apache/hive/commit/47b5b5cdb8702c9ddbca3911b6a03ca59982b3c1#diff-c39d962bbd4b5ca645cfb9275b899a68L593
[~lirui] the only "user" of this {{FileUtils::isSubDir}} method is {{Hive}},
it would be better to move this method there too? ...or...at first blink it
seems like the only uses of these methods are originating from
{{MoveTask}}...and these operations seem to be independent from the
{{Hive::hiveDB}} threadlocal...
> FileUtils.isSubDir may return incorrect result
> --
>
> Key: HIVE-14259
> URL: https://issues.apache.org/jira/browse/HIVE-14259
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14259.1.patch
>
>
> while I was working on HIVE-12244 i've looked around for utility
> methods...i've found this method; but it considers path: `/dir12` inside
> `/dir1`
> which is not true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14236) CTAS with UNION ALL puts the wrong stats in Tez
[
https://issues.apache.org/jira/browse/HIVE-14236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381269#comment-15381269
]
Hive QA commented on HIVE-14236:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818310/HIVE-14236.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 10333 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.org.apache.hadoop.hive.cli.TestMiniTezCliDriver
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_alter_merge_stats_orc
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_4
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_constprog_dpp
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_delete_where_partitioned
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_insert_update_delete
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mrr
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_nonmr_fetch_threshold
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_schema_evol_text_nonvec_mapwork_part_all_complex
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_complex_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_decimal_6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join0
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_outer_join1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_reduce1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_varchar_mapjoin1
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_limit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/551/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/551/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-551/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818310 - PreCommit-HIVE-MASTER-Build
> CTAS with UNION ALL puts the wrong stats in Tez
> ---
>
> Key: HIVE-14236
> URL: https://issues.apache.org/jira/browse/HIVE-14236
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Attachments: HIVE-14236.01.patch, HIVE-14236.02.patch
>
>
> to repo. in Tez, create table t as select * from src union all select * from
> src;
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13191:
---
Attachment: HIVE-13191.03.patch
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13191.01.patch, HIVE-13191.02.patch,
> HIVE-13191.03.patch, HIVE-13191.patch, tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13191:
---
Attachment: (was: HIVE-13191.03.patch)
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13191.01.patch, HIVE-13191.02.patch,
> HIVE-13191.03.patch, HIVE-13191.patch, tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13191:
---
Status: Open (was: Patch Available)
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13191.01.patch, HIVE-13191.02.patch,
> HIVE-13191.03.patch, HIVE-13191.patch, tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13191:
---
Status: Patch Available (was: In Progress)
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0, 2.0.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13191.01.patch, HIVE-13191.02.patch,
> HIVE-13191.03.patch, HIVE-13191.patch, tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on HIVE-13191 started by Jesus Camacho Rodriguez.
--
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13191.01.patch, HIVE-13191.02.patch,
> HIVE-13191.03.patch, HIVE-13191.patch, tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-13191:
---
Attachment: HIVE-13191.03.patch
> DummyTable map joins mix up columns between tables
> --
>
> Key: HIVE-13191
> URL: https://issues.apache.org/jira/browse/HIVE-13191
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Gopal V
>Assignee: Jesus Camacho Rodriguez
> Attachments: HIVE-13191.01.patch, HIVE-13191.02.patch,
> HIVE-13191.03.patch, HIVE-13191.patch, tez.q
>
>
> {code}
> SELECT
> a.key,
> a.a_one,
> b.b_one,
> a.a_zero,
> b.b_zero
> FROM
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 a_one,
> 0 a_zero
> ) a
> LEFT JOIN
> (
> SELECT
> 11 key,
> 0 confuse_you,
> 1 b_one,
> 0 b_zero
> ) b
> ON a.key = b.key
> ;
> 11 1 0 0 1
> {code}
> This should be 11, 1, 1, 0, 0 instead.
> Disabling map-joins & using shuffle-joins returns the right result.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[
https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381259#comment-15381259
]
Rui Li commented on HIVE-14259:
---
We also have a method {{Hive::isSubDir}}. Not sure if we can do anything to
reduce redundancy.
> FileUtils.isSubDir may return incorrect result
> --
>
> Key: HIVE-14259
> URL: https://issues.apache.org/jira/browse/HIVE-14259
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
>Priority: Minor
> Attachments: HIVE-14259.1.patch
>
>
> while I was working on HIVE-12244 i've looked around for utility
> methods...i've found this method; but it considers path: `/dir12` inside
> `/dir1`
> which is not true
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14258) Reduce task timed out because CommonJoinOperator.genUniqueJoinObject took too long to finish without reporting progress
[
https://issues.apache.org/jira/browse/HIVE-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381215#comment-15381215
]
Hive QA commented on HIVE-14258:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818308/HIVE-14258.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10330 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/550/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/550/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-550/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818308 - PreCommit-HIVE-MASTER-Build
> Reduce task timed out because CommonJoinOperator.genUniqueJoinObject took too
> long to finish without reporting progress
> ---
>
> Key: HIVE-14258
> URL: https://issues.apache.org/jira/browse/HIVE-14258
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HIVE-14258.patch
>
>
> Reduce task timed out because CommonJoinOperator.genUniqueJoinObject took too
> long to finish without reporting progress.
> This timeout happened when reducer.close() is called in ReduceTask.java.
> CommonJoinOperator.genUniqueJoinObject() called by reducer.close() will loop
> over every row in the AbstractRowContainer. This can take a long time if
> there are a large number or rows, and during this time, it does not report
> progress. If this runs for long enough more than "mapreduce.task.timeout",
> ApplicationMaster will kill the task for failing to report progress.
> we configured "mapreduce.task.timeout" as 10 minutes. I captured the stack
> trace in the 10 minutes before AM killed the reduce task at 2016-07-15
> 07:19:11.
> The following three stack traces can prove it:
> at 2016-07-15 07:09:42:
> {code}
> "main" prio=10 tid=0x7f90ec017000 nid=0xd193 runnable [0x7f90f62e5000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:272)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:154)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x0007deecefb0> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:252)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:214)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:232)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007deecb978> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
> at
> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
>
[jira] [Commented] (HIVE-14148) Add branch-2.1 branch to pre-commit tests
[ https://issues.apache.org/jira/browse/HIVE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381172#comment-15381172 ] Lefty Leverenz commented on HIVE-14148: --- Removed the TODOC2.1 label because [~spena] updated the wiki. Thanks, Sergio! > Add branch-2.1 branch to pre-commit tests > - > > Key: HIVE-14148 > URL: https://issues.apache.org/jira/browse/HIVE-14148 > Project: Hive > Issue Type: Task > Components: Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > Fix For: 2.2.0 > > Attachments: HIVE-14148.1.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14148) Add branch-2.1 branch to pre-commit tests
[ https://issues.apache.org/jira/browse/HIVE-14148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-14148: -- Labels: (was: TODOC2.1) > Add branch-2.1 branch to pre-commit tests > - > > Key: HIVE-14148 > URL: https://issues.apache.org/jira/browse/HIVE-14148 > Project: Hive > Issue Type: Task > Components: Testing Infrastructure >Reporter: Sergio Peña >Assignee: Sergio Peña > Fix For: 2.2.0 > > Attachments: HIVE-14148.1.patch > > > NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[ https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381166#comment-15381166 ] Matt McCline commented on HIVE-14214: - (Tolerate no MapWork) > ORC Schema Evolution and Predicate Push Down do not work together (no rows > returned) > > > Key: HIVE-14214 > URL: https://issues.apache.org/jira/browse/HIVE-14214 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14214.01.patch, HIVE-14214.02.patch, > HIVE-14214.WIP.patch > > > In Schema Evolution, the reader schema is different than the file schema > which is used to evaluate predicate push down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[ https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14214: Attachment: HIVE-14214.02.patch > ORC Schema Evolution and Predicate Push Down do not work together (no rows > returned) > > > Key: HIVE-14214 > URL: https://issues.apache.org/jira/browse/HIVE-14214 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14214.01.patch, HIVE-14214.02.patch, > HIVE-14214.WIP.patch > > > In Schema Evolution, the reader schema is different than the file schema > which is used to evaluate predicate push down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[ https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14214: Attachment: (was: HIVE-14214.02.patch) > ORC Schema Evolution and Predicate Push Down do not work together (no rows > returned) > > > Key: HIVE-14214 > URL: https://issues.apache.org/jira/browse/HIVE-14214 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14214.01.patch, HIVE-14214.02.patch, > HIVE-14214.WIP.patch > > > In Schema Evolution, the reader schema is different than the file schema > which is used to evaluate predicate push down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14221) set SQLStdHiveAuthorizerFactoryForTest as default HIVE_AUTHORIZATION_MANAGER
[
https://issues.apache.org/jira/browse/HIVE-14221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381131#comment-15381131
]
Hive QA commented on HIVE-14221:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818288/HIVE-14221.03.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/549/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/549/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-549/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-549/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 9764966 HIVE-14226 : Invalid check on an ASTNode#toStringTree in
CalcitePlanner (Barna Zsombor Klara via Jesus Camacho Rodriguez)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 9764966 HIVE-14226 : Invalid check on an ASTNode#toStringTree in
CalcitePlanner (Barna Zsombor Klara via Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818288 - PreCommit-HIVE-MASTER-Build
> set SQLStdHiveAuthorizerFactoryForTest as default HIVE_AUTHORIZATION_MANAGER
>
>
> Key: HIVE-14221
> URL: https://issues.apache.org/jira/browse/HIVE-14221
> Project: Hive
> Issue Type: Sub-task
> Components: Security
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.1.0
>
> Attachments: HIVE-14221.01.patch, HIVE-14221.02.patch,
> HIVE-14221.03.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14254) Correct the hive version by changing "svn" to "git"
[
https://issues.apache.org/jira/browse/HIVE-14254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381127#comment-15381127
]
Hive QA commented on HIVE-14254:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818271/HIVE-14254.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10330 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/548/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/548/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-548/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818271 - PreCommit-HIVE-MASTER-Build
> Correct the hive version by changing "svn" to "git"
> ---
>
> Key: HIVE-14254
> URL: https://issues.apache.org/jira/browse/HIVE-14254
> Project: Hive
> Issue Type: Bug
> Components: CLI
>Affects Versions: 2.1.0
>Reporter: Tao Li
>Assignee: Tao Li
>Priority: Minor
> Attachments: HIVE-14254.1.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> When running "hive --version", "subversion" is displayed below, which should
> be "git".
> $ hive --version
> Hive 2.1.0-SNAPSHOT
> Subversion git://
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
