[jira] [Commented] (HIVE-9215) Some mapjoin queries broken with IdentityProjectRemover with PPD
[
https://issues.apache.org/jira/browse/HIVE-9215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259931#comment-14259931
]
Hive QA commented on HIVE-9215:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689288/HIVE-9215.2.patch.txt
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6722 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2206/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2206/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2206/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689288 - PreCommit-HIVE-TRUNK-Build
Some mapjoin queries broken with IdentityProjectRemover with PPD
Key: HIVE-9215
URL: https://issues.apache.org/jira/browse/HIVE-9215
Project: Hive
Issue Type: Bug
Components: Logical Optimizer
Affects Versions: 0.15.0
Reporter: Szehon Ho
Assignee: Navis
Attachments: HIVE-9215.1.patch.txt, HIVE-9215.2.patch.txt,
auto_join_ppd.q
IdentityProjectRemover (hive.optimize.remove.identity.project) with PPD will
sometimes make mapjoin query that returns the wrong result in MR case as well.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.
[
https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259973#comment-14259973
]
Hive QA commented on HIVE-5718:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689289/HIVE-5718.14.patch.txt
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 6722 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_explode
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udtf_explode
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2207/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2207/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2207/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689289 - PreCommit-HIVE-TRUNK-Build
Support direct fetch for lateral views, sub queries, etc.
-
Key: HIVE-5718
URL: https://issues.apache.org/jira/browse/HIVE-5718
Project: Hive
Issue Type: Improvement
Components: Query Processor
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: D13857.1.patch, D13857.2.patch, D13857.3.patch,
HIVE-5718.10.patch.txt, HIVE-5718.11.patch.txt, HIVE-5718.12.patch.txt,
HIVE-5718.13.patch.txt, HIVE-5718.14.patch.txt, HIVE-5718.4.patch.txt,
HIVE-5718.5.patch.txt, HIVE-5718.6.patch.txt, HIVE-5718.7.patch.txt,
HIVE-5718.8.patch.txt, HIVE-5718.9.patch.txt, HIVE-5718.diff-v11-v12.patch
Extend HIVE-2925 with LV and SubQ.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-6012) restore backward compatibility of arithmetic operations
[
https://issues.apache.org/jira/browse/HIVE-6012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259983#comment-14259983
]
Lefty Leverenz commented on HIVE-6012:
--
Doc done:
* [Configuration Properties -- hive.compat |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.compat]
* [Hive Operators and UDFs -- Arithmetic Operators -- A / B |
https://cwiki.apache.org/confluence/display/Hive/LanguageManual+UDF#LanguageManualUDF-ArithmeticOperators]
restore backward compatibility of arithmetic operations
---
Key: HIVE-6012
URL: https://issues.apache.org/jira/browse/HIVE-6012
Project: Hive
Issue Type: Bug
Components: Query Processor
Affects Versions: 0.13.0
Reporter: Thejas M Nair
Assignee: Jason Dere
Fix For: 0.13.0
Attachments: HIVE-6012.1.patch, HIVE-6012.2.patch, HIVE-6012.3.patch,
HIVE-6012.4.patch, HIVE-6012.5.patch, HIVE-6012.6.patch
HIVE-5356 changed the behavior of some of the arithmetic operations, and the
change is not backward compatible, as pointed out in this [jira
comment|https://issues.apache.org/jira/browse/HIVE-5356?focusedCommentId=13813398page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13813398]
{code}
int / int = decimal
float / float = double
float * float = double
float + float = double
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8956) Hive hangs while some error/exception happens beyond job execution [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260006#comment-14260006 ] chirag aggarwal commented on HIVE-8956: --- Does this take care of instances like: ERROR util.Utils: Uncaught exception in thread Result resolver thread-1 java.lang.OutOfMemoryError: Java heap space at java.nio.HeapByteBuffer.init(Unknown Source) at java.nio.ByteBuffer.allocate(Unknown Source) at org.apache.spark.storage.BlockMessage.set(BlockMessage.scala:94) at org.apache.spark.storage.BlockMessage$.fromByteBuffer(BlockMessage.scala:176) at org.apache.spark.storage.BlockMessageArray.set(BlockMessageArray.scala:63) at org.apache.spark.storage.BlockMessageArray$.fromBufferMessage(BlockMessageArray.scala:109) at org.apache.spark.storage.BlockManagerWorker$.syncGetBlock(BlockManagerWorker.scala:138) at org.apache.spark.storage.BlockManager$$anonfun$doGetRemote$2.apply(BlockManager.scala:530) at org.apache.spark.storage.BlockManager$$anonfun$doGetRemote$2.apply(BlockManager.scala:528) at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59) at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47) at org.apache.spark.storage.BlockManager.doGetRemote(BlockManager.scala:528) at org.apache.spark.storage.BlockManager.getRemoteBytes(BlockManager.scala:522) at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply$mcV$sp(TaskResultGetter.scala:53) at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:47) at org.apache.spark.scheduler.TaskResultGetter$$anon$2$$anonfun$run$1.apply(TaskResultGetter.scala:47) at org.apache.spark.util.Utils$.logUncaughtExceptions(Utils.scala:1311) at org.apache.spark.scheduler.TaskResultGetter$$anon$2.run(TaskResultGetter.scala:46) at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) at java.lang.Thread.run(Unknown Source) Hive hangs while some error/exception happens beyond job execution [Spark Branch] - Key: HIVE-8956 URL: https://issues.apache.org/jira/browse/HIVE-8956 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8956.1-spark.patch Remote spark client communicate with remote spark context asynchronously, if error/exception is throw out during job execution in remote spark context, it would be wrapped and send back to remote spark client, but if error/exception is throw out beyond job execution, such as job serialized failed, remote spark client would never know what's going on in remote spark context, and it would hangs there. Set a timeout in remote spark client side may not a great idea, as we are not sure how long the query executed in spark cluster. we need find a way to check whether job has failed(whole life cycle) in remote spark context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8956) Hive hangs while some error/exception happens beyond job execution [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260017#comment-14260017 ] chirag aggarwal commented on HIVE-8956: --- I guess no. Is there any ticket which tracks this issue? Hive hangs while some error/exception happens beyond job execution [Spark Branch] - Key: HIVE-8956 URL: https://issues.apache.org/jira/browse/HIVE-8956 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8956.1-spark.patch Remote spark client communicate with remote spark context asynchronously, if error/exception is throw out during job execution in remote spark context, it would be wrapped and send back to remote spark client, but if error/exception is throw out beyond job execution, such as job serialized failed, remote spark client would never know what's going on in remote spark context, and it would hangs there. Set a timeout in remote spark client side may not a great idea, as we are not sure how long the query executed in spark cluster. we need find a way to check whether job has failed(whole life cycle) in remote spark context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lefty Leverenz updated HIVE-8409: - Labels: (was: TODOC14) SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch, HIVE-8409.10.patch, HIVE-8409.11.patch, HIVE-8409.2.patch, HIVE-8409.3.patch, HIVE-8409.7.patch, HIVE-8409.8.patch, HIVE-8409.9.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8409) SMB joins fail intermittently on tez
[ https://issues.apache.org/jira/browse/HIVE-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260026#comment-14260026 ] Lefty Leverenz commented on HIVE-8409: -- Documented *hive.tez.smb.number.waves* in the wiki, so removed the TODOC14 label. * [Configuration Properties -- hive.tez.smb.number.waves | https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.tez.smb.number.waves] SMB joins fail intermittently on tez Key: HIVE-8409 URL: https://issues.apache.org/jira/browse/HIVE-8409 Project: Hive Issue Type: Bug Components: Tez Affects Versions: 0.14.0 Reporter: Vikram Dixit K Assignee: Vikram Dixit K Priority: Critical Fix For: 0.14.0 Attachments: HIVE-8409.1.patch, HIVE-8409.10.patch, HIVE-8409.11.patch, HIVE-8409.2.patch, HIVE-8409.3.patch, HIVE-8409.7.patch, HIVE-8409.8.patch, HIVE-8409.9.patch Flakiness with regard to SMB joins in tez. TEZ-1647 is required to complete the fix. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8410) Typo in DOAP - incorrect category URL
[
https://issues.apache.org/jira/browse/HIVE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260033#comment-14260033
]
Hive QA commented on HIVE-8410:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689297/HIVE-8410.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6722 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2208/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2208/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2208/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689297 - PreCommit-HIVE-TRUNK-Build
Typo in DOAP - incorrect category URL
-
Key: HIVE-8410
URL: https://issues.apache.org/jira/browse/HIVE-8410
Project: Hive
Issue Type: Bug
Environment: http://svn.apache.org/repos/asf/hive/trunk/doap_Hive.rdf
Reporter: Sebb
Assignee: Ferdinand Xu
Attachments: HIVE-8410.patch
The DOAP contains the following:
{code}
category rdf:resource=http://www.apache.org/category/database; /
{code}
However, the URL is incorrect; it must be
{code}
category rdf:resource=http://projects.apache.org/category/database; /
{code}
Please fix this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8410) Typo in DOAP - incorrect category URL
[
https://issues.apache.org/jira/browse/HIVE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260037#comment-14260037
]
Sebb commented on HIVE-8410:
The patch looks OK, but it has not yet been applied to the DOAP.
Please fix the DOAP.
Typo in DOAP - incorrect category URL
-
Key: HIVE-8410
URL: https://issues.apache.org/jira/browse/HIVE-8410
Project: Hive
Issue Type: Bug
Environment: http://svn.apache.org/repos/asf/hive/trunk/doap_Hive.rdf
Reporter: Sebb
Assignee: Ferdinand Xu
Attachments: HIVE-8410.patch
The DOAP contains the following:
{code}
category rdf:resource=http://www.apache.org/category/database; /
{code}
However, the URL is incorrect; it must be
{code}
category rdf:resource=http://projects.apache.org/category/database; /
{code}
Please fix this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8956) Hive hangs while some error/exception happens beyond job execution [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-8956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260139#comment-14260139 ] Brock Noland commented on HIVE-8956: [~chirag.aggarwal], it looks to me like some memory setting is too low but it's be better to create a sub-task of HIVE-7292 and discuss this error there. Would you mind doing this? Hive hangs while some error/exception happens beyond job execution [Spark Branch] - Key: HIVE-8956 URL: https://issues.apache.org/jira/browse/HIVE-8956 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Chengxiang Li Assignee: Rui Li Labels: Spark-M3 Fix For: spark-branch Attachments: HIVE-8956.1-spark.patch Remote spark client communicate with remote spark context asynchronously, if error/exception is throw out during job execution in remote spark context, it would be wrapped and send back to remote spark client, but if error/exception is throw out beyond job execution, such as job serialized failed, remote spark client would never know what's going on in remote spark context, and it would hangs there. Set a timeout in remote spark client side may not a great idea, as we are not sure how long the query executed in spark cluster. we need find a way to check whether job has failed(whole life cycle) in remote spark context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9215) Some mapjoin queries broken with IdentityProjectRemover with PPD
[ https://issues.apache.org/jira/browse/HIVE-9215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260143#comment-14260143 ] Xuefu Zhang commented on HIVE-9215: --- +1 Some mapjoin queries broken with IdentityProjectRemover with PPD Key: HIVE-9215 URL: https://issues.apache.org/jira/browse/HIVE-9215 Project: Hive Issue Type: Bug Components: Logical Optimizer Affects Versions: 0.15.0 Reporter: Szehon Ho Assignee: Navis Attachments: HIVE-9215.1.patch.txt, HIVE-9215.2.patch.txt, auto_join_ppd.q IdentityProjectRemover (hive.optimize.remove.identity.project) with PPD will sometimes make mapjoin query that returns the wrong result in MR case as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9218) Remove authorization_admin_almighty1 from spark tests [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9218:
--
Attachment: HIVE-9218.1-spark.patch
Remove authorization_admin_almighty1 from spark tests [Spark Branch]
Key: HIVE-9218
URL: https://issues.apache.org/jira/browse/HIVE-9218
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Attachments: HIVE-9218.1-spark.patch
The {{authorization_admin_almighty1}} test is authorization only so I don't
think we need to run it on spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-9218) Remove authorization_admin_almighty1 from spark tests [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang reassigned HIVE-9218:
-
Assignee: Xuefu Zhang
Remove authorization_admin_almighty1 from spark tests [Spark Branch]
Key: HIVE-9218
URL: https://issues.apache.org/jira/browse/HIVE-9218
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Xuefu Zhang
Attachments: HIVE-9218.1-spark.patch
The {{authorization_admin_almighty1}} test is authorization only so I don't
think we need to run it on spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9218) Remove authorization_admin_almighty1 from spark tests [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9218:
--
Status: Patch Available (was: Open)
Remove authorization_admin_almighty1 from spark tests [Spark Branch]
Key: HIVE-9218
URL: https://issues.apache.org/jira/browse/HIVE-9218
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Xuefu Zhang
Attachments: HIVE-9218.1-spark.patch
The {{authorization_admin_almighty1}} test is authorization only so I don't
think we need to run it on spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9213) Improve the mask pattern in QTestUtil to save partial directory info in test result
[ https://issues.apache.org/jira/browse/HIVE-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260146#comment-14260146 ] Brock Noland commented on HIVE-9213: +1 this is a great change!! :) Improve the mask pattern in QTestUtil to save partial directory info in test result --- Key: HIVE-9213 URL: https://issues.apache.org/jira/browse/HIVE-9213 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Fix For: encryption-branch Attachments: HIVE-9213.1.patch, HIVE-9213.patch NO PRECOMMIT TESTS The mask pattern in QTestUtil will mask directory in test result, since the directory varies in different test env. However, in Encryption test, the directory info is needed to verify the intermediate files are put in proper table. The whole directory is not necessary, and part of it is enough. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-9213) Improve the mask pattern in QTestUtil to save partial directory info in test result
[
https://issues.apache.org/jira/browse/HIVE-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9213:
---
Comment: was deleted
(was:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689015/HIVE-9213.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2192/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2192/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2192/
Messages:
{noformat}
This message was trimmed, see log for full details
[copy] Copying 8 files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-it ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-it ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-it/0.15.0-SNAPSHOT/hive-it-0.15.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Integration - Custom Serde 0.15.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-custom-serde
---
[INFO] Deleting
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-it-custom-serde ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-it-custom-serde ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-it-custom-serde ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @
hive-it-custom-serde ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
hive-it-custom-serde ---
[INFO] Compiling 10 source files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/classes
[WARNING]
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/main/java/org/apache/hadoop/hive/serde2/CustomSerDe2.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/main/java/org/apache/hadoop/hive/serde2/CustomSerDe2.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-it-custom-serde ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-custom-serde
---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp/conf
[copy] Copying 8 files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-it-custom-serde ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @
hive-it-custom-serde ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-it-custom-serde ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/hive-it-custom-serde-0.15.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-it-custom-serde ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @
hive-it-custom-serde ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/hive-it-custom-serde-0.15.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-it-custom-serde/0.15.0-SNAPSHOT/hive-it-custom-serde-0.15.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/pom.xml to
[jira] [Issue Comment Deleted] (HIVE-9213) Improve the mask pattern in QTestUtil to save partial directory info in test result
[
https://issues.apache.org/jira/browse/HIVE-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9213:
---
Comment: was deleted
(was:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689150/HIVE-9213.1.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2199/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2199/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2199/
Messages:
{noformat}
This message was trimmed, see log for full details
[copy] Copying 8 files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-it ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @ hive-it ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/pom.xml to
/data/hive-ptest/working/maven/org/apache/hive/hive-it/0.15.0-SNAPSHOT/hive-it-0.15.0-SNAPSHOT.pom
[INFO]
[INFO]
[INFO] Building Hive Integration - Custom Serde 0.15.0-SNAPSHOT
[INFO]
[INFO]
[INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hive-it-custom-serde
---
[INFO] Deleting
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde (includes
= [datanucleus.log, derby.log], excludes = [])
[INFO]
[INFO] --- maven-enforcer-plugin:1.3.1:enforce (enforce-no-snapshots) @
hive-it-custom-serde ---
[INFO]
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @
hive-it-custom-serde ---
[INFO]
[INFO] --- maven-resources-plugin:2.6:resources (default-resources) @
hive-it-custom-serde ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] Copying 1 resource
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (define-classpath) @
hive-it-custom-serde ---
[INFO] Executing tasks
main:
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @
hive-it-custom-serde ---
[INFO] Compiling 10 source files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/classes
[WARNING]
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/main/java/org/apache/hadoop/hive/serde2/CustomSerDe2.java:
Some input files use or override a deprecated API.
[WARNING]
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/main/java/org/apache/hadoop/hive/serde2/CustomSerDe2.java:
Recompile with -Xlint:deprecation for details.
[INFO]
[INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @
hive-it-custom-serde ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/src/test/resources
[INFO] Copying 3 resources
[INFO]
[INFO] --- maven-antrun-plugin:1.7:run (setup-test-dirs) @ hive-it-custom-serde
---
[INFO] Executing tasks
main:
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/warehouse
[mkdir] Created dir:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp/conf
[copy] Copying 8 files to
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/tmp/conf
[INFO] Executed tasks
[INFO]
[INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @
hive-it-custom-serde ---
[INFO] No sources to compile
[INFO]
[INFO] --- maven-surefire-plugin:2.16:test (default-test) @
hive-it-custom-serde ---
[INFO] Tests are skipped.
[INFO]
[INFO] --- maven-jar-plugin:2.2:jar (default-jar) @ hive-it-custom-serde ---
[INFO] Building jar:
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/hive-it-custom-serde-0.15.0-SNAPSHOT.jar
[INFO]
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @
hive-it-custom-serde ---
[INFO]
[INFO] --- maven-install-plugin:2.4:install (default-install) @
hive-it-custom-serde ---
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/target/hive-it-custom-serde-0.15.0-SNAPSHOT.jar
to
/data/hive-ptest/working/maven/org/apache/hive/hive-it-custom-serde/0.15.0-SNAPSHOT/hive-it-custom-serde-0.15.0-SNAPSHOT.jar
[INFO] Installing
/data/hive-ptest/working/apache-svn-trunk-source/itests/custom-serde/pom.xml
[jira] [Updated] (HIVE-9213) Improve the mask pattern in QTestUtil to save partial directory info in test result
[ https://issues.apache.org/jira/browse/HIVE-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9213: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thank you Dong! I have committed your contribution to branch! Improve the mask pattern in QTestUtil to save partial directory info in test result --- Key: HIVE-9213 URL: https://issues.apache.org/jira/browse/HIVE-9213 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Fix For: encryption-branch Attachments: HIVE-9213.1.patch, HIVE-9213.patch NO PRECOMMIT TESTS The mask pattern in QTestUtil will mask directory in test result, since the directory varies in different test env. However, in Encryption test, the directory info is needed to verify the intermediate files are put in proper table. The whole directory is not necessary, and part of it is enough. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9218) Remove authorization_admin_almighty1 from spark tests [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260148#comment-14260148
]
Brock Noland commented on HIVE-9218:
+1
Remove authorization_admin_almighty1 from spark tests [Spark Branch]
Key: HIVE-9218
URL: https://issues.apache.org/jira/browse/HIVE-9218
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Xuefu Zhang
Attachments: HIVE-9218.1-spark.patch
The {{authorization_admin_almighty1}} test is authorization only so I don't
think we need to run it on spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9220) HIVE-9109 missed updating result of list_bucket_dml_10
[ https://issues.apache.org/jira/browse/HIVE-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260151#comment-14260151 ] Xuefu Zhang commented on HIVE-9220: --- +1 HIVE-9109 missed updating result of list_bucket_dml_10 -- Key: HIVE-9220 URL: https://issues.apache.org/jira/browse/HIVE-9220 Project: Hive Issue Type: Sub-task Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-9109.1.patch.txt list_bucket_dml_10.q.java1.7.out is missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9206) Fix Desc Formatted related Java 8 ordering differences
[
https://issues.apache.org/jira/browse/HIVE-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9206:
---
Resolution: Fixed
Fix Version/s: 0.15.0
Status: Resolved (was: Patch Available)
Actually I noted that I committed this four days back. My mistake, not sure
what happened there.
Fix Desc Formatted related Java 8 ordering differences
--
Key: HIVE-9206
URL: https://issues.apache.org/jira/browse/HIVE-9206
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Fix For: 0.15.0
Attachments: HIVE-9206.1.patch, HIVE-9206.patch
This patch fixes the following tests for Java 8:
(1) list_bucket_dml_*.q
{{DESC FORMATTED}} calls {{StorageDescriptor.getSkewedInfo()}} HMS API, which
returns a thrift (unordered) map. Generate java version specific out file for
these tests.
(2) partitions_json.q
{{SHOW PARTITIONS}} uses {{MapBuilder}} via {{JsonMetaDataFormatter}} which
uses {{HashMap}}. Changed it to ordered map.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9157) Merge from trunk to spark 12/26/2014 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9157: --- Attachment: HIVE-9157.1-spark.patch.txt Merge from trunk to spark 12/26/2014 [Spark Branch] --- Key: HIVE-9157 URL: https://issues.apache.org/jira/browse/HIVE-9157 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Brock Noland Assignee: Brock Noland Fix For: spark-branch Attachments: HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9221) Remove deprecation warning for hive.metastore.local
Brock Noland created HIVE-9221: -- Summary: Remove deprecation warning for hive.metastore.local Key: HIVE-9221 URL: https://issues.apache.org/jira/browse/HIVE-9221 Project: Hive Issue Type: Bug Reporter: Brock Noland Assignee: Brock Noland Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9221) Remove deprecation warning for hive.metastore.local
[
https://issues.apache.org/jira/browse/HIVE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9221:
---
Description: The property {{hive.metastore.local}} has been removed for
years. We can remove the warning.
Remove deprecation warning for hive.metastore.local
---
Key: HIVE-9221
URL: https://issues.apache.org/jira/browse/HIVE-9221
Project: Hive
Issue Type: Bug
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
The property {{hive.metastore.local}} has been removed for years. We can
remove the warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9221) Remove deprecation warning for hive.metastore.local
[
https://issues.apache.org/jira/browse/HIVE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9221:
---
Affects Version/s: 0.15.0
Status: Patch Available (was: Open)
Remove deprecation warning for hive.metastore.local
---
Key: HIVE-9221
URL: https://issues.apache.org/jira/browse/HIVE-9221
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Attachments: HIVE-9221.patch
The property {{hive.metastore.local}} has been removed for years. We can
remove the warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9221) Remove deprecation warning for hive.metastore.local
[
https://issues.apache.org/jira/browse/HIVE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9221:
---
Attachment: HIVE-9221.patch
Remove deprecation warning for hive.metastore.local
---
Key: HIVE-9221
URL: https://issues.apache.org/jira/browse/HIVE-9221
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Attachments: HIVE-9221.patch
The property {{hive.metastore.local}} has been removed for years. We can
remove the warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Issue Comment Deleted] (HIVE-8821) Create unit test where we insert into dynamically partitioned table
[
https://issues.apache.org/jira/browse/HIVE-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-8821:
---
Comment: was deleted
(was:
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689149/HIVE-8821.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2198/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2198/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2198/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2198/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveInputFormat.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/CombineHiveRecordReader.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/io/HiveInputFormat.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target
shims/scheduler/target packaging/target hbase-handler/target testutils/target
jdbc/target metastore/target itests/target itests/hcatalog-unit/target
itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target
itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target
itests/util/target hcatalog/target hcatalog/core/target
hcatalog/streaming/target hcatalog/server-extensions/target
hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target
hcatalog/webhcat/java-client/target accumulo-handler/target hwi/target
common/target common/src/gen contrib/target service/target serde/target
beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1647934.
At revision 1647934.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689149 - PreCommit-HIVE-TRUNK-Build)
Create unit test where we insert into dynamically partitioned table
---
Key: HIVE-8821
URL: https://issues.apache.org/jira/browse/HIVE-8821
Project: Hive
Issue Type: Sub-task
Reporter: Brock Noland
Assignee: Dong Chen
Fix For: encryption-branch
Attachments: HIVE-8821.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8821) Create unit test where we insert into dynamically partitioned table
[ https://issues.apache.org/jira/browse/HIVE-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-8821: --- Description: NO PRECOMMIT TESTS Create unit test where we insert into dynamically partitioned table --- Key: HIVE-8821 URL: https://issues.apache.org/jira/browse/HIVE-8821 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Dong Chen Fix For: encryption-branch Attachments: HIVE-8821.patch NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8821) Create unit test where we insert into dynamically partitioned table
[ https://issues.apache.org/jira/browse/HIVE-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260215#comment-14260215 ] Brock Noland commented on HIVE-8821: LGTM but I guess this will need an update post HIVE-9213? Create unit test where we insert into dynamically partitioned table --- Key: HIVE-8821 URL: https://issues.apache.org/jira/browse/HIVE-8821 Project: Hive Issue Type: Sub-task Reporter: Brock Noland Assignee: Dong Chen Fix For: encryption-branch Attachments: HIVE-8821.patch NO PRECOMMIT TESTS -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9218) Remove authorization_admin_almighty1 from spark tests [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260223#comment-14260223
]
Hive QA commented on HIVE-9218:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689338/HIVE-9218.1-spark.patch
{color:red}ERROR:{color} -1 due to 20 failed/errored test(s), 7280 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_stats2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join_without_localtask
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_bucket_map_join_spark4
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_mixed
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel_join0
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parallel_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_6
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_auto_sortmerge_join_8
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_limit_pushdown
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_ppd_join
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
org.apache.hive.hcatalog.streaming.TestStreaming.testRemainingTransactions
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/595/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/595/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-595/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 20 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689338 - PreCommit-HIVE-SPARK-Build
Remove authorization_admin_almighty1 from spark tests [Spark Branch]
Key: HIVE-9218
URL: https://issues.apache.org/jira/browse/HIVE-9218
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Xuefu Zhang
Attachments: HIVE-9218.1-spark.patch
The {{authorization_admin_almighty1}} test is authorization only so I don't
think we need to run it on spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9218) Remove authorization_admin_almighty1 from spark tests [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260228#comment-14260228
]
Brock Noland commented on HIVE-9218:
Those tests should be fixed now due to my latest commits to HIVE-9157. Just
waiting on the results to confirm before closing it.
Remove authorization_admin_almighty1 from spark tests [Spark Branch]
Key: HIVE-9218
URL: https://issues.apache.org/jira/browse/HIVE-9218
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Xuefu Zhang
Attachments: HIVE-9218.1-spark.patch
The {{authorization_admin_almighty1}} test is authorization only so I don't
think we need to run it on spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9153) Perf enhancement on CombineHiveInputFormat and HiveInputFormat
[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9153:
--
Summary: Perf enhancement on CombineHiveInputFormat and HiveInputFormat
(was: Evaluate CombineHiveInputFormat versus HiveInputFormat)
Perf enhancement on CombineHiveInputFormat and HiveInputFormat
--
Key: HIVE-9153
URL: https://issues.apache.org/jira/browse/HIVE-9153
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Rui Li
Attachments: HIVE-9153.1-spark.patch, HIVE-9153.1-spark.patch,
HIVE-9153.2.patch, HIVE-9153.3.patch, screenshot.PNG
The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
should evaluate this on a query which has many input splits such as {{select
count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9153) Perf enhancement on CombineHiveInputFormat and HiveInputFormat
[
https://issues.apache.org/jira/browse/HIVE-9153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xuefu Zhang updated HIVE-9153:
--
Resolution: Fixed
Fix Version/s: 0.15.0
spark-branch
Status: Resolved (was: Patch Available)
Committed to trunk and merged to spark branch. Thanks, Rui.
Perf enhancement on CombineHiveInputFormat and HiveInputFormat
--
Key: HIVE-9153
URL: https://issues.apache.org/jira/browse/HIVE-9153
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Rui Li
Fix For: spark-branch, 0.15.0
Attachments: HIVE-9153.1-spark.patch, HIVE-9153.1-spark.patch,
HIVE-9153.2.patch, HIVE-9153.3.patch, screenshot.PNG
The default InputFormat is {{CombineHiveInputFormat}} and thus HOS uses this.
However, Tez uses {{HiveInputFormat}}. Since tasks are relatively cheap in
Spark, it might make sense for us to use {{HiveInputFormat}} as well. We
should evaluate this on a query which has many input splits such as {{select
count(\*) from store_sales where something is not null}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9133) CBO (Calcite Return Path): Refactor Semantic Analyzer to Move CBO code out
[ https://issues.apache.org/jira/browse/HIVE-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260281#comment-14260281 ] Laljo John Pullokkaran commented on HIVE-9133: -- Committed to trunk CBO (Calcite Return Path): Refactor Semantic Analyzer to Move CBO code out --- Key: HIVE-9133 URL: https://issues.apache.org/jira/browse/HIVE-9133 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.15.0 Attachments: HIVE-9133.1.patch, HIVE-9133.2.patch, HIVE-9133.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9133) CBO (Calcite Return Path): Refactor Semantic Analyzer to Move CBO code out
[ https://issues.apache.org/jira/browse/HIVE-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-9133: - Resolution: Fixed Status: Resolved (was: Patch Available) CBO (Calcite Return Path): Refactor Semantic Analyzer to Move CBO code out --- Key: HIVE-9133 URL: https://issues.apache.org/jira/browse/HIVE-9133 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.15.0 Attachments: HIVE-9133.1.patch, HIVE-9133.2.patch, HIVE-9133.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9157) Merge from trunk to spark 12/26/2014 [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260292#comment-14260292
]
Hive QA commented on HIVE-9157:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689348/HIVE-9157.1-spark.patch.txt
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 7281 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_covar_samp
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_authorization_admin_almighty1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_multi_insert_lateral_view
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/596/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/596/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-596/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689348 - PreCommit-HIVE-SPARK-Build
Merge from trunk to spark 12/26/2014 [Spark Branch]
---
Key: HIVE-9157
URL: https://issues.apache.org/jira/browse/HIVE-9157
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: spark-branch
Attachments: HIVE-9157.1-spark.patch.txt,
HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9140) Add ReduceExpressionRules from Calcite into Hive
[ https://issues.apache.org/jira/browse/HIVE-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260299#comment-14260299 ] Laljo John Pullokkaran commented on HIVE-9140: -- [~ashutoshc] patch needs to be rebased after HIVE-9133. Also could we resubmit the patch to get a QA run? Add ReduceExpressionRules from Calcite into Hive Key: HIVE-9140 URL: https://issues.apache.org/jira/browse/HIVE-9140 Project: Hive Issue Type: Improvement Components: CBO, Logical Optimizer Affects Versions: 0.14.0 Reporter: Ashutosh Chauhan Assignee: Ashutosh Chauhan Attachments: HIVE-9140.1.patch, HIVE-9140.2.patch, HIVE-9140.patch These rules provide a form of constant folding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8181) Upgrade JavaEWAH version to allow for unsorted bitset creation
[
https://issues.apache.org/jira/browse/HIVE-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-8181:
--
Affects Version/s: 0.14.0
0.13.1
Upgrade JavaEWAH version to allow for unsorted bitset creation
--
Key: HIVE-8181
URL: https://issues.apache.org/jira/browse/HIVE-8181
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.14.0, 0.13.1
Reporter: Gopal V
JavaEWAH has removed the restriction that bitsets can only be set in order in
the latest release.
Currently the use of {{ewah_bitmap}} UDAF requires a {{SORT BY}}.
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:249)
... 7 more
Caused by: java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-8181) Upgrade JavaEWAH version to allow for unsorted bitset creation
[
https://issues.apache.org/jira/browse/HIVE-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V reassigned HIVE-8181:
-
Assignee: Gopal V
Upgrade JavaEWAH version to allow for unsorted bitset creation
--
Key: HIVE-8181
URL: https://issues.apache.org/jira/browse/HIVE-8181
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.14.0, 0.13.1
Reporter: Gopal V
Assignee: Gopal V
JavaEWAH has removed the restriction that bitsets can only be set in order in
the latest release.
Currently the use of {{ewah_bitmap}} UDAF requires a {{SORT BY}}.
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:249)
... 7 more
Caused by: java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9215) Some mapjoin queries broken with IdentityProjectRemover with PPD
[ https://issues.apache.org/jira/browse/HIVE-9215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260307#comment-14260307 ] Szehon Ho commented on HIVE-9215: - +1 from me too. I also thought that, but am a bit new to the compiler code. Thanks Navis for looking and confirming my suspicion. Some mapjoin queries broken with IdentityProjectRemover with PPD Key: HIVE-9215 URL: https://issues.apache.org/jira/browse/HIVE-9215 Project: Hive Issue Type: Bug Components: Logical Optimizer Affects Versions: 0.15.0 Reporter: Szehon Ho Assignee: Navis Attachments: HIVE-9215.1.patch.txt, HIVE-9215.2.patch.txt, auto_join_ppd.q IdentityProjectRemover (hive.optimize.remove.identity.project) with PPD will sometimes make mapjoin query that returns the wrong result in MR case as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9221) Remove deprecation warning for hive.metastore.local
[
https://issues.apache.org/jira/browse/HIVE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260316#comment-14260316
]
Hive QA commented on HIVE-9221:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689351/HIVE-9221.patch
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 6722 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2209/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2209/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2209/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689351 - PreCommit-HIVE-TRUNK-Build
Remove deprecation warning for hive.metastore.local
---
Key: HIVE-9221
URL: https://issues.apache.org/jira/browse/HIVE-9221
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Attachments: HIVE-9221.patch
The property {{hive.metastore.local}} has been removed for years. We can
remove the warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8181) Upgrade JavaEWAH version to allow for unsorted bitset creation
[
https://issues.apache.org/jira/browse/HIVE-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-8181:
--
Attachment: HIVE-8181.1.patch
Upgrade JavaEWAH version to allow for unsorted bitset creation
--
Key: HIVE-8181
URL: https://issues.apache.org/jira/browse/HIVE-8181
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.14.0, 0.13.1
Reporter: Gopal V
Assignee: Gopal V
Attachments: HIVE-8181.1.patch
JavaEWAH has removed the restriction that bitsets can only be set in order in
the latest release.
Currently the use of {{ewah_bitmap}} UDAF requires a {{SORT BY}}.
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:249)
... 7 more
Caused by: java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8181) Upgrade JavaEWAH version to allow for unsorted bitset creation
[
https://issues.apache.org/jira/browse/HIVE-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-8181:
--
Release Note: Upgrade JavaEWAH to support unsorted bit-set creation
Status: Patch Available (was: Open)
Upgrade JavaEWAH version to allow for unsorted bitset creation
--
Key: HIVE-8181
URL: https://issues.apache.org/jira/browse/HIVE-8181
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.13.1, 0.14.0
Reporter: Gopal V
Assignee: Gopal V
Attachments: HIVE-8181.1.patch
JavaEWAH has removed the restriction that bitsets can only be set in order in
the latest release.
Currently the use of {{ewah_bitmap}} UDAF requires a {{SORT BY}}.
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:249)
... 7 more
Caused by: java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9201) Lazy functions do not handle newlines and carriage returns properly
[ https://issues.apache.org/jira/browse/HIVE-9201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260340#comment-14260340 ] Yongzhi Chen commented on HIVE-9201: The four failed tests are not related to the change: The first 2 tests are old failures aged 3 or more. I ran tests org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_remote_script org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority on my own machine, both succeeded. Lazy functions do not handle newlines and carriage returns properly --- Key: HIVE-9201 URL: https://issues.apache.org/jira/browse/HIVE-9201 Project: Hive Issue Type: Bug Affects Versions: 0.14.0, 0.13.1 Reporter: Yongzhi Chen Assignee: Yongzhi Chen Attachments: HIVE-9201.1.patch Hive returns wrong result when returning string has char \r or \n in it. This happens when the query can trigger mapreduce jobs. For example, for a table named strsim with only one row: As shown following, query 1 returns 1 row while query 2 returns 3 rows. Query 1: select abc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; Query 2: select a\rb\nc, narray from strsim LATERAL VIEW explode(array(1)) C AS narray; select abc, narray from strsim LATERAL VIEW e xplode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:00:08,958 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1178499218_0015 +--+-+--+ 1 row selected (1.283 seconds) | _c0 | narray | +--+-+--+ | abc | 1 | +--+-+--+ select a\rb\nc, narray from strsim LATERAL VI EW explode(array(1)) C AS narray; INFO : Number of reduce tasks is set to 0 since there's no reduce operator INFO : Job running in-process (local Hadoop) INFO : 2014-12-23 15:04:35,441 Stage-1 map = 0%, reduce = 0% INFO : Ended Job = job_local1816711099_0016 +--+-+--+ 3 rows selected (1.135 seconds) | _c0 | narray | +--+-+--+ | a| NULL| | b| NULL| | c| 1 | +--+-+--+ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9157) Merge from trunk to spark 12/26/2014 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9157: --- Attachment: HIVE-9157.1-spark.patch.txt Merge from trunk to spark 12/26/2014 [Spark Branch] --- Key: HIVE-9157 URL: https://issues.apache.org/jira/browse/HIVE-9157 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Brock Noland Assignee: Brock Noland Fix For: spark-branch Attachments: HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9039) Support Union Distinct
[ https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9039: -- Status: Open (was: Patch Available) Support Union Distinct -- Key: HIVE-9039 URL: https://issues.apache.org/jira/browse/HIVE-9039 Project: Hive Issue Type: New Feature Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, HIVE-9039.06.patch, HIVE-9039.07.patch Current version (Hive 0.14) does not support union (or union distinct). It only supports union all. In this patch, we try to add this new feature by rewriting union distinct to union all followed by group by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9039) Support Union Distinct
[ https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9039: -- Status: Patch Available (was: Open) Support Union Distinct -- Key: HIVE-9039 URL: https://issues.apache.org/jira/browse/HIVE-9039 Project: Hive Issue Type: New Feature Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch Current version (Hive 0.14) does not support union (or union distinct). It only supports union all. In this patch, we try to add this new feature by rewriting union distinct to union all followed by group by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9039) Support Union Distinct
[ https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9039: -- Attachment: HIVE-9039.08.patch rebase on HIVE-9133 Support Union Distinct -- Key: HIVE-9039 URL: https://issues.apache.org/jira/browse/HIVE-9039 Project: Hive Issue Type: New Feature Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch Current version (Hive 0.14) does not support union (or union distinct). It only supports union all. In this patch, we try to add this new feature by rewriting union distinct to union all followed by group by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-4776) Add option for skipping first n-rows of files
[ https://issues.apache.org/jira/browse/HIVE-4776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-4776: --- Fix Version/s: 0.13.0 Add option for skipping first n-rows of files - Key: HIVE-4776 URL: https://issues.apache.org/jira/browse/HIVE-4776 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Trivial Fix For: 0.13.0 Attachments: HIVE-4776.D11445.1.patch Some CSV file has header information which should be removed before used in hive. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-1955) Support non-constant expressions for array indexes.
[ https://issues.apache.org/jira/browse/HIVE-1955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-1955: --- Fix Version/s: 0.14.0 Support non-constant expressions for array indexes. --- Key: HIVE-1955 URL: https://issues.apache.org/jira/browse/HIVE-1955 Project: Hive Issue Type: Improvement Reporter: Adam Kramer Assignee: Navis Fix For: 0.14.0 FAILED: Error in semantic analysis: line 4:8 Non Constant Expressions for Array Indexes not Supported dut ...just wrote my own UDF to do this, and it is trivial. We should support this natively. Let foo have these rows: arr i [1,2,3] 1 [3,4,5] 2 [5,4,3] 2 [0,0,1] 0 Then, SELECT arr[i] FROM foo should return: 2 5 3 1 Similarly, for the same table, SELECT 3 IN arr FROM foo should return: true true true false ...these use cases are needless limitations of functionality. We shouldn't need UDFs to accomplish these goals. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9039) Support Union Distinct
[ https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9039: -- Status: Open (was: Patch Available) Support Union Distinct -- Key: HIVE-9039 URL: https://issues.apache.org/jira/browse/HIVE-9039 Project: Hive Issue Type: New Feature Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch Current version (Hive 0.14) does not support union (or union distinct). It only supports union all. In this patch, we try to add this new feature by rewriting union distinct to union all followed by group by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9039) Support Union Distinct
[ https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9039: -- Attachment: HIVE-9039.08.patch Support Union Distinct -- Key: HIVE-9039 URL: https://issues.apache.org/jira/browse/HIVE-9039 Project: Hive Issue Type: New Feature Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch Current version (Hive 0.14) does not support union (or union distinct). It only supports union all. In this patch, we try to add this new feature by rewriting union distinct to union all followed by group by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9039) Support Union Distinct
[ https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong updated HIVE-9039: -- Attachment: (was: HIVE-9039.08.patch) Support Union Distinct -- Key: HIVE-9039 URL: https://issues.apache.org/jira/browse/HIVE-9039 Project: Hive Issue Type: New Feature Reporter: Pengcheng Xiong Assignee: Pengcheng Xiong Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch, HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch, HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch Current version (Hive 0.14) does not support union (or union distinct). It only supports union all. In this patch, we try to add this new feature by rewriting union distinct to union all followed by group by. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9167) Enhance encryption testing framework to allow create keys zones inside .q files
[ https://issues.apache.org/jira/browse/HIVE-9167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260374#comment-14260374 ] Sergio Peña commented on HIVE-9167: --- Hi [~brocknoland]. I am still doing some changes on that RB patch. I'll let you know when it is ready. Thanks [~Ferd] for the comments. I working on those and other changes. Enhance encryption testing framework to allow create keys zones inside .q files - Key: HIVE-9167 URL: https://issues.apache.org/jira/browse/HIVE-9167 Project: Hive Issue Type: Sub-task Reporter: Sergio Peña Assignee: Sergio Peña The current implementation of the encryption testing framework on HIVE-8900 initializes a couple of encrypted databases to be used on .q test files. This is useful in order to make tests small, but it does not test all details found on the encryption implementation, such as: encrypted tables with different encryption strength in the same database. We need to allow this kind of encryption as it is how it will be used in the real world where a database will have a few encrypted tables (not all the DB). Also, we need to make this encryption framework flexible so that we can create/delete keys zones on demand when running the .q files. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8181) Upgrade JavaEWAH version to allow for unsorted bitset creation
[
https://issues.apache.org/jira/browse/HIVE-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260381#comment-14260381
]
Ashutosh Chauhan commented on HIVE-8181:
+1
Upgrade JavaEWAH version to allow for unsorted bitset creation
--
Key: HIVE-8181
URL: https://issues.apache.org/jira/browse/HIVE-8181
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.14.0, 0.13.1
Reporter: Gopal V
Assignee: Gopal V
Attachments: HIVE-8181.1.patch
JavaEWAH has removed the restriction that bitsets can only be set in order in
the latest release.
Currently the use of {{ewah_bitmap}} UDAF requires a {{SORT BY}}.
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:249)
... 7 more
Caused by: java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9221) Remove deprecation warning for hive.metastore.local
[
https://issues.apache.org/jira/browse/HIVE-9221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260382#comment-14260382
]
Ashutosh Chauhan commented on HIVE-9221:
+1
Remove deprecation warning for hive.metastore.local
---
Key: HIVE-9221
URL: https://issues.apache.org/jira/browse/HIVE-9221
Project: Hive
Issue Type: Bug
Affects Versions: 0.15.0
Reporter: Brock Noland
Assignee: Brock Noland
Priority: Minor
Attachments: HIVE-9221.patch
The property {{hive.metastore.local}} has been removed for years. We can
remove the warning.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 28797: Support Union Distinct
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28797/ --- (Updated Dec. 29, 2014, 8:05 p.m.) Review request for hive and John Pullokkaran. Changes --- rebase on HIVE-9133 Repository: hive-git Description --- Current version (Hive 0.14) does not support union (or union distinct). It only supports union all. In this patch, we try to add this new feature by rewriting union distinct to union all followed by group by. Diffs (updated) - itests/src/test/resources/testconfiguration.properties b0b5b2e ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java c02a65e ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java a985c4f ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 5d72e15 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 4e98e9b ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d9da15e ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 9ad6714 ql/src/test/queries/clientpositive/selectDistinctStar.q PRE-CREATION ql/src/test/queries/clientpositive/unionDistinct_1.q PRE-CREATION ql/src/test/queries/clientpositive/unionDistinct_2.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_6.q bfce26d ql/src/test/queries/clientpositive/union_remove_6_subq.q PRE-CREATION ql/src/test/results/clientpositive/complex_alias.q.out 133ce91 ql/src/test/results/clientpositive/explain_logical.q.out 41c8a2b ql/src/test/results/clientpositive/groupby_sort_1_23.q.out cb94faa ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out bbc0de4 ql/src/test/results/clientpositive/input_part7.q.out 6094f9c ql/src/test/results/clientpositive/join34.q.out e053e44 ql/src/test/results/clientpositive/join35.q.out 579920b ql/src/test/results/clientpositive/multi_join_union.q.out 466f34b ql/src/test/results/clientpositive/optimize_nullscan.q.out fb66f5e ql/src/test/results/clientpositive/ppd_union_view.q.out 1729464 ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out 3630332 ql/src/test/results/clientpositive/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out 0a4c26a ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out aaf7de3 ql/src/test/results/clientpositive/skewjoinopt9.q.out 502376e ql/src/test/results/clientpositive/tez/optimize_nullscan.q.out e598191 ql/src/test/results/clientpositive/tez/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/tez_smb_main.q.out b7c287d ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/unionDistinct_2.q.out PRE-CREATION ql/src/test/results/clientpositive/union22.q.out 672fa77 ql/src/test/results/clientpositive/union24.q.out 8755025 ql/src/test/results/clientpositive/union27.q.out f23f580 ql/src/test/results/clientpositive/union34.q.out 9ee382b ql/src/test/results/clientpositive/unionDistinct_1.q.out PRE-CREATION ql/src/test/results/clientpositive/unionDistinct_2.q.out PRE-CREATION ql/src/test/results/clientpositive/union_ppr.q.out 0610337 ql/src/test/results/clientpositive/union_remove_12.q.out f9fd323 ql/src/test/results/clientpositive/union_remove_13.q.out b8913e2 ql/src/test/results/clientpositive/union_remove_14.q.out 28e4bb6 ql/src/test/results/clientpositive/union_remove_6_subq.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28797/diff/ Testing --- Thanks, pengcheng xiong
Re: Review Request 28797: Support Union Distinct
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28797/ --- (Updated Dec. 29, 2014, 8:05 p.m.) Review request for hive and John Pullokkaran. Repository: hive-git Description --- Current version (Hive 0.14) does not support union (or union distinct). It only supports union all. In this patch, we try to add this new feature by rewriting union distinct to union all followed by group by. Diffs - itests/src/test/resources/testconfiguration.properties b0b5b2e ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/translator/ASTConverter.java c02a65e ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java a985c4f ql/src/java/org/apache/hadoop/hive/ql/parse/CalcitePlanner.java 5d72e15 ql/src/java/org/apache/hadoop/hive/ql/parse/HiveParser.g 4e98e9b ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java d9da15e ql/src/java/org/apache/hadoop/hive/ql/parse/SubQueryUtils.java 57868b7 ql/src/java/org/apache/hadoop/hive/ql/parse/UnparseTranslator.java 9ad6714 ql/src/test/queries/clientpositive/selectDistinctStar.q PRE-CREATION ql/src/test/queries/clientpositive/unionDistinct_1.q PRE-CREATION ql/src/test/queries/clientpositive/unionDistinct_2.q PRE-CREATION ql/src/test/queries/clientpositive/union_remove_6.q bfce26d ql/src/test/queries/clientpositive/union_remove_6_subq.q PRE-CREATION ql/src/test/results/clientpositive/complex_alias.q.out 133ce91 ql/src/test/results/clientpositive/explain_logical.q.out 41c8a2b ql/src/test/results/clientpositive/groupby_sort_1_23.q.out cb94faa ql/src/test/results/clientpositive/groupby_sort_skew_1_23.q.out bbc0de4 ql/src/test/results/clientpositive/input_part7.q.out 6094f9c ql/src/test/results/clientpositive/join34.q.out e053e44 ql/src/test/results/clientpositive/join35.q.out 579920b ql/src/test/results/clientpositive/multi_join_union.q.out 466f34b ql/src/test/results/clientpositive/optimize_nullscan.q.out fb66f5e ql/src/test/results/clientpositive/ppd_union_view.q.out 1729464 ql/src/test/results/clientpositive/ql_rewrite_gbtoidx.q.out 3630332 ql/src/test/results/clientpositive/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/skewjoin_mapjoin5.q.out 0a4c26a ql/src/test/results/clientpositive/skewjoin_mapjoin7.q.out aaf7de3 ql/src/test/results/clientpositive/skewjoinopt9.q.out 502376e ql/src/test/results/clientpositive/tez/optimize_nullscan.q.out e598191 ql/src/test/results/clientpositive/tez/selectDistinctStar.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/tez_smb_main.q.out b7c287d ql/src/test/results/clientpositive/tez/unionDistinct_1.q.out PRE-CREATION ql/src/test/results/clientpositive/tez/unionDistinct_2.q.out PRE-CREATION ql/src/test/results/clientpositive/union22.q.out 672fa77 ql/src/test/results/clientpositive/union24.q.out 8755025 ql/src/test/results/clientpositive/union27.q.out f23f580 ql/src/test/results/clientpositive/union34.q.out 9ee382b ql/src/test/results/clientpositive/unionDistinct_1.q.out PRE-CREATION ql/src/test/results/clientpositive/unionDistinct_2.q.out PRE-CREATION ql/src/test/results/clientpositive/union_ppr.q.out 0610337 ql/src/test/results/clientpositive/union_remove_12.q.out f9fd323 ql/src/test/results/clientpositive/union_remove_13.q.out b8913e2 ql/src/test/results/clientpositive/union_remove_14.q.out 28e4bb6 ql/src/test/results/clientpositive/union_remove_6_subq.q.out PRE-CREATION Diff: https://reviews.apache.org/r/28797/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Created] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
Mohit Sabharwal created HIVE-9222:
-
Summary: Fix ordering differences due to Java 8 (Part 4)
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver_unset_view_property and
TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
{{DDLSemanticAnalyzer.analyzeAlterTableProps()} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets {{((ASTNode)
tree.getChild(0)}} in different order between Java 7 and Java 8. The order is
different in {{HiveParser.statement()}} itself in {{ParseDriver.parse()}} so
this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver tests:
stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q, auto_join_without_localtask.q,
auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on these
tasks may get filtered. We see that different tasks are getting filtered out by
the condition resolver in {{ConditionTask}} in Java 8 compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8181) Upgrade JavaEWAH version to allow for unsorted bitset creation
[
https://issues.apache.org/jira/browse/HIVE-8181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260411#comment-14260411
]
Hive QA commented on HIVE-8181:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689360/HIVE-8181.1.patch
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 6722 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_bitmap_and
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_bitmap_empty
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udf_bitmap_or
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap3
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_index_bitmap_auto
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2210/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2210/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2210/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689360 - PreCommit-HIVE-TRUNK-Build
Upgrade JavaEWAH version to allow for unsorted bitset creation
--
Key: HIVE-8181
URL: https://issues.apache.org/jira/browse/HIVE-8181
Project: Hive
Issue Type: Bug
Components: UDF
Affects Versions: 0.14.0, 0.13.1
Reporter: Gopal V
Assignee: Gopal V
Attachments: HIVE-8181.1.patch
JavaEWAH has removed the restriction that bitsets can only be set in order in
the latest release.
Currently the use of {{ewah_bitmap}} UDAF requires a {{SORT BY}}.
{code}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
org.apache.hadoop.hive.ql.exec.GroupByOperator.processOp(GroupByOperator.java:824)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at
org.apache.hadoop.hive.ql.exec.ExtractOperator.processOp(ExtractOperator.java:45)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at
org.apache.hadoop.hive.ql.exec.ExecReducer.reduce(ExecReducer.java:249)
... 7 more
Caused by: java.lang.RuntimeException: Can't set bits out of order with
EWAHCompressedBitmap
at
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal reassigned HIVE-9222:
-
Assignee: Mohit Sabharwal
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver_unset_view_property and
TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
{{DDLSemanticAnalyzer.analyzeAlterTableProps()} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9222:
--
Description:
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets {{((ASTNode)
tree.getChild(0)}} in different order between Java 7 and Java 8. The order is
different in {{HiveParser.statement()}} itself in {{ParseDriver.parse()}} so
this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver tests:
stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q, auto_join_without_localtask.q,
auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on these
tasks may get filtered. We see that different tasks are getting filtered out by
the condition resolver in {{ConditionTask}} in Java 8 compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
was:
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver_unset_view_property and
TestNegativeCliDriver.testNegativeCliDriver_unset_table_property
{{DDLSemanticAnalyzer.analyzeAlterTableProps()} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets {{((ASTNode)
tree.getChild(0)}} in different order between Java 7 and Java 8. The order is
different in {{HiveParser.statement()}} itself in {{ParseDriver.parse()}} so
this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver tests:
stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q, auto_join_without_localtask.q,
auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on these
tasks may get filtered. We see that different tasks are getting filtered out by
the condition resolver in {{ConditionTask}} in Java 8 compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
Review Request 29465: HIVE-9222: Fix ordering differences due to Java 8 (Part 4)
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29465/ --- Review request for hive. Bugs: HIVE-9222 https://issues.apache.org/jira/browse/HIVE-9222 Repository: hive-git Description --- This patch fixes the following tests: (1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and unset_table_property.q DDLSemanticAnalyzer.analyzeAlterTableProps() gets table properties via getProps() which must be an insert order map. (2) TestCliDriver.testCliDriver_overridden_confs VerifyOverriddenConfigsHook emits overridden configs. Changed SessionState.overriddenConfigurations to insert order map. (3) TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST() gets ((ASTNode) tree.getChild(0) in different order between Java 7 and Java 8. The order is different in HiveParser.statement() itself in ParseDriver.parse() so this difference comes from antlr library. Generated java version specific output. (4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q Looks like these need rebase after HIVE-9206? Not sure what happened here... (5) TestCliDriver.testCliDriver: mapjoin_hook.q, auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q PrintCompletedTasksHook prints completed task list, which depends on the list of tasks added to runnable task list in DriverContext. Some on these tasks may get filtered. We see that different tasks are getting filtered out by the condition resolver in ConditionTask in Java 8 compared to Java 7. ConditionalTask.execute() calls ConditionalResolverCommonJoin.resolveDriverAlias() via getTasks(), which returns a single task based on task to alias map. The next mapred task in the task list gets filtered out by the resolver in ConditionalTask.resolveTask(). In other words, the the mapred task that shows up first will be kept and the next one will be filtered. Converted task to alias map to an insert order map so order is same with Java 8 and Java 7. Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java 3df1c2652f13dba8f1b0aa720f2649b738c576a1 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java c37b6fd57335f284587d0ca6ff5f8d479fafa0c5 ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java 3fcccb0fdf918627c6e8b046120442f1a3a1869b ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java 537708271cae1deeb2ac3de2dd876559528949d3 ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 1b5864e220fbaa1ac7d8f338844c3fae41d9ecec ql/src/test/org/apache/hadoop/hive/ql/plan/TestConditionalResolverCommonJoin.java 3af0257477e17b217e01c0285532cf2a3e961caa ql/src/test/queries/clientnegative/columnstats_partlvl_invalid_values.q 34f91fc8d1de846af0c0b6d1961139f08a34cf20 ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.7.out PRE-CREATION ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.8.out PRE-CREATION ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out 3261f78fab2341208c61d3cd5728f9a9d710ec80 ql/src/test/results/clientnegative/unset_table_property.q.out 8f94c9d28dc7cbf03688e56de21e9f0ba84afa73 ql/src/test/results/clientpositive/auto_join_without_localtask.q.out 8bb92e48a4e26eb72e80460cf9e13b5cbba8755c ql/src/test/results/clientpositive/list_bucket_dml_10.q.java1.7.out PRE-CREATION ql/src/test/results/clientpositive/list_bucket_dml_10.q.java1.8.out PRE-CREATION ql/src/test/results/clientpositive/list_bucket_dml_10.q.out 92c4d4455d3d6b6ef0fc4a5db686c3ec7597454b ql/src/test/results/clientpositive/list_bucket_dml_12.q.java1.8.out 68856e0fe4450e49726cd01448423f2edbceb47b ql/src/test/results/clientpositive/multiMapJoin2.q.out eb3bc9c9ff9464a52eb63adc5edaffe18c188abb ql/src/test/results/clientpositive/stats_list_bucket.q.java1.8.out 80a0f9c8683eb527c05854a38b1f6f03d48dd467 Diff: https://reviews.apache.org/r/29465/diff/ Testing --- Tested with Java 8 and Java 7. Thanks, Mohit Sabharwal
[jira] [Updated] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9222:
--
Attachment: HIVE-9222.patch
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9222.patch
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohit Sabharwal updated HIVE-9222:
--
Status: Patch Available (was: Open)
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9222.patch
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260428#comment-14260428
]
Brock Noland commented on HIVE-9222:
LGTM! I noted HIVE-9220 which seems to conflict with this patch. How should we
proceed?
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9222.patch
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260431#comment-14260431 ] Brock Noland commented on HIVE-8950: [~singhashish] the latest patch was only on RB. Can you upload here fore testing? Add support in ParquetHiveSerde to create table schema from a parquet file -- Key: HIVE-8950 URL: https://issues.apache.org/jira/browse/HIVE-8950 Project: Hive Issue Type: Improvement Reporter: Ashish Kumar Singh Assignee: Ashish Kumar Singh Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch, HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.patch PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without having to specify the column names and types. As, parquet files store schema in their footer, it is possible to generate hive schema from parquet file's metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9157) Merge from trunk to spark 12/26/2014 [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260434#comment-14260434
]
Hive QA commented on HIVE-9157:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689366/HIVE-9157.1-spark.patch.txt
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 7281 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample_islocalmode_hook
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_cast_constant
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_authorization_admin_almighty1
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver_windowing
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/597/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-SPARK-Build/597/console
Test logs:
http://ec2-50-18-27-0.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-SPARK-Build-597/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689366 - PreCommit-HIVE-SPARK-Build
Merge from trunk to spark 12/26/2014 [Spark Branch]
---
Key: HIVE-9157
URL: https://issues.apache.org/jira/browse/HIVE-9157
Project: Hive
Issue Type: Sub-task
Components: Spark
Affects Versions: spark-branch
Reporter: Brock Noland
Assignee: Brock Noland
Fix For: spark-branch
Attachments: HIVE-9157.1-spark.patch.txt,
HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt,
HIVE-9157.1-spark.patch.txt
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9218) Remove authorization_admin_almighty1 from spark tests [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-9218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brock Noland updated HIVE-9218:
---
Resolution: Fixed
Fix Version/s: spark-branch
Status: Resolved (was: Patch Available)
Remove authorization_admin_almighty1 from spark tests [Spark Branch]
Key: HIVE-9218
URL: https://issues.apache.org/jira/browse/HIVE-9218
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Brock Noland
Assignee: Xuefu Zhang
Fix For: spark-branch
Attachments: HIVE-9218.1-spark.patch
The {{authorization_admin_almighty1}} test is authorization only so I don't
think we need to run it on spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260447#comment-14260447
]
Mohit Sabharwal commented on HIVE-9222:
---
HIVE-9220 is not deleting list_bucket_dml_10.q.out, so changes in this patch
are more accurate.
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9222.patch
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Building Hive-0.14 is failing because artifact pentaho-aggdesigner-algorithm-5.1.3-jhyde could not be resolved
Hi folks! I'm still somewhat stuck on this issue. Could someone please guide me? I had to manually download these jars and put them into my ~/.m2/repository from http://repo.spring.io/libs-release/org/pentaho/pentaho-aggdesigner-algorithm/5.1.3-jhyde/ . (When I configured this repository, maven somehow still didn't find the artifacts) I had to do this for the following artifacts:http://repo.spring.io/libs-release/org/pentaho/pentaho-aggdesigner/5.1.3-jhyde/pentaho-aggdesigner-5.1.3-jhyde.pomhttp://repo.spring.io/libs-release/net/hydromatic/linq4j/0.4/linq4j-0.4.jarhttp://repo.spring.io/libs-release/net/hydromatic/quidem/0.1.1/quidem-0.1.1.pom Consequently I was able to build hive, however I don't believe this is a kosher way of producing production binaries, so I am still waiting to hear from you all. Is there another trusted repository I can try which has these artifacts indexed? ThanksRavi On Monday, December 22, 2014 2:14 PM, Ravi Prakash ravi...@ymail.com wrote: Hi! Has anyone tried building Hive-0.14 from source? I'm using the tag for release-0.14.0 https://github.com/apache/hive/releases/tag/release-0.14.0 The command I use is: mvn install -DskipTests -Phadoop-2 -DcreateChecksum=true -Dtez.version=0.5.3 -Dcalcite.version=0.9.2-incubating The build fails for me with the following error:[ERROR] Failed to execute goal on project hive-exec: Could not resolve dependencies for project org.apache.hive:hive-exec:jar:0.14.0: The following artifacts could not be resolved: org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde, net.hydromatic:linq4j:jar:0.4, net.hydromatic:quidem:jar:0.1.1: Could not find artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde in nexus (http://localhost:8081/nexus/content/groups/public) - [Help 1] This is a transitive dependency via the calcite-0.9.2-incubating artifact. Is there a JIRA which someone can please point me to? It seems wrong that an artifact with version 5.1.3-jhyde is required to build Apache Hive, no disrespect to Julian. Am I missing something? ThanksRavi Â
[jira] [Commented] (HIVE-9112) Query may generate different results depending on the number of reducers
[
https://issues.apache.org/jira/browse/HIVE-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260450#comment-14260450
]
Xuefu Zhang commented on HIVE-9112:
---
[~ashutoshc], any further idea on this? Thanks.
Query may generate different results depending on the number of reducers
Key: HIVE-9112
URL: https://issues.apache.org/jira/browse/HIVE-9112
Project: Hive
Issue Type: Bug
Reporter: Chao
Some queries may generate different results depending on the number of
reducers, for example, tests like ppd_multi_insert.q, join_nullsafe.q,
subquery_in.q, etc.
Take subquery_in.q as example, if we add
{noformat}
set mapred.reduce.tasks=3;
{noformat}
to this test file, the result will be different (and wrong):
{noformat}
@@ -903,5 +903,3 @@ where li.l_linenumber = 1 and
POSTHOOK: type: QUERY
POSTHOOK: Input: default@lineitem
A masked pattern was here
-108570 8571
-4297 1798
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9220) HIVE-9109 missed updating result of list_bucket_dml_10
[
https://issues.apache.org/jira/browse/HIVE-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260453#comment-14260453
]
Mohit Sabharwal commented on HIVE-9220:
---
Thanks for catching this, [~navis]! In this case, we have to rename
{{list_bucket_dml_10.q.out}} to {{list_bucket_dml_10.java1.7.q.out}}. And also
create the java 1.8 version {{list_bucket_dml_10.java1.8.q.out}}. I have
included all these changes in HIVE-9222 patch.
If it's ok with you, should we use the changes in HIVE-9222 instead of this
patch ?
HIVE-9109 missed updating result of list_bucket_dml_10
--
Key: HIVE-9220
URL: https://issues.apache.org/jira/browse/HIVE-9220
Project: Hive
Issue Type: Sub-task
Reporter: Navis
Assignee: Navis
Priority: Trivial
Attachments: HIVE-9109.1.patch.txt
list_bucket_dml_10.q.java1.7.out is missing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 29439: UnionProcessor misses results for multi-insert when hive.optimize.union.remove=true
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29439/#review66305 --- ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java https://reviews.apache.org/r/29439/#comment109699 Code before your change was assuming that for union if child operators is 1 it implies multi table insert. Now this change is generalising it across all the operators. 1) I don't know if its a correct assumption to say if childe operators 1 it must be multi table insert. 2) Can't this be done just for union? - John Pullokkaran On Dec. 26, 2014, 10:46 p.m., pengcheng xiong wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29439/ --- (Updated Dec. 26, 2014, 10:46 p.m.) Review request for hive and John Pullokkaran. Repository: hive-git Description --- In union_remove_6.q, just change FROM ( SELECT key, count(1) as values from inputTbl1 group by key UNION ALL SELECT key, count(1) as values from inputTbl1 group by key ) a insert overwrite table outputTbl1 select * insert overwrite table outputTbl2 select *; to FROM ( select * from( SELECT key, count(1) as values from inputTbl1 group by key UNION ALL SELECT key, count(1) as values from inputTbl1 group by key )subq ) a insert overwrite table outputTbl1 select * insert overwrite table outputTbl2 select *; select * from outputtbl2 will output nothing Diffs - ql/src/java/org/apache/hadoop/hive/ql/optimizer/unionproc/UnionProcFactory.java a985c4f ql/src/test/queries/clientpositive/union_remove_6_subq.q PRE-CREATION ql/src/test/results/clientpositive/union_remove_6_subq.q.out PRE-CREATION Diff: https://reviews.apache.org/r/29439/diff/ Testing --- Thanks, pengcheng xiong
[jira] [Resolved] (HIVE-9145) authorization_admin_almighty1.q fails with result diff [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang resolved HIVE-9145. --- Resolution: Duplicate Test disabled via HIVE-9218 authorization_admin_almighty1.q fails with result diff [Spark Branch] - Key: HIVE-9145 URL: https://issues.apache.org/jira/browse/HIVE-9145 Project: Hive Issue Type: Sub-task Components: Spark Reporter: Xuefu Zhang HIVE-7979 enabled this test. However, the test result seems having a timestamp that depends on the date when the test run, which makes the test fail. The same test on trunk give -1 for the timestamp value and thus pass all the time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9039) Support Union Distinct
[
https://issues.apache.org/jira/browse/HIVE-9039?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260462#comment-14260462
]
Hive QA commented on HIVE-9039:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689369/HIVE-9039.08.patch
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 6729 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2211/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2211/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2211/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689369 - PreCommit-HIVE-TRUNK-Build
Support Union Distinct
--
Key: HIVE-9039
URL: https://issues.apache.org/jira/browse/HIVE-9039
Project: Hive
Issue Type: New Feature
Reporter: Pengcheng Xiong
Assignee: Pengcheng Xiong
Attachments: HIVE-9039.01.patch, HIVE-9039.02.patch,
HIVE-9039.03.patch, HIVE-9039.04.patch, HIVE-9039.05.patch,
HIVE-9039.06.patch, HIVE-9039.07.patch, HIVE-9039.08.patch
Current version (Hive 0.14) does not support union (or union distinct). It
only supports union all. In this patch, we try to add this new feature by
rewriting union distinct to union all followed by group by.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-8155) In select statement after * any random characters are allowed in hive but in RDBMS its not allowed
[ https://issues.apache.org/jira/browse/HIVE-8155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-8155: --- Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks, Dong! In select statement after * any random characters are allowed in hive but in RDBMS its not allowed --- Key: HIVE-8155 URL: https://issues.apache.org/jira/browse/HIVE-8155 Project: Hive Issue Type: Improvement Reporter: Ferdinand Xu Assignee: Dong Chen Priority: Critical Fix For: 0.15.0 Attachments: HIVE-8155.1.patch, HIVE-8155.patch In select statement after * any random characters are allowed in hive but in RDBMS its not allowed. Steps: In the below query abcdef is random characters. In RDBMS(oracle): select *abcdef from mytable; Output: ERROR prepare() failed with: ORA-00923: FROM keyword not found where expected In Hive: select *abcdef from mytable; Output: Query worked fine and display all the records of mytable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260465#comment-14260465
]
Brock Noland commented on HIVE-9222:
+1
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9222.patch
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9112) Query may generate different results depending on the number of reducers
[
https://issues.apache.org/jira/browse/HIVE-9112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260469#comment-14260469
]
Ashutosh Chauhan commented on HIVE-9112:
Not really. Probably need to do git bisect to figure out which commit brought
in this bug.
Query may generate different results depending on the number of reducers
Key: HIVE-9112
URL: https://issues.apache.org/jira/browse/HIVE-9112
Project: Hive
Issue Type: Bug
Reporter: Chao
Some queries may generate different results depending on the number of
reducers, for example, tests like ppd_multi_insert.q, join_nullsafe.q,
subquery_in.q, etc.
Take subquery_in.q as example, if we add
{noformat}
set mapred.reduce.tasks=3;
{noformat}
to this test file, the result will be different (and wrong):
{noformat}
@@ -903,5 +903,3 @@ where li.l_linenumber = 1 and
POSTHOOK: type: QUERY
POSTHOOK: Input: default@lineitem
A masked pattern was here
-108570 8571
-4297 1798
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Building Hive-0.14 is failing because artifact pentaho-aggdesigner-algorithm-5.1.3-jhyde could not be resolved
There was an issue with Hive 0.14 when it was released. We missed the fact that it still had two SNAPSHOT dependencies in the pom. If you apply the patches on HIVE-8845 (for Tez) and HIVE-8873 (for Calcite) that should address your issue. This will be fixed in Hive 0.14.1. Alan. Ravi Prakash mailto:ravi...@ymail.com December 22, 2014 at 14:14 Hi! Has anyone tried building Hive-0.14 from source? I'm using the tag for release-0.14.0 https://github.com/apache/hive/releases/tag/release-0.14.0 The command I use is: mvn install -DskipTests -Phadoop-2 -DcreateChecksum=true -Dtez.version=0.5.3 -Dcalcite.version=0.9.2-incubating The build fails for me with the following error:[ERROR] Failed to execute goal on project hive-exec: Could not resolve dependencies for project org.apache.hive:hive-exec:jar:0.14.0: The following artifacts could not be resolved: org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde, net.hydromatic:linq4j:jar:0.4, net.hydromatic:quidem:jar:0.1.1: Could not find artifact org.pentaho:pentaho-aggdesigner-algorithm:jar:5.1.3-jhyde in nexus (http://localhost:8081/nexus/content/groups/public) - [Help 1] This is a transitive dependency via the calcite-0.9.2-incubating artifact. Is there a JIRA which someone can please point me to? It seems wrong that an artifact with version 5.1.3-jhyde is required to build Apache Hive, no disrespect to Julian. Am I missing something? ThanksRavi -- Sent with Postbox http://www.getpostbox.com -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Updated] (HIVE-9157) Merge from trunk to spark 12/26/2014 [Spark Branch]
[ https://issues.apache.org/jira/browse/HIVE-9157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brock Noland updated HIVE-9157: --- Resolution: Fixed Status: Resolved (was: Patch Available) Merge from trunk to spark 12/26/2014 [Spark Branch] --- Key: HIVE-9157 URL: https://issues.apache.org/jira/browse/HIVE-9157 Project: Hive Issue Type: Sub-task Components: Spark Affects Versions: spark-branch Reporter: Brock Noland Assignee: Brock Noland Fix For: spark-branch Attachments: HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt, HIVE-9157.1-spark.patch.txt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[ https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Kumar Singh updated HIVE-8950: - Attachment: HIVE-8950.6.patch Add support in ParquetHiveSerde to create table schema from a parquet file -- Key: HIVE-8950 URL: https://issues.apache.org/jira/browse/HIVE-8950 Project: Hive Issue Type: Improvement Reporter: Ashish Kumar Singh Assignee: Ashish Kumar Singh Attachments: HIVE-8950.1.patch, HIVE-8950.2.patch, HIVE-8950.3.patch, HIVE-8950.4.patch, HIVE-8950.5.patch, HIVE-8950.6.patch, HIVE-8950.patch PARQUET-76 and PARQUET-47 ask for creating parquet backed tables without having to specify the column names and types. As, parquet files store schema in their footer, it is possible to generate hive schema from parquet file's metadata. This will improve usability of parquet backed tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[ https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Na Yang updated HIVE-9119: -- Attachment: HIVE-9119.2.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way - Key: HIVE-9119 URL: https://issues.apache.org/jira/browse/HIVE-9119 Project: Hive Issue Type: Improvement Components: Locking Affects Versions: 0.13.0, 0.14.0, 0.13.1 Reporter: Na Yang Assignee: Na Yang Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch ZooKeeperHiveLockManager does not use zookeeper in the proper way. Currently a new zookeeper client instance is created for each getlock/releaselock query which sometimes causes the number of open connections between HiveServer2 and ZooKeeper exceed the max connection number that zookeeper server allows. To use zookeeper as a distributed lock, there is no need to create a new zookeeper instance for every getlock try. A single zookeeper instance could be reused and shared by ZooKeeperHiveLockManagers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9222) Fix ordering differences due to Java 8 (Part 4)
[
https://issues.apache.org/jira/browse/HIVE-9222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260521#comment-14260521
]
Hive QA commented on HIVE-9222:
---
{color:green}Overall{color}: +1 all checks pass
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689374/HIVE-9222.patch
{color:green}SUCCESS:{color} +1 6723 tests passed
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2212/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2212/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2212/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689374 - PreCommit-HIVE-TRUNK-Build
Fix ordering differences due to Java 8 (Part 4)
---
Key: HIVE-9222
URL: https://issues.apache.org/jira/browse/HIVE-9222
Project: Hive
Issue Type: Sub-task
Components: Tests
Reporter: Mohit Sabharwal
Assignee: Mohit Sabharwal
Attachments: HIVE-9222.patch
This patch fixes the following tests:
(1) TestNegativeCliDriver.testNegativeCliDriver: unset_view_property.q and
unset_table_property.q
{{DDLSemanticAnalyzer.analyzeAlterTableProps()}} gets table properties via
getProps() which must be an insert order map.
(2) TestCliDriver.testCliDriver_overridden_confs
{{VerifyOverriddenConfigsHook}} emits overridden configs. Changed
{{SessionState.overriddenConfigurations}} to insert order map.
(3)
TestNegativeCliDriver.testNegativeCliDriver_columnstats_partlvl_invalid_values
{{ColumnStatsSemanticAnalyzer.getPartKeyValuePairsFromAST()}} gets
{{((ASTNode) tree.getChild(0)}} in different order between Java 7 and Java 8.
The order is different in {{HiveParser.statement()}} itself in
{{ParseDriver.parse()}} so this difference comes from antlr library.
Generated java version specific output.
(4) TestMinimrCliDriver.testCliDriver_list_bucket_dml_10, TestCliDriver
tests: stats_list_bucket.q, list_bucket_dml_12.q and list_bucket_dml_13.q
Looks like these need rebase after HIVE-9206? Not sure what happened here...
(5) TestCliDriver.testCliDriver: mapjoin_hook.q,
auto_join_without_localtask.q, auto_join25.q, multiMapJoin2.q
{{PrintCompletedTasksHook}} prints completed task list, which depends on the
list of tasks added to runnable task list in {{DriverContext}}. Some on
these tasks may get filtered. We see that different tasks are getting
filtered out by the condition resolver in {{ConditionTask}} in Java 8
compared to Java 7.
{{ConditionalTask.execute()}} calls
{{ConditionalResolverCommonJoin.resolveDriverAlias()}} via getTasks(), which
returns a single task based on task to alias map. The next mapred task in the
task list gets filtered out by the resolver in
{{ConditionalTask.resolveTask()}}. In other words, the the mapred task that
shows up first will be kept and the next one will be filtered. Converted task
to alias map to an insert order map so order is same with Java 8 and Java 7.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8950) Add support in ParquetHiveSerde to create table schema from a parquet file
[
https://issues.apache.org/jira/browse/HIVE-8950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260522#comment-14260522
]
Hive QA commented on HIVE-8950:
---
{color:red}Overall{color}: -1 no tests executed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689387/HIVE-8950.6.patch
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2213/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2213/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2213/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.7.0_45-cloudera ]]
+ export JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ JAVA_HOME=/usr/java/jdk1.7.0_45-cloudera
+ export
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+
PATH=/usr/java/jdk1.7.0_45-cloudera/bin/:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-maven-3.0.5/bin:/usr/local/apache-maven-3.0.5/bin:/usr/java/jdk1.7.0_45-cloudera/bin:/usr/local/apache-ant-1.9.1/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/sbin:/home/hiveptest/bin
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-TRUNK-Build-2213/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ svn = \s\v\n ]]
+ [[ -n '' ]]
+ [[ -d apache-svn-trunk-source ]]
+ [[ ! -d apache-svn-trunk-source/.svn ]]
+ [[ ! -d apache-svn-trunk-source ]]
+ cd apache-svn-trunk-source
+ svn revert -R .
Reverted 'ql/src/test/results/clientnegative/unset_table_property.q.out'
Reverted
'ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.out'
Reverted 'ql/src/test/results/clientpositive/list_bucket_dml_12.q.java1.8.out'
Reverted 'ql/src/test/results/clientpositive/stats_list_bucket.q.java1.8.out'
Reverted 'ql/src/test/results/clientpositive/multiMapJoin2.q.out'
Reverted 'ql/src/test/results/clientpositive/auto_join_without_localtask.q.out'
Reverted 'ql/src/test/results/clientpositive/list_bucket_dml_10.q.out'
Reverted
'ql/src/test/org/apache/hadoop/hive/ql/plan/TestConditionalResolverCommonJoin.java'
Reverted
'ql/src/test/queries/clientnegative/columnstats_partlvl_invalid_values.q'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/CommonJoinTaskDispatcher.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/MapJoinResolver.java'
Reverted
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/physical/SortMergeJoinTaskDispatcher.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java'
Reverted 'ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java'
++ egrep -v '^X|^Performing status on external'
++ awk '{print $2}'
++ svn status --no-ignore
+ rm -rf target datanucleus.log ant/target shims/target shims/0.20S/target
shims/0.23/target shims/aggregator/target shims/common/target
shims/scheduler/target packaging/target hbase-handler/target testutils/target
jdbc/target metastore/target itests/target itests/hcatalog-unit/target
itests/test-serde/target itests/qtest/target itests/hive-unit-hadoop2/target
itests/hive-minikdc/target itests/hive-unit/target itests/custom-serde/target
itests/util/target hcatalog/target hcatalog/core/target
hcatalog/streaming/target hcatalog/server-extensions/target
hcatalog/hcatalog-pig-adapter/target hcatalog/webhcat/svr/target
hcatalog/webhcat/java-client/target accumulo-handler/target hwi/target
common/target common/src/gen service/target contrib/target serde/target
beeline/target odbc/target cli/target ql/dependency-reduced-pom.xml ql/target
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.7.out
ql/src/test/results/clientnegative/columnstats_partlvl_invalid_values.q.java1.8.out
ql/src/test/results/clientpositive/list_bucket_dml_10.q.java1.7.out
ql/src/test/results/clientpositive/list_bucket_dml_10.q.java1.8.out
+ svn update
Fetching external item into 'hcatalog/src/test/e2e/harness'
External at revision 1648442.
At revision 1648442.
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+
[jira] [Commented] (HIVE-9195) CBO changes constant to column type
[
https://issues.apache.org/jira/browse/HIVE-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260539#comment-14260539
]
Ashutosh Chauhan commented on HIVE-9195:
+1
CBO changes constant to column type
---
Key: HIVE-9195
URL: https://issues.apache.org/jira/browse/HIVE-9195
Project: Hive
Issue Type: Bug
Components: CBO
Reporter: Navis
Attachments: HIVE-9195.1.patch.txt, HIVE-9195.2.patch.txt,
HIVE-9195.3.patch.txt
Making testcase for HIVE-8613, I've found CBO changes constant expr to column
expr. For example (only in test mode).
{code}
CREATE TABLE bucket (key double, value string) CLUSTERED BY (key) SORTED BY
(key DESC) INTO 4 BUCKETS STORED AS TEXTFILE;
load data local inpath '../../data/files/srcsortbucket1outof4.txt' INTO TABLE
bucket;
load data local inpath '../../data/files/srcsortbucket2outof4.txt' INTO TABLE
bucket;
load data local inpath '../../data/files/srcsortbucket3outof4.txt' INTO TABLE
bucket;
load data local inpath '../../data/files/srcsortbucket4outof4.txt' INTO TABLE
bucket;
select percentile_approx(case when key 100 then cast('NaN' as double) else
key end, 0.5) from bucket;
{code}
It works in shell but in TestCliDriver, that induces argument type exception
creating udaf evaluator, which expects constant OI for second argument.
{noformat}
2014-12-22 17:03:31,433 ERROR parse.SemanticAnalyzer
(SemanticAnalyzer.java:analyzeInternal(10102)) - CBO failed, skipping CBO.
org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: The second argument
must be a constant, but double was passed instead.
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox.getEvaluator(GenericUDAFPercentileApprox.java:146)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1160)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:3794)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:4467)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:5536)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8884)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9745)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9638)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10086)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1107)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1155)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1034)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:206)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:877)
at
org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:136)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23(TestCliDriver.java:120)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-9195) CBO changes constant to column type
[
https://issues.apache.org/jira/browse/HIVE-9195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-9195:
---
Assignee: Navis
CBO changes constant to column type
---
Key: HIVE-9195
URL: https://issues.apache.org/jira/browse/HIVE-9195
Project: Hive
Issue Type: Bug
Components: CBO
Reporter: Navis
Assignee: Navis
Attachments: HIVE-9195.1.patch.txt, HIVE-9195.2.patch.txt,
HIVE-9195.3.patch.txt
Making testcase for HIVE-8613, I've found CBO changes constant expr to column
expr. For example (only in test mode).
{code}
CREATE TABLE bucket (key double, value string) CLUSTERED BY (key) SORTED BY
(key DESC) INTO 4 BUCKETS STORED AS TEXTFILE;
load data local inpath '../../data/files/srcsortbucket1outof4.txt' INTO TABLE
bucket;
load data local inpath '../../data/files/srcsortbucket2outof4.txt' INTO TABLE
bucket;
load data local inpath '../../data/files/srcsortbucket3outof4.txt' INTO TABLE
bucket;
load data local inpath '../../data/files/srcsortbucket4outof4.txt' INTO TABLE
bucket;
select percentile_approx(case when key 100 then cast('NaN' as double) else
key end, 0.5) from bucket;
{code}
It works in shell but in TestCliDriver, that induces argument type exception
creating udaf evaluator, which expects constant OI for second argument.
{noformat}
2014-12-22 17:03:31,433 ERROR parse.SemanticAnalyzer
(SemanticAnalyzer.java:analyzeInternal(10102)) - CBO failed, skipping CBO.
org.apache.hadoop.hive.ql.exec.UDFArgumentTypeException: The second argument
must be a constant, but double was passed instead.
at
org.apache.hadoop.hive.ql.udf.generic.GenericUDAFPercentileApprox.getEvaluator(GenericUDAFPercentileApprox.java:146)
at
org.apache.hadoop.hive.ql.exec.FunctionRegistry.getGenericUDAFEvaluator(FunctionRegistry.java:1160)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.getGenericUDAFEvaluator(SemanticAnalyzer.java:3794)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapGroupByOperator(SemanticAnalyzer.java:4467)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genGroupByPlanMapAggrNoSkew(SemanticAnalyzer.java:5536)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:8884)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9745)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:9638)
at
org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10086)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at
org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(ExplainSemanticAnalyzer.java:74)
at
org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:224)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:419)
at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:305)
at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1107)
at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1155)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1044)
at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1034)
at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:206)
at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:158)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:369)
at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:304)
at
org.apache.hadoop.hive.ql.QTestUtil.executeClient(QTestUtil.java:877)
at
org.apache.hadoop.hive.cli.TestCliDriver.runTest(TestCliDriver.java:136)
at
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_udaf_percentile_approx_23(TestCliDriver.java:120)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HIVE-9223) HiveServer2 on Tez doesn't support concurrent queries
Pala M Muthaia created HIVE-9223: Summary: HiveServer2 on Tez doesn't support concurrent queries Key: HIVE-9223 URL: https://issues.apache.org/jira/browse/HIVE-9223 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Pala M Muthaia When a user submits multiple queries in the same HS2 session (using thrift interface) concurrently, the query goes through the same TezSessionState and ends up being submitted to the same Tez AM, and the second query fails with the error App master already running a DAG Is this by design? I looked into the code, and the comments as well as the code suggest support only for serial execution of queries within the same HiveServer2 session (on tez). This works for CLI environment but in a server, it is plausible that client sends multiple concurrent queries under the same session (e.g: a web app that executes queries for user, such as Cloudera Hue). So shouldn't HS2 on Tez implementation support concurrent queries? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9223) HiveServer2 on Tez doesn't support concurrent queries
[ https://issues.apache.org/jira/browse/HIVE-9223?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260599#comment-14260599 ] Pala M Muthaia commented on HIVE-9223: -- [~hagleitn] [~vikram.dixit], could you comment? I am trying to get the big picture to determine what the right fix should be. Thanks. HiveServer2 on Tez doesn't support concurrent queries -- Key: HIVE-9223 URL: https://issues.apache.org/jira/browse/HIVE-9223 Project: Hive Issue Type: Bug Components: HiveServer2 Reporter: Pala M Muthaia When a user submits multiple queries in the same HS2 session (using thrift interface) concurrently, the query goes through the same TezSessionState and ends up being submitted to the same Tez AM, and the second query fails with the error App master already running a DAG Is this by design? I looked into the code, and the comments as well as the code suggest support only for serial execution of queries within the same HiveServer2 session (on tez). This works for CLI environment but in a server, it is plausible that client sends multiple concurrent queries under the same session (e.g: a web app that executes queries for user, such as Cloudera Hue). So shouldn't HS2 on Tez implementation support concurrent queries? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9220) HIVE-9109 missed updating result of list_bucket_dml_10
[ https://issues.apache.org/jira/browse/HIVE-9220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260608#comment-14260608 ] Navis commented on HIVE-9220: - [~mohitsabharwal] I'm ok with that. Just wanted others not to seek the cause of fail again. HIVE-9109 missed updating result of list_bucket_dml_10 -- Key: HIVE-9220 URL: https://issues.apache.org/jira/browse/HIVE-9220 Project: Hive Issue Type: Sub-task Reporter: Navis Assignee: Navis Priority: Trivial Attachments: HIVE-9109.1.patch.txt list_bucket_dml_10.q.java1.7.out is missing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9215) Some mapjoin queries broken with IdentityProjectRemover with PPD
[ https://issues.apache.org/jira/browse/HIVE-9215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-9215: Resolution: Fixed Fix Version/s: 0.15.0 Status: Resolved (was: Patch Available) Committed to trunk. Thanks Xuefu and Szehon, for the review! Some mapjoin queries broken with IdentityProjectRemover with PPD Key: HIVE-9215 URL: https://issues.apache.org/jira/browse/HIVE-9215 Project: Hive Issue Type: Bug Components: Logical Optimizer Affects Versions: 0.15.0 Reporter: Szehon Ho Assignee: Navis Fix For: 0.15.0 Attachments: HIVE-9215.1.patch.txt, HIVE-9215.2.patch.txt, auto_join_ppd.q IdentityProjectRemover (hive.optimize.remove.identity.project) with PPD will sometimes make mapjoin query that returns the wrong result in MR case as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-8956) Hive hangs while some error/exception happens beyond job execution [Spark Branch]
[
https://issues.apache.org/jira/browse/HIVE-8956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260652#comment-14260652
]
Rui Li commented on HIVE-8956:
--
Hi [~chirag.aggarwal], which query triggered this error? And you may need to
check if the configured memory is not enough. Some related properties are
{{spark.executor.memory}} and {{spark.storage.memoryFraction}}.
Hive hangs while some error/exception happens beyond job execution [Spark
Branch]
-
Key: HIVE-8956
URL: https://issues.apache.org/jira/browse/HIVE-8956
Project: Hive
Issue Type: Sub-task
Components: Spark
Reporter: Chengxiang Li
Assignee: Rui Li
Labels: Spark-M3
Fix For: spark-branch
Attachments: HIVE-8956.1-spark.patch
Remote spark client communicate with remote spark context asynchronously, if
error/exception is throw out during job execution in remote spark context, it
would be wrapped and send back to remote spark client, but if error/exception
is throw out beyond job execution, such as job serialized failed, remote
spark client would never know what's going on in remote spark context, and it
would hangs there.
Set a timeout in remote spark client side may not a great idea, as we are not
sure how long the query executed in spark cluster. we need find a way to
check whether job has failed(whole life cycle) in remote spark context.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-5718) Support direct fetch for lateral views, sub queries, etc.
[ https://issues.apache.org/jira/browse/HIVE-5718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Navis updated HIVE-5718: Attachment: HIVE-5718.15.patch.txt Updated golden file Support direct fetch for lateral views, sub queries, etc. - Key: HIVE-5718 URL: https://issues.apache.org/jira/browse/HIVE-5718 Project: Hive Issue Type: Improvement Components: Query Processor Reporter: Navis Assignee: Navis Priority: Trivial Attachments: D13857.1.patch, D13857.2.patch, D13857.3.patch, HIVE-5718.10.patch.txt, HIVE-5718.11.patch.txt, HIVE-5718.12.patch.txt, HIVE-5718.13.patch.txt, HIVE-5718.14.patch.txt, HIVE-5718.15.patch.txt, HIVE-5718.4.patch.txt, HIVE-5718.5.patch.txt, HIVE-5718.6.patch.txt, HIVE-5718.7.patch.txt, HIVE-5718.8.patch.txt, HIVE-5718.9.patch.txt, HIVE-5718.diff-v11-v12.patch Extend HIVE-2925 with LV and SubQ. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9213) Improve the mask pattern in QTestUtil to save partial directory info in test result
[ https://issues.apache.org/jira/browse/HIVE-9213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260663#comment-14260663 ] Dong Chen commented on HIVE-9213: - Thanks for your review and commit! Improve the mask pattern in QTestUtil to save partial directory info in test result --- Key: HIVE-9213 URL: https://issues.apache.org/jira/browse/HIVE-9213 Project: Hive Issue Type: Sub-task Reporter: Dong Chen Assignee: Dong Chen Fix For: encryption-branch Attachments: HIVE-9213.1.patch, HIVE-9213.patch NO PRECOMMIT TESTS The mask pattern in QTestUtil will mask directory in test result, since the directory varies in different test env. However, in Encryption test, the directory info is needed to verify the intermediate files are put in proper table. The whole directory is not necessary, and part of it is enough. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-9119) ZooKeeperHiveLockManager does not use zookeeper in the proper way
[
https://issues.apache.org/jira/browse/HIVE-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260673#comment-14260673
]
Hive QA commented on HIVE-9119:
---
{color:red}Overall{color}: -1 at least one tests failed
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12689389/HIVE-9119.2.patch
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 6700 tests
executed
*Failed tests:*
{noformat}
TestHBaseCliDriver-hbase_binary_map_queries_prefix.q-hbase_stats.q-ppd_key_ranges.q
- did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_handler_bulk.q-hbase_scan_params.q-hbase_single_sourced_multi_insert.q
- did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_handler_snapshot.q-hbase_stats3.q-hbase_custom_key3.q
- did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_ppd_key_range.q-hbase_custom_key.q-hbase_pushdown.q -
did not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_stats2.q-hbase_custom_key2.q-hbase_timestamp.q - did
not produce a TEST-*.xml file
TestHBaseCliDriver-hbase_stats_empty_partition.q-external_table_ppd.q-hbase_ppd_join.q
- did not produce a TEST-*.xml file
TestHBaseNegativeCliDriver - did not produce a TEST-*.xml file
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_list_bucket_dml_10
org.apache.hadoop.hive.cli.TestMinimrCliDriver.testCliDriver_schemeAuthority
{noformat}
Test results:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2214/testReport
Console output:
http://ec2-174-129-184-35.compute-1.amazonaws.com/jenkins/job/PreCommit-HIVE-TRUNK-Build/2214/console
Test logs:
http://ec2-174-129-184-35.compute-1.amazonaws.com/logs/PreCommit-HIVE-TRUNK-Build-2214/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12689389 - PreCommit-HIVE-TRUNK-Build
ZooKeeperHiveLockManager does not use zookeeper in the proper way
-
Key: HIVE-9119
URL: https://issues.apache.org/jira/browse/HIVE-9119
Project: Hive
Issue Type: Improvement
Components: Locking
Affects Versions: 0.13.0, 0.14.0, 0.13.1
Reporter: Na Yang
Assignee: Na Yang
Attachments: HIVE-9119.1.patch, HIVE-9119.2.patch
ZooKeeperHiveLockManager does not use zookeeper in the proper way.
Currently a new zookeeper client instance is created for each
getlock/releaselock query which sometimes causes the number of open
connections between
HiveServer2 and ZooKeeper exceed the max connection number that zookeeper
server allows.
To use zookeeper as a distributed lock, there is no need to create a new
zookeeper instance for every getlock try. A single zookeeper instance could
be reused and shared by ZooKeeperHiveLockManagers.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-8410) Typo in DOAP - incorrect category URL
[
https://issues.apache.org/jira/browse/HIVE-8410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260691#comment-14260691
]
Ferdinand Xu commented on HIVE-8410:
Hi [~s...@apache.org], how can I apply it to the DOAP? Sorry for my newbie
question.
Typo in DOAP - incorrect category URL
-
Key: HIVE-8410
URL: https://issues.apache.org/jira/browse/HIVE-8410
Project: Hive
Issue Type: Bug
Environment: http://svn.apache.org/repos/asf/hive/trunk/doap_Hive.rdf
Reporter: Sebb
Assignee: Ferdinand Xu
Attachments: HIVE-8410.patch
The DOAP contains the following:
{code}
category rdf:resource=http://www.apache.org/category/database; /
{code}
However, the URL is incorrect; it must be
{code}
category rdf:resource=http://projects.apache.org/category/database; /
{code}
Please fix this
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-9215) Some mapjoin queries broken with IdentityProjectRemover with PPD
[ https://issues.apache.org/jira/browse/HIVE-9215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260695#comment-14260695 ] Xuefu Zhang commented on HIVE-9215: --- Merged to Spark branch as well. Thanks, Navis. Some mapjoin queries broken with IdentityProjectRemover with PPD Key: HIVE-9215 URL: https://issues.apache.org/jira/browse/HIVE-9215 Project: Hive Issue Type: Bug Components: Logical Optimizer Affects Versions: 0.15.0 Reporter: Szehon Ho Assignee: Navis Fix For: 0.15.0 Attachments: HIVE-9215.1.patch.txt, HIVE-9215.2.patch.txt, auto_join_ppd.q IdentityProjectRemover (hive.optimize.remove.identity.project) with PPD will sometimes make mapjoin query that returns the wrong result in MR case as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9215) Some mapjoin queries broken with IdentityProjectRemover with PPD
[ https://issues.apache.org/jira/browse/HIVE-9215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xuefu Zhang updated HIVE-9215: -- Affects Version/s: spark-branch Some mapjoin queries broken with IdentityProjectRemover with PPD Key: HIVE-9215 URL: https://issues.apache.org/jira/browse/HIVE-9215 Project: Hive Issue Type: Bug Components: Logical Optimizer Affects Versions: spark-branch, 0.15.0 Reporter: Szehon Ho Assignee: Navis Fix For: 0.15.0 Attachments: HIVE-9215.1.patch.txt, HIVE-9215.2.patch.txt, auto_join_ppd.q IdentityProjectRemover (hive.optimize.remove.identity.project) with PPD will sometimes make mapjoin query that returns the wrong result in MR case as well. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HIVE-9224) CBO (Calcite Return Path): Inline Table, Properties
Laljo John Pullokkaran created HIVE-9224: Summary: CBO (Calcite Return Path): Inline Table, Properties Key: HIVE-9224 URL: https://issues.apache.org/jira/browse/HIVE-9224 Project: Hive Issue Type: Sub-task Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-9224) CBO (Calcite Return Path): Inline Table, Properties
[ https://issues.apache.org/jira/browse/HIVE-9224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laljo John Pullokkaran updated HIVE-9224: - Attachment: HIVE-9224.patch CBO (Calcite Return Path): Inline Table, Properties --- Key: HIVE-9224 URL: https://issues.apache.org/jira/browse/HIVE-9224 Project: Hive Issue Type: Sub-task Components: CBO Reporter: Laljo John Pullokkaran Assignee: Laljo John Pullokkaran Fix For: 0.15.0 Attachments: HIVE-9224.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
