[jira] [Commented] (HIVE-13027) Configuration changes to improve logging performance
[ https://issues.apache.org/jira/browse/HIVE-13027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381081#comment-15381081 ] Lefty Leverenz commented on HIVE-13027: --- [~prasanth_j], does this need to be documented in the wiki? I ask because of your comment about async logging being added in 2.1 on the thread "hive 2.1.0 beeline cannot show verbose log" on user@hive. * [user@hive -- Re: hive 2.1.0 beeline cannot show verbose log | http://mail-archives.apache.org/mod_mbox/hive-user/201607.mbox/%3cc6dbd783301a42bf.cdaf89e4-afbc-4329-a557-db1ea7e44...@mail.outlook.com%3e] * [Getting Started -- Hive Logging | https://cwiki.apache.org/confluence/display/Hive/GettingStarted#GettingStarted-HiveLogging] > Configuration changes to improve logging performance > > > Key: HIVE-13027 > URL: https://issues.apache.org/jira/browse/HIVE-13027 > Project: Hive > Issue Type: Improvement > Components: Logging >Affects Versions: 2.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 2.1.0 > > Attachments: HIVE-13027.1.patch, HIVE-13027.2.patch, > HIVE-13027.3.patch, HIVE-13027.3.patch, HIVE-13027.4.patch > > > For LLAP and HS2, some configuration changes can be made to improve logging > performance > 1) LOG4j2's async logger claims to have 6-68 times better performance than > synchronous logger. https://logging.apache.org/log4j/2.x/manual/async.html > 2) Replace File appenders with RandomAccessFileAppender that claims to be > 20-200% more performant. > https://logging.apache.org/log4j/2.x/manual/appenders.html#RandomAccessFileAppender > Also make async logging configurable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14224) LLAP rename query specific log files once a query is complete
[
https://issues.apache.org/jira/browse/HIVE-14224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381076#comment-15381076
]
Hive QA commented on HIVE-14224:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818265/HIVE-14224.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 10330 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.llap.tezplugins.TestLlapTaskSchedulerService.testDelayedLocalityNodeCommErrorImmediateAllocation
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/547/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/547/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-547/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818265 - PreCommit-HIVE-MASTER-Build
> LLAP rename query specific log files once a query is complete
> -
>
> Key: HIVE-14224
> URL: https://issues.apache.org/jira/browse/HIVE-14224
> Project: Hive
> Issue Type: Improvement
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-14224.02.patch, HIVE-14224.wip.01.patch
>
>
> Once a query is complete, rename the query specific log file so that YARN can
> aggregate the logs (once it's configured to do so).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381074#comment-15381074
]
Lefty Leverenz commented on HIVE-13884:
---
Thanks Sergio. I moved *hive.metastore.limit.partition.request* into the
MetaStore section and put internal links on the crossreferences between it and
*hive.limit.query.max.table.partition*.
Here's the link to the new configuration parameter:
* [ConfigurationProperties -- MetaStore --
hive.metastore.limit.partition.request |
https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.metastore.limit.partition.request]
Removing the TODOC2.2 label.
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Fix For: 2.2.0
>
> Attachments: HIVE-13884.1.patch, HIVE-13884.10.patch,
> HIVE-13884.2.patch, HIVE-13884.3.patch, HIVE-13884.4.patch,
> HIVE-13884.5.patch, HIVE-13884.6.patch, HIVE-13884.7.patch,
> HIVE-13884.8.patch, HIVE-13884.9.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13884) Disallow queries in HMS fetching more than a configured number of partitions
[
https://issues.apache.org/jira/browse/HIVE-13884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lefty Leverenz updated HIVE-13884:
--
Labels: (was: TODOC2.2)
> Disallow queries in HMS fetching more than a configured number of partitions
>
>
> Key: HIVE-13884
> URL: https://issues.apache.org/jira/browse/HIVE-13884
> Project: Hive
> Issue Type: Improvement
>Reporter: Mohit Sabharwal
>Assignee: Sergio Peña
> Fix For: 2.2.0
>
> Attachments: HIVE-13884.1.patch, HIVE-13884.10.patch,
> HIVE-13884.2.patch, HIVE-13884.3.patch, HIVE-13884.4.patch,
> HIVE-13884.5.patch, HIVE-13884.6.patch, HIVE-13884.7.patch,
> HIVE-13884.8.patch, HIVE-13884.9.patch
>
>
> Currently the PartitionPruner requests either all partitions or partitions
> based on filter expression. In either scenarios, if the number of partitions
> accessed is large there can be significant memory pressure at the HMS server
> end.
> We already have a config {{hive.limit.query.max.table.partition}} that
> enforces limits on number of partitions that may be scanned per operator. But
> this check happens after the PartitionPruner has already fetched all
> partitions.
> We should add an option at PartitionPruner level to disallow queries that
> attempt to access number of partitions beyond a configurable limit.
> Note that {{hive.mapred.mode=strict}} disallow queries without a partition
> filter in PartitionPruner, but this check accepts any query with a pruning
> condition, even if partitions fetched are large. In multi-tenant
> environments, admins could use more control w.r.t. number of partitions
> allowed based on HMS memory capacity.
> One option is to have PartitionPruner first fetch the partition names
> (instead of partition specs) and throw an exception if number of partitions
> exceeds the configured value. Otherwise, fetch the partition specs.
> Looks like the existing {{listPartitionNames}} call could be used if extended
> to take partition filter expressions like {{getPartitionsByExpr}} call does.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14258) Reduce task timed out because CommonJoinOperator.genUniqueJoinObject took too long to finish without reporting progress
[
https://issues.apache.org/jira/browse/HIVE-14258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381071#comment-15381071
]
Jimmy Xiang commented on HIVE-14258:
+1
> Reduce task timed out because CommonJoinOperator.genUniqueJoinObject took too
> long to finish without reporting progress
> ---
>
> Key: HIVE-14258
> URL: https://issues.apache.org/jira/browse/HIVE-14258
> Project: Hive
> Issue Type: Bug
> Components: Query Processor
>Affects Versions: 2.1.0
>Reporter: zhihai xu
>Assignee: zhihai xu
> Attachments: HIVE-14258.patch
>
>
> Reduce task timed out because CommonJoinOperator.genUniqueJoinObject took too
> long to finish without reporting progress.
> This timeout happened when reducer.close() is called in ReduceTask.java.
> CommonJoinOperator.genUniqueJoinObject() called by reducer.close() will loop
> over every row in the AbstractRowContainer. This can take a long time if
> there are a large number or rows, and during this time, it does not report
> progress. If this runs for long enough more than "mapreduce.task.timeout",
> ApplicationMaster will kill the task for failing to report progress.
> we configured "mapreduce.task.timeout" as 10 minutes. I captured the stack
> trace in the 10 minutes before AM killed the reduce task at 2016-07-15
> 07:19:11.
> The following three stack traces can prove it:
> at 2016-07-15 07:09:42:
> {code}
> "main" prio=10 tid=0x7f90ec017000 nid=0xd193 runnable [0x7f90f62e5000]
>java.lang.Thread.State: RUNNABLE
> at java.io.FileInputStream.readBytes(Native Method)
> at java.io.FileInputStream.read(FileInputStream.java:272)
> at
> org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.read(RawLocalFileSystem.java:154)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:235)
> at java.io.BufferedInputStream.read1(BufferedInputStream.java:275)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:334)
> - locked <0x0007deecefb0> (a
> org.apache.hadoop.fs.BufferedFSInputStream)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at
> org.apache.hadoop.fs.FSInputChecker.readFully(FSInputChecker.java:436)
> at
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.readChunk(ChecksumFileSystem.java:252)
> at
> org.apache.hadoop.fs.FSInputChecker.readChecksumChunk(FSInputChecker.java:276)
> at org.apache.hadoop.fs.FSInputChecker.fill(FSInputChecker.java:214)
> at org.apache.hadoop.fs.FSInputChecker.read1(FSInputChecker.java:232)
> at org.apache.hadoop.fs.FSInputChecker.read(FSInputChecker.java:196)
> - locked <0x0007deecb978> (a
> org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at
> org.apache.hadoop.io.DataOutputBuffer$Buffer.write(DataOutputBuffer.java:70)
> at
> org.apache.hadoop.io.DataOutputBuffer.write(DataOutputBuffer.java:120)
> at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2359)
> - locked <0x0007deec8f70> (a
> org.apache.hadoop.io.SequenceFile$Reader)
> at
> org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2491)
> - locked <0x0007deec8f70> (a
> org.apache.hadoop.io.SequenceFile$Reader)
> at
> org.apache.hadoop.mapred.SequenceFileRecordReader.next(SequenceFileRecordReader.java:82)
> - locked <0x0007deec82f0> (a
> org.apache.hadoop.mapred.SequenceFileRecordReader)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.nextBlock(RowContainer.java:360)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.next(RowContainer.java:267)
> at
> org.apache.hadoop.hive.ql.exec.persistence.RowContainer.next(RowContainer.java:74)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.genUniqueJoinObject(CommonJoinOperator.java:644)
> at
> org.apache.hadoop.hive.ql.exec.CommonJoinOperator.checkAndGenObject(CommonJoinOperator.java:750)
> at
> org.apache.hadoop.hive.ql.exec.JoinOperator.endGroup(JoinOperator.java:256)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecReducer.close(ExecReducer.java:284)
> at
> org.apache.hadoop.mapred.ReduceTask.runOldReducer(ReduceTask.java:453)
> at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:392)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
>
[jira] [Commented] (HIVE-13369) AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing the "best" base file
[
https://issues.apache.org/jira/browse/HIVE-13369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381045#comment-15381045
]
Hive QA commented on HIVE-13369:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818264/HIVE-13369.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10330 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
org.apache.hadoop.hive.ql.io.TestAcidUtils.testBestBase
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/546/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/546/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-546/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818264 - PreCommit-HIVE-MASTER-Build
> AcidUtils.getAcidState() is not paying attention toValidTxnList when choosing
> the "best" base file
> --
>
> Key: HIVE-13369
> URL: https://issues.apache.org/jira/browse/HIVE-13369
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 1.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Blocker
> Attachments: HIVE-13369.1.patch, HIVE-13369.2.patch,
> HIVE-13369.3.patch, HIVE-13369.4.patch
>
>
> The JavaDoc on getAcidState() reads, in part:
> "Note that because major compactions don't
>preserve the history, we can't use a base directory that includes a
>transaction id that we must exclude."
> which is correct but there is nothing in the code that does this.
> And if we detect a situation where txn X must be excluded but and there are
> deltas that contain X, we'll have to abort the txn. This can't (reasonably)
> happen with auto commit mode, but with multi statement txns it's possible.
> Suppose some long running txn starts and lock in snapshot at 17 (HWM). An
> hour later it decides to access some partition for which all txns < 20 (for
> example) have already been compacted (i.e. GC'd).
> ==
> Here is a more concrete example. Let's say the file for table A are as
> follows and created in the order listed.
> delta_4_4
> delta_5_5
> delta_4_5
> base_5
> delta_16_16
> delta_17_17
> base_17 (for example user ran major compaction)
> let's say getAcidState() is called with ValidTxnList(20:16), i.e. with HWM=20
> and ExceptionList=<16>
> Assume that all txns <= 20 commit.
> Reader can't use base_17 because it has result of txn16. So it should chose
> base_5 "TxnBase bestBase" in _getChildState()_.
> Then the reset of the logic in _getAcidState()_ should choose delta_16_16 and
> delta_17_17 in _Directory_ object. This would represent acceptable snapshot
> for such reader.
> The issue is if at the same time the Cleaner process is running. It will see
> everything with txnid<17 as obsolete. Then it will check lock manger state
> and decide to delete (as there may not be any locks in LM for table A). The
> order in which the files are deleted is undefined right now. It may delete
> delta_16_16 and delta_17_17 first and right at this moment the read request
> with ValidTxnList(20:16) arrives (such snapshot may have bee locked in by
> some multi-stmt txn that started some time ago. It acquires locks after the
> Cleaner checks LM state and calls getAcidState(). This request will choose
> base_5 but it won't see delta_16_16 and delta_17_17 and thus return the
> snapshot w/o modifications made by those txns.
> [This is not possible currently since we only support autoCommit=true. The
> reason is the a query (0) opens txn (if
[jira] [Commented] (HIVE-13934) Configure Tez to make nocondiional task size memory available for the Processor
[
https://issues.apache.org/jira/browse/HIVE-13934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381002#comment-15381002
]
Hive QA commented on HIVE-13934:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818251/HIVE-13934.9.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10331 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/544/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/544/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-544/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818251 - PreCommit-HIVE-MASTER-Build
> Configure Tez to make nocondiional task size memory available for the
> Processor
> ---
>
> Key: HIVE-13934
> URL: https://issues.apache.org/jira/browse/HIVE-13934
> Project: Hive
> Issue Type: Bug
>Reporter: Wei Zheng
>Assignee: Wei Zheng
> Attachments: HIVE-13934.1.patch, HIVE-13934.2.patch,
> HIVE-13934.3.patch, HIVE-13934.4.patch, HIVE-13934.6.patch,
> HIVE-13934.7.patch, HIVE-13934.8.patch, HIVE-13934.9.patch
>
>
> Currently, noconditionaltasksize is not validated against the container size,
> the reservations made in the container by Tez for Inputs / Outputs etc.
> Check this at compile time to see if enough memory is available, or set up
> the vertex to reserve additional memory for the Processor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14251) Union All of different types resolves to incorrect data
[
https://issues.apache.org/jira/browse/HIVE-14251?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380966#comment-15380966
]
Hive QA commented on HIVE-14251:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818248/HIVE-14251.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10331 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_unionall_typeconversion
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
org.apache.hive.spark.client.TestSparkClient.testJobSubmission
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/543/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/543/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-543/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818248 - PreCommit-HIVE-MASTER-Build
> Union All of different types resolves to incorrect data
> ---
>
> Key: HIVE-14251
> URL: https://issues.apache.org/jira/browse/HIVE-14251
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14251.1.patch
>
>
> create table src(c1 date, c2 int, c3 double);
> insert into src values ('2016-01-01',5,1.25);
> select * from
> (select c1 from src union all
> select c2 from src union all
> select c3 from src) t;
> It will return NULL for the c1 values. Seems the common data type is resolved
> to the last c3 which is double.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14260) show WriteEntity writetype in explain output
[ https://issues.apache.org/jira/browse/HIVE-14260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14260: - Attachment: HIVE-14260.1.patch > show WriteEntity writetype in explain output > > > Key: HIVE-14260 > URL: https://issues.apache.org/jira/browse/HIVE-14260 > Project: Hive > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Thejas M Nair > Attachments: HIVE-14260.1.patch > > > It is useful to see the WriteEntity writeType in explain output, specially > for 'explain authorization'. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14167) Use work directories provided by Tez instead of directly using YARN local dirs
[
https://issues.apache.org/jira/browse/HIVE-14167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380931#comment-15380931
]
Hive QA commented on HIVE-14167:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818208/HIVE-14167.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/541/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/541/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-541/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818208 - PreCommit-HIVE-MASTER-Build
> Use work directories provided by Tez instead of directly using YARN local dirs
> --
>
> Key: HIVE-14167
> URL: https://issues.apache.org/jira/browse/HIVE-14167
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 2.1.0
>Reporter: Siddharth Seth
>Assignee: Wei Zheng
> Attachments: HIVE-14167.1.patch, HIVE-14167.2.patch,
> HIVE-14167.3.patch
>
>
> HIVE-13303 fixed things to use multiple directories instead of a single tmp
> directory. However it's using yarn-local-dirs directly.
> I'm not sure how well using the yarn-local-dir will work on a secure cluster.
> Would be better to use Tez*Context.getWorkDirs. This provides an app specific
> directory - writable by the user.
> cc [~sershe]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[ https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14214: Status: Patch Available (was: In Progress) > ORC Schema Evolution and Predicate Push Down do not work together (no rows > returned) > > > Key: HIVE-14214 > URL: https://issues.apache.org/jira/browse/HIVE-14214 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14214.01.patch, HIVE-14214.02.patch, > HIVE-14214.WIP.patch > > > In Schema Evolution, the reader schema is different than the file schema > which is used to evaluate predicate push down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[ https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-14214: Attachment: HIVE-14214.02.patch > ORC Schema Evolution and Predicate Push Down do not work together (no rows > returned) > > > Key: HIVE-14214 > URL: https://issues.apache.org/jira/browse/HIVE-14214 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Attachments: HIVE-14214.01.patch, HIVE-14214.02.patch, > HIVE-14214.WIP.patch > > > In Schema Evolution, the reader schema is different than the file schema > which is used to evaluate predicate push down. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-11863) FS based stats collection generates wrong results for tez (for union queries)
[ https://issues.apache.org/jira/browse/HIVE-11863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong resolved HIVE-11863. Resolution: Fixed Assignee: Pengcheng Xiong duplicate of HIVE-14236 > FS based stats collection generates wrong results for tez (for union queries) > - > > Key: HIVE-11863 > URL: https://issues.apache.org/jira/browse/HIVE-11863 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran >Assignee: Pengcheng Xiong > > FS based stats collection is the default way to collect stats. However, there > are some cases (involving unions) where it generates wrong results. Refer > test case in HIVE-11860 and compare test cli driver results against tez > results. Also it will be good to extend statsfs.q test case with union > queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14226) Invalid check on an ASTNode#toStringTree in CalcitePlanner
[ https://issues.apache.org/jira/browse/HIVE-14226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14226: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Barna! > Invalid check on an ASTNode#toStringTree in CalcitePlanner > -- > > Key: HIVE-14226 > URL: https://issues.apache.org/jira/browse/HIVE-14226 > Project: Hive > Issue Type: Bug >Reporter: Barna Zsombor Klara >Assignee: Barna Zsombor Klara >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-14226.patch > > > The patch for HIVE-12590 has modified the string tree representation of an > ASTNode, so that everything except string literals are now lowercase. > However CalcitePlanner#genSelectLogicalPlan is still checking the occurrence > of an uppercase TOK_FUNCTIONDI constant in the string tree. > (Line 2997 in CalcitePlanner on master) > I think this check will never evaluate to true, so we should either be using > a lowercase String, or remove the check if it is unnecessary. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-11863) FS based stats collection generates wrong results for tez (for union queries)
[ https://issues.apache.org/jira/browse/HIVE-11863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380896#comment-15380896 ] Ashutosh Chauhan commented on HIVE-11863: - [~pxiong] Can this be closed as dupe of HIVE-14236 ? > FS based stats collection generates wrong results for tez (for union queries) > - > > Key: HIVE-11863 > URL: https://issues.apache.org/jira/browse/HIVE-11863 > Project: Hive > Issue Type: Bug >Affects Versions: 1.3.0, 2.0.0 >Reporter: Prasanth Jayachandran > > FS based stats collection is the default way to collect stats. However, there > are some cases (involving unions) where it generates wrong results. Refer > test case in HIVE-11860 and compare test cli driver results against tez > results. Also it will be good to extend statsfs.q test case with union > queries. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-14259: Status: Patch Available (was: Open) > FileUtils.isSubDir may return incorrect result > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-12244: Resolution: Fixed Fix Version/s: (was: 1.2.1) 2.2.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Zoltan! > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Zoltan Haindrich >Priority: Minor > Labels: patch > Fix For: 2.2.0 > > Attachments: HIVE-12244.1.patch, HIVE-12244.10.patch, > HIVE-12244.11.patch, HIVE-12244.12.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, > HIVE-12244.8.patch, HIVE-12244.9.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13191) DummyTable map joins mix up columns between tables
[
https://issues.apache.org/jira/browse/HIVE-13191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380888#comment-15380888
]
Hive QA commented on HIVE-13191:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818175/HIVE-13191.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 86 failed/errored test(s), 10327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ambiguous_col
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cbo_rp_join1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_cluster
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_constant_prop_1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_correlationoptimizer14
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join15
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join20
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join21
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join40
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join44
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_nullsafe
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_reorder3
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_join_thrift
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_lateral_view_noalias
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_masking_disablecbo_2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multiMapJoin1
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_multi_insert_with_join2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_pcs
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd2
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_clusterby
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ppd_multi_insert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_sample8
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_select_transform_hint
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_semijoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_skewjoinopt10
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union26
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_union_lateralview
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_join30
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vector_leftsemi_mapjoin
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_vectorized_ptf
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join21
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join29
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_join30
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_auto_sortmerge_join_13
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_explainuser_2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_join_nullsafe
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_llap_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_lvj_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_optimize_nullscan
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_ptf_streaming
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_skewjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_auto_smb_mapjoin_14
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_join30
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_leftsemi_mapjoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vector_nullsafe_join
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_bucketmapjoin1
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380887#comment-15380887 ] Ashutosh Chauhan commented on HIVE-12244: - +1 > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Zoltan Haindrich >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.10.patch, > HIVE-12244.11.patch, HIVE-12244.12.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, > HIVE-12244.8.patch, HIVE-12244.9.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14244) bucketmap right outer join query throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14244:
Fix Version/s: 2.1.1
2.2.0
> bucketmap right outer join query throws ArrayIndexOutOfBoundsException
> --
>
> Key: HIVE-14244
> URL: https://issues.apache.org/jira/browse/HIVE-14244
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer, Tez
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Jagruti Varia
>Assignee: Zhiyuan Yang
> Fix For: 2.2.0, 2.1.1
>
> Attachments: HIVE-14244.1.patch
>
>
> bucketmap right outer join on partitioned bucketed table throws this error:
> {noformat}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1466710232033_0539_6_00,
> diagnostics=[Task failed, taskId=task_1466710232033_0539_6_00_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1466710232033_0539_6_00_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:850)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:416)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:104)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:314)
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:257)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:253)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:552)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:398)
> ... 22 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
>
[jira] [Updated] (HIVE-14244) bucketmap right outer join query throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14244:
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master & branch-2.1. Thanks, Zhiyuan!
> bucketmap right outer join query throws ArrayIndexOutOfBoundsException
> --
>
> Key: HIVE-14244
> URL: https://issues.apache.org/jira/browse/HIVE-14244
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer, Tez
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Jagruti Varia
>Assignee: Zhiyuan Yang
> Attachments: HIVE-14244.1.patch
>
>
> bucketmap right outer join on partitioned bucketed table throws this error:
> {noformat}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1466710232033_0539_6_00,
> diagnostics=[Task failed, taskId=task_1466710232033_0539_6_00_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1466710232033_0539_6_00_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:850)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:416)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:104)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:314)
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:257)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:253)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:552)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:398)
> ... 22 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
>
[jira] [Updated] (HIVE-14244) bucketmap right outer join query throws ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/HIVE-14244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-14244:
Component/s: Tez
Physical Optimizer
> bucketmap right outer join query throws ArrayIndexOutOfBoundsException
> --
>
> Key: HIVE-14244
> URL: https://issues.apache.org/jira/browse/HIVE-14244
> Project: Hive
> Issue Type: Bug
> Components: Physical Optimizer, Tez
>Affects Versions: 1.3.0, 2.1.0
>Reporter: Jagruti Varia
>Assignee: Zhiyuan Yang
> Attachments: HIVE-14244.1.patch
>
>
> bucketmap right outer join on partitioned bucketed table throws this error:
> {noformat}
> Vertex failed, vertexName=Map 1, vertexId=vertex_1466710232033_0539_6_00,
> diagnostics=[Task failed, taskId=task_1466710232033_0539_6_00_00,
> diagnostics=[TaskAttempt 0 failed, info=[Error: Error while running task (
> failure ) :
> attempt_1466710232033_0539_6_00_00_0:java.lang.RuntimeException:
> java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException:
> Hive Runtime Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:211)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.run(TezProcessor.java:168)
> at
> org.apache.tez.runtime.LogicalIOProcessorRuntimeTask.run(LogicalIOProcessorRuntimeTask.java:370)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:73)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable$1.run(TaskRunner2Callable.java:61)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1724)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:61)
> at
> org.apache.tez.runtime.task.TaskRunner2Callable.callInternal(TaskRunner2Callable.java:37)
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:95)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.pushRecord(MapRecordSource.java:70)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordProcessor.run(MapRecordProcessor.java:393)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185)
> ... 14 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:850)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:416)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorReduceSinkOperator.process(VectorReduceSinkOperator.java:104)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:879)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:762)
> ... 18 more
> Caused by: java.lang.ArrayIndexOutOfBoundsException: -1
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:314)
> at
> org.apache.tez.runtime.library.common.writers.UnorderedPartitionedKVWriter.write(UnorderedPartitionedKVWriter.java:257)
> at
> org.apache.hadoop.hive.ql.exec.tez.TezProcessor$TezKVOutputCollector.collect(TezProcessor.java:253)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.collect(ReduceSinkOperator.java:552)
> at
> org.apache.hadoop.hive.ql.exec.ReduceSinkOperator.process(ReduceSinkOperator.java:398)
> ... 22 more
> ], TaskAttempt 1 failed, info=[Error: Error while running task ( failure ) :
> attempt_1466710232033_0539_6_00_00_1:java.lang.RuntimeException:
>
[jira] [Commented] (HIVE-13883) WebHCat leaves token crc file never gets deleted
[
https://issues.apache.org/jira/browse/HIVE-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380850#comment-15380850
]
Thejas M Nair commented on HIVE-13883:
--
Looks like the patch might need another rebase.
> WebHCat leaves token crc file never gets deleted
>
>
> Key: HIVE-13883
> URL: https://issues.apache.org/jira/browse/HIVE-13883
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 1.2.0, 1.1.1, 1.2.1, 2.0.1
>Reporter: niklaus xiao
>Priority: Minor
> Attachments: HIVE-13883.2.patch, HIVE-13883.patch
>
>
> In one of our long run environment, there are thousands of
> /tmp/.templeton*.tmp.crc files,
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc
> ...
> -rw-r--r-- 1 omm wheel 12 May 26 18:15
> /tmp/.templeton6676048390600607654.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:14
> /tmp/.templeton2733383617337556503.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:12
> /tmp/.templeton2183121761801669064.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:11
> /tmp/.templeton2689764046140543879.tmp.crc
> ...
> {quote}
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc | wc -l
> 17986
> {quote}
> It's created by webhcat,
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L193]
> and never gets deleted
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L110]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14239) Change OperationManager to serve each session and make it threadsafe
[
https://issues.apache.org/jira/browse/HIVE-14239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380848#comment-15380848
]
Hive QA commented on HIVE-14239:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818166/HIVE-14239.1.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 10326 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_12
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
org.apache.hive.jdbc.TestJdbcDriver2.testFetchFirstDfsCmds
org.apache.hive.jdbc.miniHS2.TestHiveServer2SessionTimeout.testConnection
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/539/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/539/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-539/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818166 - PreCommit-HIVE-MASTER-Build
> Change OperationManager to serve each session and make it threadsafe
>
>
> Key: HIVE-14239
> URL: https://issues.apache.org/jira/browse/HIVE-14239
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2
>Affects Versions: 2.0.1
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14239.1.patch
>
>
> Currently OperationManager is designed to hold all the operations from all
> the sessions. It will cause the functions to unnecessarily sync between the
> operations across the sessions. And also there are some functions are not
> threadsafe which could be the cause for invalid OperationHandle.
> Plan to change OperationManager to just serve one session and make it a
> helper class for HiveSession rather than a service which doesn't seem to be
> needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380755#comment-15380755 ] Xuefu Zhang commented on HIVE-14259: The original implementation seems more interesting and full of imagination. :) +1 on the patch > FileUtils.isSubDir may return incorrect result > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14198) Refactor aux jar related code to make them more consistent
[
https://issues.apache.org/jira/browse/HIVE-14198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380740#comment-15380740
]
Hive QA commented on HIVE-14198:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818214/HIVE-14198.2.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 15 failed/errored test(s), 10326 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_binary_map_queries_prefix
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_joins
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_null_first_col
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_queries
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_single_sourced_multi_insert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbasestats
org.apache.hadoop.hive.cli.TestHBaseMinimrCliDriver.testCliDriver_hbase_bulk
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/537/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/537/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-537/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 15 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818214 - PreCommit-HIVE-MASTER-Build
> Refactor aux jar related code to make them more consistent
> --
>
> Key: HIVE-14198
> URL: https://issues.apache.org/jira/browse/HIVE-14198
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Affects Versions: 2.2.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
> Attachments: HIVE-14198.1.patch, HIVE-14198.2.patch
>
>
> There are some redundancy and inconsistency between hive.aux.jar.paths and
> hive.reloadable.aux.jar.paths and also between MR and spark.
> Refactor the code to reuse the same code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14214) ORC Schema Evolution and Predicate Push Down do not work together (no rows returned)
[
https://issues.apache.org/jira/browse/HIVE-14214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380681#comment-15380681
]
Hive QA commented on HIVE-14214:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818138/HIVE-14214.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 101 failed/errored test(s), 10327 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_list_bucket_dml_13
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_parquet_join
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_stats_list_bucket
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver_subquery_multiinsert
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_join
org.apache.hadoop.hive.cli.TestHBaseCliDriver.testCliDriver_hbase_ppd_key_range
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_acid_globallimit
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_mergejoin
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge5
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge6
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge7
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_merge_incompat2
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_ppd_basic
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_orc_vectorization_ppd
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorization_pushdown
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_distinct_gby
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver_vectorized_dynamic_partition_pruning
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testCheckPermissions
org.apache.hadoop.hive.llap.daemon.impl.TestLlapTokenChecker.testGetToken
org.apache.hadoop.hive.metastore.TestMetaStoreMetrics.testConnections
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerialize
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderFooterSerializeWithDeltas
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerialize
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testACIDReaderNoFooterSerializeWithDeltas
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testDefaultTypes
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testDoAs
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testEmptyFile
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testInOutFormat
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testMROutput
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testNonVectorReaderFooterSerialize
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testNonVectorReaderNoFooterSerialize
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitElimination
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitEliminationNullStats
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOps
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCache
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeFileLen
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testSplitGenReadOpsLocalCacheChangeModificationTime
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorReaderFooterSerialize
org.apache.hadoop.hive.ql.io.orc.TestInputOutputFormat.testVectorReaderNoFooterSerialize
org.apache.hadoop.hive.ql.io.orc.TestNewInputOutputFormat.testNewInputFormat
org.apache.hadoop.hive.ql.io.orc.TestNewInputOutputFormat.testNewInputFormatPruning
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testRecordReaderDelta
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testRecordReaderIncompleteDelta
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testRecordReaderNewBaseAndDelta
org.apache.hadoop.hive.ql.io.orc.TestOrcRawRecordMerger.testRecordReaderOldBaseAndDelta
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testExternalFooterCache
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testExternalFooterCachePpd
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testSplitEliminationComplexExpr
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testSplitEliminationLargeMaxSplit
org.apache.hadoop.hive.ql.io.orc.TestOrcSplitElimination.testSplitEliminationSmallMaxSplit
org.apache.hive.hcatalog.mapreduce.TestHCatDynamicPartitioned.testHCatDynamicPartitionedTableMultipleTask[3]
[jira] [Updated] (HIVE-14205) Hive doesn't support union type with AVRO file format
[
https://issues.apache.org/jira/browse/HIVE-14205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yibing Shi updated HIVE-14205:
--
Attachment: HIVE-14205.2.patch
submit a new patch based on code review
> Hive doesn't support union type with AVRO file format
> -
>
> Key: HIVE-14205
> URL: https://issues.apache.org/jira/browse/HIVE-14205
> Project: Hive
> Issue Type: Bug
> Components: Serializers/Deserializers
>Reporter: Yibing Shi
>Assignee: Yibing Shi
> Attachments: HIVE-14205.1.patch, HIVE-14205.2.patch
>
>
> Reproduce steps:
> {noformat}
> hive> CREATE TABLE avro_union_test
> > PARTITIONED BY (p int)
> > ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> > STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> > OUTPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat'
> > TBLPROPERTIES ('avro.schema.literal'='{
> >"type":"record",
> >"name":"nullUnionTest",
> >"fields":[
> > {
> > "name":"value",
> > "type":[
> > "null",
> > "int",
> > "long"
> > ],
> > "default":null
> > }
> >]
> > }');
> OK
> Time taken: 0.105 seconds
> hive> alter table avro_union_test add partition (p=1);
> OK
> Time taken: 0.093 seconds
> hive> select * from avro_union_test;
> FAILED: RuntimeException org.apache.hadoop.hive.ql.metadata.HiveException:
> Failed with exception Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported
> yet.java.lang.RuntimeException: Hive internal error inside
> isAssignableFromSettablePrimitiveOI void not supported yet.
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1140)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1149)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1187)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1220)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1200)
> at
> org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:219)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.setupOutputObjectInspector(FetchOperator.java:581)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.initialize(FetchOperator.java:172)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.(FetchOperator.java:140)
> at
> org.apache.hadoop.hive.ql.exec.FetchTask.initialize(FetchTask.java:79)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:482)
> at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:311)
> at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1194)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1289)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1120)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1108)
> at
> org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:218)
> at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:170)
> at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:381)
> at
> org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:773)
> at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:691)
> at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:626)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> {noformat}
> Another test case to show this problem is:
> {noformat}
> hive> create table avro_union_test2 (value uniontype) stored as
> avro;
> OK
> Time taken: 0.053 seconds
> hive> show create table avro_union_test2;
> OK
> CREATE TABLE `avro_union_test2`(
> `value` uniontype COMMENT '')
> ROW FORMAT SERDE
> 'org.apache.hadoop.hive.serde2.avro.AvroSerDe'
> STORED AS INPUTFORMAT
> 'org.apache.hadoop.hive.ql.io.avro.AvroContainerInputFormat'
> OUTPUTFORMAT
>
[jira] [Commented] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380674#comment-15380674 ] Zoltan Haindrich commented on HIVE-14259: - i've changed the method to use the one I added in HIVE-12244; and i've added some further tests... > FileUtils.isSubDir may return incorrect result > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14259) FileUtils.isSubDir may return incorrect result
[ https://issues.apache.org/jira/browse/HIVE-14259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-14259: Attachment: HIVE-14259.1.patch > FileUtils.isSubDir may return incorrect result > -- > > Key: HIVE-14259 > URL: https://issues.apache.org/jira/browse/HIVE-14259 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Minor > Attachments: HIVE-14259.1.patch > > > while I was working on HIVE-12244 i've looked around for utility > methods...i've found this method; but it considers path: `/dir12` inside > `/dir1` > which is not true -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13883) WebHCat leaves token crc file never gets deleted
[
https://issues.apache.org/jira/browse/HIVE-13883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380646#comment-15380646
]
Hive QA commented on HIVE-13883:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12818124/HIVE-13883.2.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/535/testReport
Console output:
https://builds.apache.org/job/PreCommit-HIVE-MASTER-Build/535/console
Test logs:
http://ec2-204-236-174-241.us-west-1.compute.amazonaws.com/logs/PreCommit-HIVE-MASTER-Build-535/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hive-ptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ [[ -n /usr/java/jdk1.8.0_25 ]]
+ export JAVA_HOME=/usr/java/jdk1.8.0_25
+ JAVA_HOME=/usr/java/jdk1.8.0_25
+ export
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/java/jdk1.8.0_25/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'M2_OPTS=-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ M2_OPTS='-Xmx1g -XX:MaxPermSize=256m -Dhttp.proxyHost=localhost
-Dhttp.proxyPort=3128'
+ cd /data/hive-ptest/working/
+ tee /data/hive-ptest/logs/PreCommit-HIVE-MASTER-Build-535/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 0459768 HIVE-9756: LLAP: use log4j 2 for llap (log to separate
files, etc.) (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git clean -f -d
Removing
ql/src/test/org/apache/hadoop/hive/ql/exec/tez/TestCustomPartitionVertex.java
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 0459768 HIVE-9756: LLAP: use log4j 2 for llap (log to separate
files, etc.) (Prasanth Jayachandran reviewed by Siddharth Seth)
+ git merge --ff-only origin/master
Already up-to-date.
+ git gc
+ patchCommandPath=/data/hive-ptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hive-ptest/working/scratch/build.patch
+ [[ -f /data/hive-ptest/working/scratch/build.patch ]]
+ chmod +x /data/hive-ptest/working/scratch/smart-apply-patch.sh
+ /data/hive-ptest/working/scratch/smart-apply-patch.sh
/data/hive-ptest/working/scratch/build.patch
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12818124 - PreCommit-HIVE-MASTER-Build
> WebHCat leaves token crc file never gets deleted
>
>
> Key: HIVE-13883
> URL: https://issues.apache.org/jira/browse/HIVE-13883
> Project: Hive
> Issue Type: Bug
> Components: WebHCat
>Affects Versions: 1.2.0, 1.1.1, 1.2.1, 2.0.1
>Reporter: niklaus xiao
>Priority: Minor
> Attachments: HIVE-13883.2.patch, HIVE-13883.patch
>
>
> In one of our long run environment, there are thousands of
> /tmp/.templeton*.tmp.crc files,
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc
> ...
> -rw-r--r-- 1 omm wheel 12 May 26 18:15
> /tmp/.templeton6676048390600607654.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:14
> /tmp/.templeton2733383617337556503.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:12
> /tmp/.templeton2183121761801669064.tmp.crc
> -rw-r--r-- 1 omm wheel 12 May 26 18:11
> /tmp/.templeton2689764046140543879.tmp.crc
> ...
> {quote}
> {quote}
> omm@szxciitslx17645:/> ll /tmp/.templeton*.tmp.crc | wc -l
> 17986
> {quote}
> It's created by webhcat,
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L193]
> and never gets deleted
> [https://github.com/apache/hive/blob/master/hcatalog/webhcat/svr/src/main/java/org/apache/hive/hcatalog/templeton/SecureProxySupport.java#L110]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12244) Refactoring code for avoiding of comparison of Strings and do comparison on Path
[ https://issues.apache.org/jira/browse/HIVE-12244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380566#comment-15380566 ] Zoltan Haindrich commented on HIVE-12244: - [~ashutoshc] sure...I've created my first review request: https://reviews.apache.org/r/50104/ > Refactoring code for avoiding of comparison of Strings and do comparison on > Path > > > Key: HIVE-12244 > URL: https://issues.apache.org/jira/browse/HIVE-12244 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 0.13.0, 0.14.0, 1.0.0, 1.2.1 >Reporter: Alina Abramova >Assignee: Zoltan Haindrich >Priority: Minor > Labels: patch > Fix For: 1.2.1 > > Attachments: HIVE-12244.1.patch, HIVE-12244.10.patch, > HIVE-12244.11.patch, HIVE-12244.12.patch, HIVE-12244.2.patch, > HIVE-12244.3.patch, HIVE-12244.4.patch, HIVE-12244.5.patch, > HIVE-12244.6.patch, HIVE-12244.7.patch, HIVE-12244.8.patch, > HIVE-12244.8.patch, HIVE-12244.9.patch > > > In Hive often String is used for representation path and it causes new issues. > We need to compare it with equals() but comparing Strings often is not right > in terms comparing paths . > I think if we use Path from org.apache.hadoop.fs we will avoid new problems > in future. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14067) Rename pendingCount to activeCalls in HiveSessionImpl for easier understanding.
[ https://issues.apache.org/jira/browse/HIVE-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380547#comment-15380547 ] zhihai xu commented on HIVE-14067: -- Thanks for the review [~thejas]! The old patch can't be applied. I attached a new patch HIVE-14067.001.patch based on the latest code. > Rename pendingCount to activeCalls in HiveSessionImpl for easier > understanding. > > > Key: HIVE-14067 > URL: https://issues.apache.org/jira/browse/HIVE-14067 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Trivial > Attachments: HIVE-14067.000.patch, HIVE-14067.000.patch, > HIVE-14067.001.patch > > > Rename pendingCount to activeCalls in HiveSessionImpl for easier > understanding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14067) Rename pendingCount to activeCalls in HiveSessionImpl for easier understanding.
[ https://issues.apache.org/jira/browse/HIVE-14067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhihai xu updated HIVE-14067: - Attachment: HIVE-14067.001.patch > Rename pendingCount to activeCalls in HiveSessionImpl for easier > understanding. > > > Key: HIVE-14067 > URL: https://issues.apache.org/jira/browse/HIVE-14067 > Project: Hive > Issue Type: Improvement > Components: HiveServer2 >Reporter: zhihai xu >Assignee: zhihai xu >Priority: Trivial > Attachments: HIVE-14067.000.patch, HIVE-14067.000.patch, > HIVE-14067.001.patch > > > Rename pendingCount to activeCalls in HiveSessionImpl for easier > understanding. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
