[jira] [Updated] (HIVE-15161) migrate ColumnStats to use jackson
[ https://issues.apache.org/jira/browse/HIVE-15161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-15161: Attachment: (was: HIVE-15161.2.patch) > migrate ColumnStats to use jackson > -- > > Key: HIVE-15161 > URL: https://issues.apache.org/jira/browse/HIVE-15161 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 2.2.0 > > Attachments: HIVE-15161.1.patch, HIVE-15161.2.patch > > > * json.org has license issues > * jackson can provide a fully compatible alternative to it > * there are a few flakiness issues caused by the order of the map entries of > the columns...this cat be addressed, org.json api was unfriendly in this > manner ;) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15161) migrate ColumnStats to use jackson
[ https://issues.apache.org/jira/browse/HIVE-15161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-15161: Attachment: HIVE-15161.2.patch > migrate ColumnStats to use jackson > -- > > Key: HIVE-15161 > URL: https://issues.apache.org/jira/browse/HIVE-15161 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 2.2.0 > > Attachments: HIVE-15161.1.patch, HIVE-15161.2.patch, > HIVE-15161.2.patch > > > * json.org has license issues > * jackson can provide a fully compatible alternative to it > * there are a few flakiness issues caused by the order of the map entries of > the columns...this cat be addressed, org.json api was unfriendly in this > manner ;) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15161) migrate ColumnStats to use jackson
[ https://issues.apache.org/jira/browse/HIVE-15161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-15161: Attachment: HIVE-15161.2.patch #2 - fix bug in case params are null > migrate ColumnStats to use jackson > -- > > Key: HIVE-15161 > URL: https://issues.apache.org/jira/browse/HIVE-15161 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 2.2.0 > > Attachments: HIVE-15161.1.patch, HIVE-15161.2.patch > > > * json.org has license issues > * jackson can provide a fully compatible alternative to it > * there are a few flakiness issues caused by the order of the map entries of > the columns...this cat be addressed, org.json api was unfriendly in this > manner ;) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649767#comment-15649767
]
Hive QA commented on HIVE-14089:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838107/HIVE-14089.10.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_complex_join]
(batchId=143)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown]
(batchId=121)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2045/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2045/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2045/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838107 - PreCommit-HIVE-Build
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.04.patch, HIVE-14089.05.patch,
> HIVE-14089.06.patch, HIVE-14089.07.patch, HIVE-14089.08.patch,
> HIVE-14089.09.patch, HIVE-14089.10.patch, HIVE-14089.10.patch,
> HIVE-14089.10.patch, HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
> Note to self - need to add multi-stripe test, and also test complex types
> with some nulls so that present stream is not suppressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15164) Change default RPC port for llap to be a dynamic port
[
https://issues.apache.org/jira/browse/HIVE-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649694#comment-15649694
]
Hive QA commented on HIVE-15164:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838092/HIVE-15164.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 10632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown]
(batchId=121)
org.apache.hadoop.hive.llap.daemon.impl.TestLlapDaemonProtocolServerImpl.test
(batchId=277)
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testCreateTableLike (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testDropTableException (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testEmptyTableInstantiation
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testGetMessageBusTopicName
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testOtherFailure (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSchema (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testRenameTable (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testUpdateTableSchema (batchId=166)
org.apache.hive.hcatalog.pig.TestAvroHCatStorer.testWriteDecimal (batchId=170)
org.apache.hive.hcatalog.pig.TestSequenceFileHCatStorer.testWriteDecimal
(batchId=170)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2044/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2044/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2044/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838092 - PreCommit-HIVE-Build
> Change default RPC port for llap to be a dynamic port
> -
>
> Key: HIVE-15164
> URL: https://issues.apache.org/jira/browse/HIVE-15164
> Project: Hive
> Issue Type: Task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-15164.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15151) Bootstrap support for replv2
[
https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649623#comment-15649623
]
Hive QA commented on HIVE-15151:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838091/HIVE-15151.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 10634 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_1]
(batchId=90)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[exim_00_unsupported_schema]
(batchId=83)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown]
(batchId=121)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testTaskStatus
(batchId=207)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2043/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2043/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2043/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838091 - PreCommit-HIVE-Build
> Bootstrap support for replv2
>
>
> Key: HIVE-15151
> URL: https://issues.apache.org/jira/browse/HIVE-15151
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-15151.2.patch, HIVE-15151.addendum.patch,
> HIVE-15151.patch
>
>
> We need to support the ability to bootstrap an initial state, dumping out
> currently existing dbs/tables, etc, so that incremental replication can take
> over from that point. To this end, we should implement commands such as REPL
> DUMP, REPL LOAD, REPL STATUS, as described over at
> https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15139) HoS local mode fails with NumberFormatException
[
https://issues.apache.org/jira/browse/HIVE-15139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rui Li updated HIVE-15139:
--
Resolution: Fixed
Fix Version/s: 2.2.0
Status: Resolved (was: Patch Available)
Committed to master. Thanks [~ctang.ma] and [~xuefuz] for the review!
> HoS local mode fails with NumberFormatException
> ---
>
> Key: HIVE-15139
> URL: https://issues.apache.org/jira/browse/HIVE-15139
> Project: Hive
> Issue Type: Bug
>Reporter: Rui Li
>Assignee: Rui Li
> Fix For: 2.2.0
>
> Attachments: HIVE-15139.1.patch
>
>
> It's because we store {{stageId_attemptNum}} in JobMetricsListener but expect
> only {{stageId}} in LocalSparkJobStatus.
> {noformat}
> java.lang.NumberFormatException: For input string: "0_0"
> at
> java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
> at java.lang.Integer.parseInt(Integer.java:580)
> at java.lang.Integer.parseInt(Integer.java:615)
> at
> org.apache.hadoop.hive.ql.exec.spark.status.impl.LocalSparkJobStatus.getSparkStatistics(LocalSparkJobStatus.java:146)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkTask.execute(SparkTask.java:104)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:199)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Attachment: HIVE-14089.10.patch
Removing temporary logging; also fixing the explain-time derivation, some out
files will probably change. But it should be ready for review.
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.04.patch, HIVE-14089.05.patch,
> HIVE-14089.06.patch, HIVE-14089.07.patch, HIVE-14089.08.patch,
> HIVE-14089.09.patch, HIVE-14089.10.patch, HIVE-14089.10.patch,
> HIVE-14089.10.patch, HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
> Note to self - need to add multi-stripe test, and also test complex types
> with some nulls so that present stream is not suppressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15085) Reduce the memory used by unit tests, MiniCliDriver, MiniLlapLocal, MiniSpark
[
https://issues.apache.org/jira/browse/HIVE-15085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649524#comment-15649524
]
Hive QA commented on HIVE-15085:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12837816/HIVE-15085.02.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=133)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown]
(batchId=121)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2042/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2042/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2042/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12837816 - PreCommit-HIVE-Build
> Reduce the memory used by unit tests, MiniCliDriver, MiniLlapLocal, MiniSpark
> -
>
> Key: HIVE-15085
> URL: https://issues.apache.org/jira/browse/HIVE-15085
> Project: Hive
> Issue Type: Sub-task
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-15085.01.patch, HIVE-15085.02.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15167) undeprecate Deserializer, SerDe etc
[ https://issues.apache.org/jira/browse/HIVE-15167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649503#comment-15649503 ] Sergey Shelukhin commented on HIVE-15167: - [~namit] [~ashutoshc] fyi > undeprecate Deserializer, SerDe etc > --- > > Key: HIVE-15167 > URL: https://issues.apache.org/jira/browse/HIVE-15167 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin > > SerDe interfaces were deprecated in HIVE-4007 to suggest that users do not > implement them. However, this results in deprecation warnings all over the > codebase where they are actually used. > We should un-deprecate (reprecate? precate?) them. We can add a comment for > implementers instead (we could add a method with a clearly bogus name like > useThisAbstractClassInstead, and implement it in the class, so it would be > noticeable, but that would break compat). -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HIVE-15011) fix issues with MoveTask.releaseLocks()
[ https://issues.apache.org/jira/browse/HIVE-15011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-15011. --- Resolution: Duplicate Turns out that this is not an issue for Acid code path (doesn't apply). Added 1 line to patch 10 in HIVE-14943 to make that logic clear > fix issues with MoveTask.releaseLocks() > --- > > Key: HIVE-15011 > URL: https://issues.apache.org/jira/browse/HIVE-15011 > Project: Hive > Issue Type: Sub-task > Components: Query Planning, Query Processor, Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > in Merge we can have multiple move tasks so releasing the locks should be > done on from the "last one" - in practice they run concurrently > see if there is a quick fix for short term > (slightly) longer term - get rid of MoveTask for Acid writes -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649429#comment-15649429
]
Hive QA commented on HIVE-14089:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838083/HIVE-14089.10.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 10632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown]
(batchId=121)
org.apache.hive.hcatalog.api.TestHCatClient.testBasicDDLCommands (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testCreateTableLike (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testDatabaseLocation (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testDropPartitionsWithPartialSpec
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testDropTableException (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testEmptyTableInstantiation
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testGetMessageBusTopicName
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testGetPartitionsWithPartialSpec
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testObjectNotFoundException
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testOtherFailure (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionRegistrationWithCustomSchema
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSchema (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionSpecRegistrationWithCustomSchema
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testPartitionsHCatClientImpl
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testRenameTable (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testReplicationTaskIter
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testTableSchemaPropagation
(batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testTransportFailure (batchId=166)
org.apache.hive.hcatalog.api.TestHCatClient.testUpdateTableSchema (batchId=166)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2041/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2041/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2041/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838083 - PreCommit-HIVE-Build
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.04.patch, HIVE-14089.05.patch,
> HIVE-14089.06.patch, HIVE-14089.07.patch, HIVE-14089.08.patch,
> HIVE-14089.09.patch, HIVE-14089.10.patch, HIVE-14089.10.patch,
> HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch, HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
> Note to self - need to add multi-stripe test, and also test complex types
> with some nulls so that present stream is not suppressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14271) FileSinkOperator should not rename files to final paths when S3 is the default destination
[
https://issues.apache.org/jira/browse/HIVE-14271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649356#comment-15649356
]
Sahil Takiar commented on HIVE-14271:
-
We might want to consider re-opening this ticket, but changing the original
approach. To clarify, right now the FileSinkOperator (FSOP) will always write
all its data to a scratch directory. The FSOP first writes to a {{outPaths}}
and then renames the data to {{finalPaths}}, but all the data is still under
the scratch directory. No data is exposed to users or future ETL jobs yet.
There are two different ways to modify this to improve performance on S3:
1: FSOP implements the "direct output committer" strategy (similar to
HIVE-1620) and all data is written directly to the final table location, no
data is written to a staging file or in the scratch directory. Hive's MoveTask
(which runs in HiveServer2) does nothing.
2: FSOP writes data to a scratch directory, but it doesn't write to
{{outPaths}} it writes to {{finalPaths}} instead (remember both of these
directories are still under the scratch directory). Hive's MoveTask (which runs
inside HiveServer2) copies the data from the scratch directory to the final
table location. The FSOP writes directly to the final location in the scratch
directory, no writing to a temp file is done. This improves performance since
it avoids copying data from {{outPaths}} to {{finalPaths}}.
For reasons stated in earlier comments, there are a number of issues with
approach 1. Implementing approach 2 should be better, and should improve
performance significantly.
> FileSinkOperator should not rename files to final paths when S3 is the
> default destination
> --
>
> Key: HIVE-14271
> URL: https://issues.apache.org/jira/browse/HIVE-14271
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergio Peña
>Assignee: Sergio Peña
>
> FileSinkOperator does a rename of {{outPaths -> finalPaths}} when it finished
> writing all rows to a temporary path. The problem is that S3 does not support
> renaming.
> Two options can be considered:
> a. Use a copy operation instead. After FileSinkOperator writes all rows to
> outPaths, then the commit method will do a copy() call instead of move().
> b. Write row by row directly to the S3 path (see HIVE-1620). This may add
> better performance calls, but we should take care of the cleanup part in case
> of writing errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15164) Change default RPC port for llap to be a dynamic port
[ https://issues.apache.org/jira/browse/HIVE-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-15164: -- Status: Patch Available (was: Open) > Change default RPC port for llap to be a dynamic port > - > > Key: HIVE-15164 > URL: https://issues.apache.org/jira/browse/HIVE-15164 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-15164.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15164) Change default RPC port for llap to be a dynamic port
[ https://issues.apache.org/jira/browse/HIVE-15164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-15164: -- Attachment: HIVE-15164.01.patch cc [~sershe], [~gopalv] for review. > Change default RPC port for llap to be a dynamic port > - > > Key: HIVE-15164 > URL: https://issues.apache.org/jira/browse/HIVE-15164 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-15164.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15151: Attachment: (was: HIVE-15152.2.patch) > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-15151.2.patch, HIVE-15151.addendum.patch, > HIVE-15151.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-15151: --- Assignee: Vaibhav Gumashta (was: Sushanth Sowmyan) > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Vaibhav Gumashta > Attachments: HIVE-15151.2.patch, HIVE-15151.addendum.patch, > HIVE-15151.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15151) Bootstrap support for replv2
[
https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649316#comment-15649316
]
Hive QA commented on HIVE-15151:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838078/HIVE-15151.addendum.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2040/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2040/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2040/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2016-11-09 00:22:50.502
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-2040/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2016-11-09 00:22:50.505
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
4aae400..9de529a master -> origin/master
fd55bd3..6b64b98 branch-1 -> origin/branch-1
+ git reset --hard HEAD
HEAD is now at 4aae400 HIVE-15023: SimpleFetchOptimizer needs to optimize
limit=0 (Pengcheng Xiong, reviewed by Ashutosh Chauhan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 3 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 9de529a HIVE-15159. LLAP ContainerRunner should not reduce the
available heap while distributing to individual executors. (Siddharth Seth,
reviewed by Gopal V)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2016-11-09 00:22:52.215
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/ql/src/java/org/apache/hadoop/hive/ql/parse/ReplicationSemanticAnalyzer.java:
No such file or directory
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838078 - PreCommit-HIVE-Build
> Bootstrap support for replv2
>
>
> Key: HIVE-15151
> URL: https://issues.apache.org/jira/browse/HIVE-15151
> Project: Hive
> Issue Type: Sub-task
> Components: repl
>Reporter: Sushanth Sowmyan
>Assignee: Sushanth Sowmyan
> Attachments: HIVE-15151.2.patch, HIVE-15151.addendum.patch,
> HIVE-15151.patch
>
>
> We need to support the ability to bootstrap an initial state, dumping out
> currently existing dbs/tables, etc, so that incremental replication can take
> over from that point. To this end, we should implement commands such as REPL
> DUMP, REPL LOAD, REPL STATUS, as described over at
> https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15151: Attachment: HIVE-15151.2.patch > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Vaibhav Gumashta > Attachments: HIVE-15151.2.patch, HIVE-15151.addendum.patch, > HIVE-15151.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15151: Assignee: Sushanth Sowmyan (was: Vaibhav Gumashta) > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-15151.2.patch, HIVE-15151.addendum.patch, > HIVE-15151.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15151: Attachment: HIVE-15152.2.patch Adding my changes to patch 1. > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Vaibhav Gumashta > Attachments: HIVE-15151.addendum.patch, HIVE-15151.patch, > HIVE-15152.2.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15151: Assignee: Sushanth Sowmyan (was: Vaibhav Gumashta) > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-15151.addendum.patch, HIVE-15151.patch, > HIVE-15152.2.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15159) LLAP ContainerRunner should not reduce the available heap while distributing to individual executors
[ https://issues.apache.org/jira/browse/HIVE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-15159: -- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to master. > LLAP ContainerRunner should not reduce the available heap while distributing > to individual executors > > > Key: HIVE-15159 > URL: https://issues.apache.org/jira/browse/HIVE-15159 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Fix For: 2.2.0 > > Attachments: HIVE-15159.01.patch > > > hive.llap.daemon.memory.per.instance.mb is set to the heapsize of the process > when llap service driver is used. > ContainerRunner currently cuts this down to 80% of the available. This > reduction is not necessary if the heapsize is used, instead of a container > size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-15151: --- Assignee: Vaibhav Gumashta (was: Sushanth Sowmyan) > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Vaibhav Gumashta > Attachments: HIVE-15151.addendum.patch, HIVE-15151.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15162) NPE in ATSHook
[
https://issues.apache.org/jira/browse/HIVE-15162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649297#comment-15649297
]
Hive QA commented on HIVE-15162:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838073/HIVE-15162.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 10632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=91)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[limit_pushdown]
(batchId=121)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2038/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2038/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2038/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838073 - PreCommit-HIVE-Build
> NPE in ATSHook
> --
>
> Key: HIVE-15162
> URL: https://issues.apache.org/jira/browse/HIVE-15162
> Project: Hive
> Issue Type: Bug
> Components: Hooks
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-15162.1.patch
>
>
> {noformat}
> 2016-11-08T14:21:15,025 INFO [ATS Logger 0]: hooks.ATSHook
> (ATSHook.java:run(156)) - Failed to submit plan to ATS:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:141)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15159) LLAP ContainerRunner should not reduce the available heap while distributing to individual executors
[ https://issues.apache.org/jira/browse/HIVE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649298#comment-15649298 ] Siddharth Seth commented on HIVE-15159: --- Flaky tests: HIVE-15116, HIVE-15115, HIVE-15084, HIVE-15165, HIVE-14910 > LLAP ContainerRunner should not reduce the available heap while distributing > to individual executors > > > Key: HIVE-15159 > URL: https://issues.apache.org/jira/browse/HIVE-15159 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-15159.01.patch > > > hive.llap.daemon.memory.per.instance.mb is set to the heapsize of the process > when llap service driver is used. > ContainerRunner currently cuts this down to 80% of the available. This > reduction is not necessary if the heapsize is used, instead of a container > size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14943) Base Implementation
[ https://issues.apache.org/jira/browse/HIVE-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649264#comment-15649264 ] Eugene Koifman commented on HIVE-14943: --- explainanalyze_2 is flaky: HIVE-15084, other failures have age > 1 > Base Implementation > --- > > Key: HIVE-14943 > URL: https://issues.apache.org/jira/browse/HIVE-14943 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14943.10.patch, HIVE-14943.2.patch, > HIVE-14943.3.patch, HIVE-14943.4.patch, HIVE-14943.5.patch, > HIVE-14943.6.patch, HIVE-14943.7.patch, HIVE-14943.8.patch, > HIVE-14943.9.patch, HIVE-14943.patch > > > Create the 1st pass functional implementation of MERGE > This should run e2e and produce correct results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15166) Provide beeline option to set the jline history max size
[ https://issues.apache.org/jira/browse/HIVE-15166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Lin updated HIVE-15166: Priority: Minor (was: Major) > Provide beeline option to set the jline history max size > > > Key: HIVE-15166 > URL: https://issues.apache.org/jira/browse/HIVE-15166 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.1.0 >Reporter: Eric Lin >Assignee: Eric Lin >Priority: Minor > > Currently Beeline does not provide an option to limit the max size for > beeline history file, in the case that each query is very big, it will flood > the history file and slow down beeline on start up and shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Attachment: HIVE-14089.10.patch
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.04.patch, HIVE-14089.05.patch,
> HIVE-14089.06.patch, HIVE-14089.07.patch, HIVE-14089.08.patch,
> HIVE-14089.09.patch, HIVE-14089.10.patch, HIVE-14089.10.patch,
> HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch, HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
> Note to self - need to add multi-stripe test, and also test complex types
> with some nulls so that present stream is not suppressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15166) Provide beeline option to set the jline history max size
[ https://issues.apache.org/jira/browse/HIVE-15166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Lin updated HIVE-15166: Assignee: Eric Lin > Provide beeline option to set the jline history max size > > > Key: HIVE-15166 > URL: https://issues.apache.org/jira/browse/HIVE-15166 > Project: Hive > Issue Type: Improvement > Components: Beeline >Affects Versions: 2.1.0 >Reporter: Eric Lin >Assignee: Eric Lin > > Currently Beeline does not provide an option to limit the max size for > beeline history file, in the case that each query is very big, it will flood > the history file and slow down beeline on start up and shutdown. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Attachment: HIVE-14089.10.patch
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.04.patch, HIVE-14089.05.patch,
> HIVE-14089.06.patch, HIVE-14089.07.patch, HIVE-14089.08.patch,
> HIVE-14089.09.patch, HIVE-14089.10.patch, HIVE-14089.WIP.2.patch,
> HIVE-14089.WIP.3.patch, HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
> Note to self - need to add multi-stripe test, and also test complex types
> with some nulls so that present stream is not suppressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15084) Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4,5
[ https://issues.apache.org/jira/browse/HIVE-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-15084: -- Summary: Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4,5 (was: Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4) > Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4,5 > - > > Key: HIVE-15084 > URL: https://issues.apache.org/jira/browse/HIVE-15084 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15084) Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4, 5
[ https://issues.apache.org/jira/browse/HIVE-15084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-15084: -- Summary: Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4, 5 (was: Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4,5) > Flaky test: TestMiniTezCliDriver:explainanalyze_2, 3, 4, 5 > -- > > Key: HIVE-15084 > URL: https://issues.apache.org/jira/browse/HIVE-15084 > Project: Hive > Issue Type: Sub-task >Reporter: Siddharth Seth > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14910) Flaky test: TestSparkClient.testJobSubmission
[ https://issues.apache.org/jira/browse/HIVE-14910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-14910: -- Issue Type: Task (was: Sub-task) Parent: (was: HIVE-15058) > Flaky test: TestSparkClient.testJobSubmission > - > > Key: HIVE-14910 > URL: https://issues.apache.org/jira/browse/HIVE-14910 > Project: Hive > Issue Type: Task >Reporter: Siddharth Seth >Assignee: Barna Zsombor Klara > Fix For: 2.2.0 > > Attachments: HIVE-14910.1.patch, HIVE-14910.2.patch, > HIVE-14910.3.patch, HIVE-14910.patch > > > Have seen this fail in multiple runs (not consistently) > e.g. https://builds.apache.org/job/PreCommit-HIVE-Build/1426/testReport/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15120) Storage based auth: allow option to enforce write checks for external tables
[ https://issues.apache.org/jira/browse/HIVE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15120: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.2.0 1.3.0 Status: Resolved (was: Patch Available) Patch pushed to master and branch-1. > Storage based auth: allow option to enforce write checks for external tables > > > Key: HIVE-15120 > URL: https://issues.apache.org/jira/browse/HIVE-15120 > Project: Hive > Issue Type: Bug > Components: Authorization >Reporter: Thejas M Nair >Assignee: Daniel Dai > Fix For: 1.3.0, 2.2.0 > > Attachments: HIVE-15120.1.patch, HIVE-15120.2.patch, > HIVE-15120.3.patch, HIVE-15120.4.patch > > > Under storage based authorization, we don't require write permissions on > table directory for external table create/drop. > This is because external table contents are populated often from outside of > hive and are not written into from hive. So write access is not needed. Also, > we can't require write permissions to drop a table if we don't require them > for creation (users who created them should be able to drop them). > However, this difference in behavior of external tables is not well > documented. So users get surprised to learn that drop table can be done by > just any user who has read access to the directory. At that point changing > the large number of scripts that use external tables is hard. > It would be good to have a user config option to have external tables to be > treated same as managed tables. > The option should be off by default, so that the behavior is backward > compatible by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15163) LLAP Finishable state updates may not go to daemons restarted on the same node
[ https://issues.apache.org/jira/browse/HIVE-15163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649235#comment-15649235 ] Siddharth Seth commented on HIVE-15163: --- Finishable updates are sent out at the vertex level, instead of the fragment level. > LLAP Finishable state updates may not go to daemons restarted on the same node > -- > > Key: HIVE-15163 > URL: https://issues.apache.org/jira/browse/HIVE-15163 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth > > Finishable state updates only go out to nodes when the source state changes. > When a node goes down, and comes back up on the same port - updates which had > been sent to the old node will not be sent to the new node. This could lead > to a fragment being in a permanent preemptable state, and lead tp starvation > on busy clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15163) LLAP Finishable state updates may not go to daemons restarted on the same node
[ https://issues.apache.org/jira/browse/HIVE-15163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649230#comment-15649230 ] Siddharth Seth commented on HIVE-15163: --- Using dynamic ports will avoid this to a large extent. The actual fix is likely to required a change in Tez to propagate additional node information from the scheduler to the task communicator, and the task communicator to watch for node failure updates. https://issues.apache.org/jira/browse/HIVE-15164 changes the default port. > LLAP Finishable state updates may not go to daemons restarted on the same node > -- > > Key: HIVE-15163 > URL: https://issues.apache.org/jira/browse/HIVE-15163 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth > > Finishable state updates only go out to nodes when the source state changes. > When a node goes down, and comes back up on the same port - updates which had > been sent to the old node will not be sent to the new node. This could lead > to a fragment being in a permanent preemptable state, and lead tp starvation > on busy clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-14992) Relocate several common libraries in hive jdbc uber jar
[ https://issues.apache.org/jira/browse/HIVE-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-14992: - Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Patch committed to master. Thanks [~taoli-hwx]! > Relocate several common libraries in hive jdbc uber jar > --- > > Key: HIVE-14992 > URL: https://issues.apache.org/jira/browse/HIVE-14992 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Fix For: 2.2.0 > > Attachments: HIVE-14992.1.patch, HIVE-14992.2.patch, > HIVE-14992.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15163) LLAP Finishable state updates may not go to daemons restarted on the same node
[ https://issues.apache.org/jira/browse/HIVE-15163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649220#comment-15649220 ] Sergey Shelukhin commented on HIVE-15163: - Wouldn't the new node not have the fragment to start with? So, either it has both the fragment and then finishable update, or neither? > LLAP Finishable state updates may not go to daemons restarted on the same node > -- > > Key: HIVE-15163 > URL: https://issues.apache.org/jira/browse/HIVE-15163 > Project: Hive > Issue Type: Bug >Reporter: Siddharth Seth > > Finishable state updates only go out to nodes when the source state changes. > When a node goes down, and comes back up on the same port - updates which had > been sent to the old node will not be sent to the new node. This could lead > to a fragment being in a permanent preemptable state, and lead tp starvation > on busy clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14992) Relocate several common libraries in hive jdbc uber jar
[ https://issues.apache.org/jira/browse/HIVE-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649213#comment-15649213 ] Thejas M Nair commented on HIVE-14992: -- The two test failures are unrelated - https://issues.apache.org/jira/browse/HIVE-15058 > Relocate several common libraries in hive jdbc uber jar > --- > > Key: HIVE-14992 > URL: https://issues.apache.org/jira/browse/HIVE-14992 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14992.1.patch, HIVE-14992.2.patch, > HIVE-14992.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15151: Assignee: Sushanth Sowmyan (was: Vaibhav Gumashta) > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Sushanth Sowmyan > Attachments: HIVE-15151.addendum.patch, HIVE-15151.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-15151: Attachment: HIVE-15151.addendum.patch > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Vaibhav Gumashta > Attachments: HIVE-15151.addendum.patch, HIVE-15151.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-15151) Bootstrap support for replv2
[ https://issues.apache.org/jira/browse/HIVE-15151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta reassigned HIVE-15151: --- Assignee: Vaibhav Gumashta (was: Sushanth Sowmyan) > Bootstrap support for replv2 > > > Key: HIVE-15151 > URL: https://issues.apache.org/jira/browse/HIVE-15151 > Project: Hive > Issue Type: Sub-task > Components: repl >Reporter: Sushanth Sowmyan >Assignee: Vaibhav Gumashta > Attachments: HIVE-15151.addendum.patch, HIVE-15151.patch > > > We need to support the ability to bootstrap an initial state, dumping out > currently existing dbs/tables, etc, so that incremental replication can take > over from that point. To this end, we should implement commands such as REPL > DUMP, REPL LOAD, REPL STATUS, as described over at > https://cwiki.apache.org/confluence/display/Hive/HiveReplicationv2Development -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14992) Relocate several common libraries in hive jdbc uber jar
[
https://issues.apache.org/jira/browse/HIVE-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649188#comment-15649188
]
Hive QA commented on HIVE-14992:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12835883/HIVE-14992.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 10632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2037/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2037/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2037/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12835883 - PreCommit-HIVE-Build
> Relocate several common libraries in hive jdbc uber jar
> ---
>
> Key: HIVE-14992
> URL: https://issues.apache.org/jira/browse/HIVE-14992
> Project: Hive
> Issue Type: Bug
>Reporter: Tao Li
>Assignee: Tao Li
> Attachments: HIVE-14992.1.patch, HIVE-14992.2.patch,
> HIVE-14992.3.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15120) Storage based auth: allow option to enforce write checks for external tables
[ https://issues.apache.org/jira/browse/HIVE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daniel Dai updated HIVE-15120: -- Attachment: HIVE-15120.4.patch Including spelling changes from Lefty. > Storage based auth: allow option to enforce write checks for external tables > > > Key: HIVE-15120 > URL: https://issues.apache.org/jira/browse/HIVE-15120 > Project: Hive > Issue Type: Bug > Components: Authorization >Reporter: Thejas M Nair >Assignee: Daniel Dai > Attachments: HIVE-15120.1.patch, HIVE-15120.2.patch, > HIVE-15120.3.patch, HIVE-15120.4.patch > > > Under storage based authorization, we don't require write permissions on > table directory for external table create/drop. > This is because external table contents are populated often from outside of > hive and are not written into from hive. So write access is not needed. Also, > we can't require write permissions to drop a table if we don't require them > for creation (users who created them should be able to drop them). > However, this difference in behavior of external tables is not well > documented. So users get surprised to learn that drop table can be done by > just any user who has read access to the directory. At that point changing > the large number of scripts that use external tables is hard. > It would be good to have a user config option to have external tables to be > treated same as managed tables. > The option should be off by default, so that the behavior is backward > compatible by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15120) Storage based auth: allow option to enforce write checks for external tables
[ https://issues.apache.org/jira/browse/HIVE-15120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649174#comment-15649174 ] Thejas M Nair commented on HIVE-15120: -- Both test failures are tracked in HIVE-15058 . > Storage based auth: allow option to enforce write checks for external tables > > > Key: HIVE-15120 > URL: https://issues.apache.org/jira/browse/HIVE-15120 > Project: Hive > Issue Type: Bug > Components: Authorization >Reporter: Thejas M Nair >Assignee: Daniel Dai > Attachments: HIVE-15120.1.patch, HIVE-15120.2.patch, > HIVE-15120.3.patch > > > Under storage based authorization, we don't require write permissions on > table directory for external table create/drop. > This is because external table contents are populated often from outside of > hive and are not written into from hive. So write access is not needed. Also, > we can't require write permissions to drop a table if we don't require them > for creation (users who created them should be able to drop them). > However, this difference in behavior of external tables is not well > documented. So users get surprised to learn that drop table can be done by > just any user who has read access to the directory. At that point changing > the large number of scripts that use external tables is hard. > It would be good to have a user config option to have external tables to be > treated same as managed tables. > The option should be off by default, so that the behavior is backward > compatible by default. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15162) NPE in ATSHook
[
https://issues.apache.org/jira/browse/HIVE-15162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-15162:
--
Attachment: HIVE-15162.1.patch
> NPE in ATSHook
> --
>
> Key: HIVE-15162
> URL: https://issues.apache.org/jira/browse/HIVE-15162
> Project: Hive
> Issue Type: Bug
> Components: Hooks
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-15162.1.patch
>
>
> {noformat}
> 2016-11-08T14:21:15,025 INFO [ATS Logger 0]: hooks.ATSHook
> (ATSHook.java:run(156)) - Failed to submit plan to ATS:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:141)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15162) NPE in ATSHook
[
https://issues.apache.org/jira/browse/HIVE-15162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-15162:
--
Status: Patch Available (was: Open)
> NPE in ATSHook
> --
>
> Key: HIVE-15162
> URL: https://issues.apache.org/jira/browse/HIVE-15162
> Project: Hive
> Issue Type: Bug
> Components: Hooks
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-15162.1.patch
>
>
> {noformat}
> 2016-11-08T14:21:15,025 INFO [ATS Logger 0]: hooks.ATSHook
> (ATSHook.java:run(156)) - Failed to submit plan to ATS:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:141)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15140) Fix logger name in PartitionColumnsSeparator
[ https://issues.apache.org/jira/browse/HIVE-15140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chao Sun updated HIVE-15140: Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Committed to the master branch. Thanks [~tzenmyo]. > Fix logger name in PartitionColumnsSeparator > > > Key: HIVE-15140 > URL: https://issues.apache.org/jira/browse/HIVE-15140 > Project: Hive > Issue Type: Bug >Reporter: Teruyoshi Zenmyo >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15140.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15023) SimpleFetchOptimizer needs to optimize limit=0
[
https://issues.apache.org/jira/browse/HIVE-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15023:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> SimpleFetchOptimizer needs to optimize limit=0
> --
>
> Key: HIVE-15023
> URL: https://issues.apache.org/jira/browse/HIVE-15023
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15023.01.patch, HIVE-15023.02.patch
>
>
> on current master
> {code}
> hive> explain select key from src limit 0;
> OK
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 0
> Processor Tree:
> TableScan
> alias: src
> Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: key (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE
> Column stats: NONE
> Limit
> Number of rows: 0
> Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
> stats: NONE
> ListSink
> Time taken: 7.534 seconds, Fetched: 20 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15023) SimpleFetchOptimizer needs to optimize limit=0
[
https://issues.apache.org/jira/browse/HIVE-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15023:
---
Affects Version/s: 2.1.0
> SimpleFetchOptimizer needs to optimize limit=0
> --
>
> Key: HIVE-15023
> URL: https://issues.apache.org/jira/browse/HIVE-15023
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15023.01.patch, HIVE-15023.02.patch
>
>
> on current master
> {code}
> hive> explain select key from src limit 0;
> OK
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 0
> Processor Tree:
> TableScan
> alias: src
> Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: key (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE
> Column stats: NONE
> Limit
> Number of rows: 0
> Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
> stats: NONE
> ListSink
> Time taken: 7.534 seconds, Fetched: 20 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15023) SimpleFetchOptimizer needs to optimize limit=0
[
https://issues.apache.org/jira/browse/HIVE-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15023:
---
Fix Version/s: 2.2.0
> SimpleFetchOptimizer needs to optimize limit=0
> --
>
> Key: HIVE-15023
> URL: https://issues.apache.org/jira/browse/HIVE-15023
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15023.01.patch, HIVE-15023.02.patch
>
>
> on current master
> {code}
> hive> explain select key from src limit 0;
> OK
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 0
> Processor Tree:
> TableScan
> alias: src
> Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: key (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE
> Column stats: NONE
> Limit
> Number of rows: 0
> Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
> stats: NONE
> ListSink
> Time taken: 7.534 seconds, Fetched: 20 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15023) SimpleFetchOptimizer needs to optimize limit=0
[
https://issues.apache.org/jira/browse/HIVE-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649092#comment-15649092
]
Pengcheng Xiong commented on HIVE-15023:
pushed to master. Thanks [~ashutoshc] for the review.
> SimpleFetchOptimizer needs to optimize limit=0
> --
>
> Key: HIVE-15023
> URL: https://issues.apache.org/jira/browse/HIVE-15023
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.1.0
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
> Fix For: 2.2.0
>
> Attachments: HIVE-15023.01.patch, HIVE-15023.02.patch
>
>
> on current master
> {code}
> hive> explain select key from src limit 0;
> OK
> STAGE DEPENDENCIES:
> Stage-0 is a root stage
> STAGE PLANS:
> Stage: Stage-0
> Fetch Operator
> limit: 0
> Processor Tree:
> TableScan
> alias: src
> Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE
> Column stats: NONE
> Select Operator
> expressions: key (type: string)
> outputColumnNames: _col0
> Statistics: Num rows: 500 Data size: 5312 Basic stats: COMPLETE
> Column stats: NONE
> Limit
> Number of rows: 0
> Statistics: Num rows: 0 Data size: 0 Basic stats: NONE Column
> stats: NONE
> ListSink
> Time taken: 7.534 seconds, Fetched: 20 row(s)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15159) LLAP ContainerRunner should not reduce the available heap while distributing to individual executors
[ https://issues.apache.org/jira/browse/HIVE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649089#comment-15649089 ] Gopal V commented on HIVE-15159: LGTM - +1 > LLAP ContainerRunner should not reduce the available heap while distributing > to individual executors > > > Key: HIVE-15159 > URL: https://issues.apache.org/jira/browse/HIVE-15159 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-15159.01.patch > > > hive.llap.daemon.memory.per.instance.mb is set to the heapsize of the process > when llap service driver is used. > ContainerRunner currently cuts this down to 80% of the available. This > reduction is not necessary if the heapsize is used, instead of a container > size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13120) propagate doAs when generating ORC splits
[
https://issues.apache.org/jira/browse/HIVE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649086#comment-15649086
]
Chris Drome commented on HIVE-13120:
Thanks Sergey.
> propagate doAs when generating ORC splits
> -
>
> Key: HIVE-13120
> URL: https://issues.apache.org/jira/browse/HIVE-13120
> Project: Hive
> Issue Type: Improvement
>Reporter: Yi Zhang
>Assignee: Sergey Shelukhin
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HIVE-13120-branch-1.WIP.patch, HIVE-13120.patch
>
>
> ORC+HS2+doAs+FetchTask conversion = weird permission errors, e.g.
> {noformat}
> 2016-02-22 17:24:39,005 WARN [HiveServer2-Handler-Pool: Thread-587]:
> thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(681)) - Error
> fetching results:
> org.apache.hive.service.cli.HiveSQLException: java.io.IOException:
> java.lang.RuntimeException: serious problem
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:352)
> [snip]
> Caused by: java.io.IOException: java.lang.RuntimeException: serious problem
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1720)
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:347)
> ... 24 more
> Caused by: java.lang.RuntimeException: serious problem
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1059)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1086)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:363)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:295)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> ... 28 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=[snip], access=READ_EXECUTE, inode=[snip]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15076) Improve scalability of LDAP authentication provider group filter
[ https://issues.apache.org/jira/browse/HIVE-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649058#comment-15649058 ] Illya Yalovyy commented on HIVE-15076: -- [~aihuaxu], [~ashutoshc], [~ctang.ma], [~szehon], Could you please take a look at this CR? https://reviews.apache.org/r/53204/ > Improve scalability of LDAP authentication provider group filter > > > Key: HIVE-15076 > URL: https://issues.apache.org/jira/browse/HIVE-15076 > Project: Hive > Issue Type: Improvement > Components: Authentication >Affects Versions: 2.1.0 >Reporter: Illya Yalovyy >Assignee: Illya Yalovyy > Attachments: HIVE-15076.1.patch, HIVE-15076.2.patch > > > Current implementation uses following algorithm: > # For a given user find all groups that user is a member of. (A list of > LDAP groups is constructed as a result of that request) > # Match this list of groups with provided group filter. > > Time/Memory complexity of this approach is O(N) on client side, where N – is > a number of groups the user has membership in. On a large directory (800+ > groups per user) we can observe up to 2x performance degradation and failures > because of size of LDAP response (LDAP: error code 4 - Sizelimit Exceeded). > > Some Directory Services (Microsoft Active Directory for instance) provide a > virtual attribute for User Object that contains a list of groups that user > belongs to. This attribute can be used to quickly determine whether this user > passes or fails the group filter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15155) Change Hive version shortname to 2.2.0
[ https://issues.apache.org/jira/browse/HIVE-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-15155: --- Resolution: Fixed Fix Version/s: 2.2.0 Status: Resolved (was: Patch Available) Pushed to master, thanks for reviewing [~ashutoshc]! > Change Hive version shortname to 2.2.0 > -- > > Key: HIVE-15155 > URL: https://issues.apache.org/jira/browse/HIVE-15155 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Fix For: 2.2.0 > > Attachments: HIVE-15155.patch > > > Pointing to 2.1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-13452) StatsOptimizer should return no rows on empty table with group by
[
https://issues.apache.org/jira/browse/HIVE-13452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649034#comment-15649034
]
Vineet Garg commented on HIVE-13452:
Another example
{code} select (1=1) from t2 group by (1=1); {code}
This returns 1 row irrespective of statsoptimizer.
> StatsOptimizer should return no rows on empty table with group by
> -
>
> Key: HIVE-13452
> URL: https://issues.apache.org/jira/browse/HIVE-13452
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Reporter: Ashutosh Chauhan
>Assignee: Pengcheng Xiong
>
> {code}
> create table t1 (a int);
> analyze table t1 compute statistics;
> analyze table t1 compute statistics for columns;
> select count(1) from t1 group by 1;
> set hive.compute.query.using.stats=true;
> select count(1) from t1 group by 1;
> {code}
> In both cases result set should be empty. However, with statsoptimizer on
> Hive returns one row with value 0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15161) migrate ColumnStats to use jackson
[
https://issues.apache.org/jira/browse/HIVE-15161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649023#comment-15649023
]
Hive QA commented on HIVE-15161:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838042/HIVE-15161.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 10632 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=14)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=11)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats12] (batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats13] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats2] (batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats6] (batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats7] (batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats8] (batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_noscan_1]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_partscan_1_23]
(batchId=79)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_15]
(batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_16]
(batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_17]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_18]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_remove_25]
(batchId=79)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[autoColumnStats_2]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnStatsUpdateForStatsOptimizer_1]
(batchId=142)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_analyze]
(batchId=138)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[stats_noscan_1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=90)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_partialscan_autogether]
(batchId=83)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats12]
(batchId=118)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats13]
(batchId=126)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats2]
(batchId=117)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats6]
(batchId=104)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats7]
(batchId=114)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats8]
(batchId=122)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats_noscan_1]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[stats_partscan_1_23]
(batchId=130)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_15]
(batchId=128)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_16]
(batchId=123)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_17]
(batchId=121)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_18]
(batchId=95)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[union_remove_25]
(batchId=129)
org.apache.hadoop.hive.metastore.hbase.TestHBaseSchemaTool.oneMondoTest
(batchId=191)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2036/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2036/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2036/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 38 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838042 - PreCommit-HIVE-Build
> migrate ColumnStats to use jackson
> --
>
> Key: HIVE-15161
> URL: https://issues.apache.org/jira/browse/HIVE-15161
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Zoltan Haindrich
> Fix For: 2.2.0
>
> Attachments: HIVE-15161.1.patch
>
>
> *
[jira] [Comment Edited] (HIVE-15162) NPE in ATSHook
[
https://issues.apache.org/jira/browse/HIVE-15162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15649000#comment-15649000
]
Jason Dere edited comment on HIVE-15162 at 11/8/16 10:35 PM:
-
{code}
String logID =
conf.getLogIdVar(SessionState.get().getSessionId());
{code}
Since the ATSHook is running in a different thread than the thread running the
query, I suspect the thread's SessionID is null. This Session-specific
information might need to be gathered at the time the HookContext is generated
was (Author: jdere):
{code}
String logID =
conf.getLogIdVar(SessionState.get().getSessionId());
{code}
Since the ATSHook is running in a different thread than the thread running the
query, I suspect the thread's SessionID is null.
> NPE in ATSHook
> --
>
> Key: HIVE-15162
> URL: https://issues.apache.org/jira/browse/HIVE-15162
> Project: Hive
> Issue Type: Bug
> Components: Hooks
>Reporter: Jason Dere
>Assignee: Jason Dere
>
> {noformat}
> 2016-11-08T14:21:15,025 INFO [ATS Logger 0]: hooks.ATSHook
> (ATSHook.java:run(156)) - Failed to submit plan to ATS:
> java.lang.NullPointerException
> at org.apache.hadoop.hive.ql.hooks.ATSHook$2.run(ATSHook.java:141)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HIVE-13120) propagate doAs when generating ORC splits
[
https://issues.apache.org/jira/browse/HIVE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648998#comment-15648998
]
Sergey Shelukhin edited comment on HIVE-13120 at 11/8/16 10:34 PM:
---
Attaching the patch that at some point used to apply to branch-1, potentially
on top of some other patches.
Right now it has a lot of conflicts, however most of them are in the test code.
Feel free to rebase in separate JIRA ;)
Original backport was done by [~baluchicken]
was (Author: sershe):
The patch that at some point used to apply to branch-1, potentially on top of
some other patches.
Right now it has a lot of conflicts, however most of them are in the test code.
Feel free to rebase in separate JIRA ;)
Original backport was done by [~baluchicken]
> propagate doAs when generating ORC splits
> -
>
> Key: HIVE-13120
> URL: https://issues.apache.org/jira/browse/HIVE-13120
> Project: Hive
> Issue Type: Improvement
>Reporter: Yi Zhang
>Assignee: Sergey Shelukhin
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HIVE-13120-branch-1.WIP.patch, HIVE-13120.patch
>
>
> ORC+HS2+doAs+FetchTask conversion = weird permission errors, e.g.
> {noformat}
> 2016-02-22 17:24:39,005 WARN [HiveServer2-Handler-Pool: Thread-587]:
> thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(681)) - Error
> fetching results:
> org.apache.hive.service.cli.HiveSQLException: java.io.IOException:
> java.lang.RuntimeException: serious problem
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:352)
> [snip]
> Caused by: java.io.IOException: java.lang.RuntimeException: serious problem
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1720)
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:347)
> ... 24 more
> Caused by: java.lang.RuntimeException: serious problem
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1059)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1086)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:363)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:295)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> ... 28 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=[snip], access=READ_EXECUTE, inode=[snip]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-13120) propagate doAs when generating ORC splits
[
https://issues.apache.org/jira/browse/HIVE-13120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-13120:
Attachment: HIVE-13120-branch-1.WIP.patch
The patch that at some point used to apply to branch-1, potentially on top of
some other patches.
Right now it has a lot of conflicts, however most of them are in the test code.
Feel free to rebase in separate JIRA ;)
Original backport was done by [~baluchicken]
> propagate doAs when generating ORC splits
> -
>
> Key: HIVE-13120
> URL: https://issues.apache.org/jira/browse/HIVE-13120
> Project: Hive
> Issue Type: Improvement
>Reporter: Yi Zhang
>Assignee: Sergey Shelukhin
> Fix For: 2.1.0, 2.0.1
>
> Attachments: HIVE-13120-branch-1.WIP.patch, HIVE-13120.patch
>
>
> ORC+HS2+doAs+FetchTask conversion = weird permission errors, e.g.
> {noformat}
> 2016-02-22 17:24:39,005 WARN [HiveServer2-Handler-Pool: Thread-587]:
> thrift.ThriftCLIService (ThriftCLIService.java:FetchResults(681)) - Error
> fetching results:
> org.apache.hive.service.cli.HiveSQLException: java.io.IOException:
> java.lang.RuntimeException: serious problem
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:352)
> [snip]
> Caused by: java.io.IOException: java.lang.RuntimeException: serious problem
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:508)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.pushRow(FetchOperator.java:415)
> at org.apache.hadoop.hive.ql.exec.FetchTask.fetch(FetchTask.java:140)
> at org.apache.hadoop.hive.ql.Driver.getResults(Driver.java:1720)
> at
> org.apache.hive.service.cli.operation.SQLOperation.getNextRowSet(SQLOperation.java:347)
> ... 24 more
> Caused by: java.lang.RuntimeException: serious problem
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.generateSplitsInfo(OrcInputFormat.java:1059)
> at
> org.apache.hadoop.hive.ql.io.orc.OrcInputFormat.getSplits(OrcInputFormat.java:1086)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextSplits(FetchOperator.java:363)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getRecordReader(FetchOperator.java:295)
> at
> org.apache.hadoop.hive.ql.exec.FetchOperator.getNextRow(FetchOperator.java:446)
> ... 28 more
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.hadoop.security.AccessControlException: Permission denied:
> user=[snip], access=READ_EXECUTE, inode=[snip]
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14990) run all tests for MM tables and fix the issues that are found
[
https://issues.apache.org/jira/browse/HIVE-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648980#comment-15648980
]
Sergey Shelukhin commented on HIVE-14990:
-
Looked at all the remaining tests. Out of 749 failed tests, about 100 failures
and diffs are (or might be, at least) relevant.
Many of them are similar e.g. are missing stats, but I don't know if they are
missing stats for the same reason.
Many e.g. exim may be due to unsupported/path-dependant scenarios that were not
immediately obvious.
Not sure why TestSparkCliDriver fails. Fails in client init for me with no
useful logs (logs child process exited with 127, then times out).
I think we'll fix that during branch merge, if still broken.
{noformat}
TestCliDriver:
authorization_insert
create_default_prop
exim_04_evolved_parts
exim_11_managed_external
exim_12_external_location
exim_15_external_part
exim_18_part_external
exim_19_00_part_external_location
exim_19_part_external_location
insert1
list_bucket_dml_8
mm_all
orc_createas1
ppd_join4
stats_empty_dyn_part
stats_partscan_1_23
temp_table_display_colstats_tbllvl
temp_table_options1
vector_udf2
list_bucket_dml_14,list_bucket_*
llap_acid
insert_overwrite_directory2
authorization_load
autoColumnStats_9
create_like
drop_database_removes_partition_dirs
drop_table_removes_partition_dirs
index_auto_update
exim_01_nonpart,exim_02_part,exim_04_all_part,exim_05_some_part,exim_06_one_part,exim_16_part_external,exim_17_part_managed,exim_20_part_managed_location
load_overwrite
materialized_view_authorization_sqlstd,materialized_*
merge_dynamic_partition, merge_dynamic_partition*
orc_int_type_promotion
orc_vectorization_ppd
parquet_join2
partition_wise_fileformat,partition_wise_fileformat3
repl_1_drop,repl_3_exim_metadata
sample6
sample_islocalmode_hook
show_tablestatus
smb_bucket_1
smb_mapjoin_2,smb_mapjoin_3,smb_mapjoin_7
stats_list_bucket
stats_noscan_2
symlink_text_input_format
temp_table_precedence
offset_limit_global_optimizer
rand_partitionpruner2
TestEncryptedHDFSCliDriver:
encryption_ctas
encryption_drop_partition
encryption_insert_values
encryption_join_unencrypted_tbl
encryption_load_data_to_encrypted_tables
MiniLlapLocal:
exchgpartition2lel
cbo_rp_lineage2
create_merge_compressed
deleteAnalyze
delete_where_no_match
delete_where_non_partitioned
dynpart_sort_optimization
escape2
insert1
lineage2
lineage3
orc_llap
schema_evol_orc_nonvec_part
schema_evol_orc_vec_part
schema_evol_text_nonvec_part
schema_evol_text_vec_part
schema_evol_text_vecrow_part
smb_mapjoin_6
tez_dml
union_fast_stats
update_all_types
update_tmp_table
update_where_no_match
update_where_non_partitioned
vector_outer_join1
vector_outer_join4
MiniLlap:
load_fs2
orc_ppd_basic
external_table_with_space_in_location_path
file_with_header_footer
import_exported_table
schemeAuthority,schemeAuthority2
table_nonprintable
Minimr:
infer_bucket_sort_map_operators
infer_bucket_sort_merge
infer_bucket_sort_reducers_power_two
root_dir_external_table
scriptfile1
TestSymlinkTextInputFormat#testCombine
TestJdbcWithLocalClusterSpark, etc.
{noformat}
> run all tests for MM tables and fix the issues that are found
> -
>
> Key: HIVE-14990
> URL: https://issues.apache.org/jira/browse/HIVE-14990
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14990.01.patch, HIVE-14990.02.patch,
> HIVE-14990.03.patch, HIVE-14990.04.patch, HIVE-14990.04.patch,
> HIVE-14990.05.patch, HIVE-14990.05.patch, HIVE-14990.06.patch,
> HIVE-14990.06.patch, HIVE-14990.07.patch, HIVE-14990.08.patch,
> HIVE-14990.patch
>
>
> Expected failures
> 1) All HCat tests (cannot write MM tables via the HCat writer)
> 2) Almost all merge tests (alter .. concat is not supported).
> 3) Tests that run dfs commands with specific paths (path changes).
> 4) Truncate column (not supported).
> 5) Describe formatted will have the new table fields in the output (before
> merging MM with ACID).
> 6) Many tests w/explain extended - diff in partition "base file name" (path
> changes).
> 7) TestTxnCommands - all the conversion tests, as they check for bucket count
> using file lists (path changes).
> 8) HBase metastore tests cause methods are not implemented.
> 9) Some load and ExIm tests that export a table and then rely on specific
> path for load (path changes).
> 10) Bucket map join/etc. - diffs; disabled the optimization for MM tables due
> to how it accounts for buckets
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15060) Remove the autoCommit warning from beeline
[ https://issues.apache.org/jira/browse/HIVE-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-15060: - Resolution: Fixed Fix Version/s: 2.2.0 Release Note: Beeline option for autoCommit has changed to true. To turn autoCommit to false (does not work with Hive), use beeline --autoCommit=false Status: Resolved (was: Patch Available) Patch committed to master. Thanks for the contribution [~taoli-hwx]! > Remove the autoCommit warning from beeline > -- > > Key: HIVE-15060 > URL: https://issues.apache.org/jira/browse/HIVE-15060 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Fix For: 2.2.0 > > Attachments: HIVE-15060.1.patch, HIVE-15060.2.patch > > > WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not > support autoCommit=false. > Transaction isolation: TRANSACTION_REPEATABLE_READ > 0: jdbc:hive2://ctr-e89-1466633100028-0275-01 > By default, this beeline setting is false, while hive only support > autoCommit=true for now. So this warning does mot make sense and should be > removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HIVE-10901) Optimize mutli column distinct queries
[ https://issues.apache.org/jira/browse/HIVE-10901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Pengcheng Xiong reassigned HIVE-10901: -- Assignee: Pengcheng Xiong (was: Ashutosh Chauhan) > Optimize mutli column distinct queries > > > Key: HIVE-10901 > URL: https://issues.apache.org/jira/browse/HIVE-10901 > Project: Hive > Issue Type: New Feature > Components: CBO, Logical Optimizer >Affects Versions: 1.2.0 >Reporter: Mostafa Mokhtar >Assignee: Pengcheng Xiong > Attachments: HIVE-10901.patch > > > HIVE-10568 is useful only when there is a distinct on one column. It can be > expanded for multiple column cases too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15155) Change Hive version shortname to 2.2.0
[ https://issues.apache.org/jira/browse/HIVE-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648894#comment-15648894 ] Ashutosh Chauhan commented on HIVE-15155: - +1 > Change Hive version shortname to 2.2.0 > -- > > Key: HIVE-15155 > URL: https://issues.apache.org/jira/browse/HIVE-15155 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-15155.patch > > > Pointing to 2.1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14943) Base Implementation
[
https://issues.apache.org/jira/browse/HIVE-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648892#comment-15648892
]
Hive QA commented on HIVE-14943:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838040/HIVE-14943.10.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10688 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=91)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2035/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2035/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2035/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838040 - PreCommit-HIVE-Build
> Base Implementation
> ---
>
> Key: HIVE-14943
> URL: https://issues.apache.org/jira/browse/HIVE-14943
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
> Attachments: HIVE-14943.10.patch, HIVE-14943.2.patch,
> HIVE-14943.3.patch, HIVE-14943.4.patch, HIVE-14943.5.patch,
> HIVE-14943.6.patch, HIVE-14943.7.patch, HIVE-14943.8.patch,
> HIVE-14943.9.patch, HIVE-14943.patch
>
>
> Create the 1st pass functional implementation of MERGE
> This should run e2e and produce correct results.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15060) Remove the autoCommit warning from beeline
[ https://issues.apache.org/jira/browse/HIVE-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648852#comment-15648852 ] Tao Li commented on HIVE-15060: --- Sure. HIVE-15084 is being tracked for failure of: org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver HIVE-14910 has been resolved for failure of: org.apache.hive.spark.client.TestSparkClient.testJobSubmission > Remove the autoCommit warning from beeline > -- > > Key: HIVE-15060 > URL: https://issues.apache.org/jira/browse/HIVE-15060 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-15060.1.patch, HIVE-15060.2.patch > > > WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not > support autoCommit=false. > Transaction isolation: TRANSACTION_REPEATABLE_READ > 0: jdbc:hive2://ctr-e89-1466633100028-0275-01 > By default, this beeline setting is false, while hive only support > autoCommit=true for now. So this warning does mot make sense and should be > removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648809#comment-15648809
]
Sergey Shelukhin commented on HIVE-14089:
-
Hmm, all the tests that fail in the same batch. One of the tests seemingly
screws up the state... will take a look.
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.04.patch, HIVE-14089.05.patch,
> HIVE-14089.06.patch, HIVE-14089.07.patch, HIVE-14089.08.patch,
> HIVE-14089.09.patch, HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
> Note to self - need to add multi-stripe test, and also test complex types
> with some nulls so that present stream is not suppressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15137) metastore add partitions background thread should use current username
[ https://issues.apache.org/jira/browse/HIVE-15137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thejas M Nair updated HIVE-15137: - Assignee: Daniel Dai (was: Thejas M Nair) > metastore add partitions background thread should use current username > -- > > Key: HIVE-15137 > URL: https://issues.apache.org/jira/browse/HIVE-15137 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 2.2.0, 2.1.1 >Reporter: Thejas M Nair >Assignee: Daniel Dai > > The background thread used in HIVE-13901 for adding partitions needs to be > reinitialized with current UGI for each invocation. Otherwise the user in > context while thread was created would be the current UGI during the actions > in the thread. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648773#comment-15648773
]
Hive QA commented on HIVE-14089:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838027/HIVE-14089.09.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 17 failed/errored test(s), 10630 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucket_map_join_2]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constprog_when_case]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[create_udaf] (batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby4_map_skew]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[lateral_view_onview]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_uncompressed]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_constant_expr]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_unquote_not]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_7]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_ppr_all]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_in_having]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorization_1]
(batchId=52)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_string_funcs]
(batchId=52)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[orc_split_elimination]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2034/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2034/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2034/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 17 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838027 - PreCommit-HIVE-Build
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.04.patch, HIVE-14089.05.patch,
> HIVE-14089.06.patch, HIVE-14089.07.patch, HIVE-14089.08.patch,
> HIVE-14089.09.patch, HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
> Note to self - need to add multi-stripe test, and also test complex types
> with some nulls so that present stream is not suppressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15060) Remove the autoCommit warning from beeline
[ https://issues.apache.org/jira/browse/HIVE-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648768#comment-15648768 ] Thejas M Nair commented on HIVE-15060: -- [~taoli-hwx] Can you please check if the unrelated tests failures are tracked in other jiras or fixed recently ? If not, we should be opening new jiras to track them before committing this. > Remove the autoCommit warning from beeline > -- > > Key: HIVE-15060 > URL: https://issues.apache.org/jira/browse/HIVE-15060 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-15060.1.patch, HIVE-15060.2.patch > > > WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not > support autoCommit=false. > Transaction isolation: TRANSACTION_REPEATABLE_READ > 0: jdbc:hive2://ctr-e89-1466633100028-0275-01 > By default, this beeline setting is false, while hive only support > autoCommit=true for now. So this warning does mot make sense and should be > removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15060) Remove the autoCommit warning from beeline
[ https://issues.apache.org/jira/browse/HIVE-15060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648756#comment-15648756 ] Tao Li commented on HIVE-15060: --- Ping [~thejas] > Remove the autoCommit warning from beeline > -- > > Key: HIVE-15060 > URL: https://issues.apache.org/jira/browse/HIVE-15060 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-15060.1.patch, HIVE-15060.2.patch > > > WARN jdbc.HiveConnection: Request to set autoCommit to false; Hive does not > support autoCommit=false. > Transaction isolation: TRANSACTION_REPEATABLE_READ > 0: jdbc:hive2://ctr-e89-1466633100028-0275-01 > By default, this beeline setting is false, while hive only support > autoCommit=true for now. So this warning does mot make sense and should be > removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-14992) Relocate several common libraries in hive jdbc uber jar
[ https://issues.apache.org/jira/browse/HIVE-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648750#comment-15648750 ] Thejas M Nair commented on HIVE-14992: -- Trying to get a unit test on the updated patch. Just kicked off build #14992 > Relocate several common libraries in hive jdbc uber jar > --- > > Key: HIVE-14992 > URL: https://issues.apache.org/jira/browse/HIVE-14992 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14992.1.patch, HIVE-14992.2.patch, > HIVE-14992.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15090) Temporary DB failure can stop ExpiredTokenRemover thread
[ https://issues.apache.org/jira/browse/HIVE-15090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648745#comment-15648745 ] Thejas M Nair commented on HIVE-15090: -- We shouldn't be catching all throwables, that will include errors like OOM where retrying doesn't make sense. Can we limit it to certain exceptions that can be realistically thrown ? > Temporary DB failure can stop ExpiredTokenRemover thread > > > Key: HIVE-15090 > URL: https://issues.apache.org/jira/browse/HIVE-15090 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 1.3.0, 2.1.0, 2.0.1, 2.2.0 >Reporter: Peter Vary >Assignee: Peter Vary > Fix For: 2.2.0 > > Attachments: HIVE-15090.2-branch-2.1.patch, HIVE-15090.2.patch, > HIVE-15090.2.patch, HIVE-15090.patch > > > In HIVE-13090 we decided that we should not close the metastore if there is > an unexpected exception during the expired token removal process, but that > fix leaves a running metastore without ExpiredTokenRemover thread. > To fix this I will move the catch inside the running loop, and hope the > thread could recover from the exception -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15161) migrate ColumnStats to use jackson
[ https://issues.apache.org/jira/browse/HIVE-15161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-15161: Status: Patch Available (was: Open) > migrate ColumnStats to use jackson > -- > > Key: HIVE-15161 > URL: https://issues.apache.org/jira/browse/HIVE-15161 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 2.2.0 > > Attachments: HIVE-15161.1.patch > > > * json.org has license issues > * jackson can provide a fully compatible alternative to it > * there are a few flakiness issues caused by the order of the map entries of > the columns...this cat be addressed, org.json api was unfriendly in this > manner ;) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15161) migrate ColumnStats to use jackson
[ https://issues.apache.org/jira/browse/HIVE-15161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-15161: Attachment: HIVE-15161.1.patch i've experimented with migrating this thing to jackson...and I wanted to get back to itbut now that org.json should go - this could be a viable alternative to it. #1 - checking tests; there will be possibly a few - where the column name order will change... > migrate ColumnStats to use jackson > -- > > Key: HIVE-15161 > URL: https://issues.apache.org/jira/browse/HIVE-15161 > Project: Hive > Issue Type: Sub-task >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich > Fix For: 2.2.0 > > Attachments: HIVE-15161.1.patch > > > * json.org has license issues > * jackson can provide a fully compatible alternative to it > * there are a few flakiness issues caused by the order of the map entries of > the columns...this cat be addressed, org.json api was unfriendly in this > manner ;) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648686#comment-15648686
]
Jesus Camacho Rodriguez commented on HIVE-15160:
Thanks for double-checking [~pxiong], I may be wrong then.
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15159) LLAP ContainerRunner should not reduce the available heap while distributing to individual executors
[
https://issues.apache.org/jira/browse/HIVE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648669#comment-15648669
]
Hive QA commented on HIVE-15159:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12838025/HIVE-15159.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10626 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=90)
org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver.org.apache.hadoop.hive.cli.TestSparkNegativeCliDriver
(batchId=226)
org.apache.hive.spark.client.TestSparkClient.testJobSubmission (batchId=272)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2033/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2033/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2033/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12838025 - PreCommit-HIVE-Build
> LLAP ContainerRunner should not reduce the available heap while distributing
> to individual executors
>
>
> Key: HIVE-15159
> URL: https://issues.apache.org/jira/browse/HIVE-15159
> Project: Hive
> Issue Type: Task
> Components: llap
>Reporter: Siddharth Seth
>Assignee: Siddharth Seth
> Attachments: HIVE-15159.01.patch
>
>
> hive.llap.daemon.memory.per.instance.mb is set to the heapsize of the process
> when llap service driver is used.
> ContainerRunner currently cuts this down to 80% of the available. This
> reduction is not necessary if the heapsize is used, instead of a container
> size.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14943) Base Implementation
[ https://issues.apache.org/jira/browse/HIVE-14943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-14943: -- Attachment: HIVE-14943.10.patch > Base Implementation > --- > > Key: HIVE-14943 > URL: https://issues.apache.org/jira/browse/HIVE-14943 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > Attachments: HIVE-14943.10.patch, HIVE-14943.2.patch, > HIVE-14943.3.patch, HIVE-14943.4.patch, HIVE-14943.5.patch, > HIVE-14943.6.patch, HIVE-14943.7.patch, HIVE-14943.8.patch, > HIVE-14943.9.patch, HIVE-14943.patch > > > Create the 1st pass functional implementation of MERGE > This should run e2e and produce correct results. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648661#comment-15648661
]
Pengcheng Xiong commented on HIVE-15160:
{code}
select value from src order by key;
{code}
fails on Hive but passes on Postgres and Oracle.
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-14992) Relocate several common libraries in hive jdbc uber jar
[ https://issues.apache.org/jira/browse/HIVE-14992?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648657#comment-15648657 ] Tao Li commented on HIVE-14992: --- Thanks [~thejas] Would you be able to commit it? > Relocate several common libraries in hive jdbc uber jar > --- > > Key: HIVE-14992 > URL: https://issues.apache.org/jira/browse/HIVE-14992 > Project: Hive > Issue Type: Bug >Reporter: Tao Li >Assignee: Tao Li > Attachments: HIVE-14992.1.patch, HIVE-14992.2.patch, > HIVE-14992.3.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15160) Can't order by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pengcheng Xiong updated HIVE-15160:
---
Summary: Can't order by an unselected column (was: Can't group by an
unselected column)
> Can't order by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15155) Change Hive version shortname to 2.2.0
[ https://issues.apache.org/jira/browse/HIVE-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648572#comment-15648572 ] Jesus Camacho Rodriguez commented on HIVE-15155: [~ashutoshc], could you take a look? It is a very simple fix but it was preventing correct selection of scripts for initialization of metastore in master. Thanks > Change Hive version shortname to 2.2.0 > -- > > Key: HIVE-15155 > URL: https://issues.apache.org/jira/browse/HIVE-15155 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-15155.patch > > > Pointing to 2.1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15155) Change Hive version shortname to 2.2.0
[
https://issues.apache.org/jira/browse/HIVE-15155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648536#comment-15648536
]
Hive QA commented on HIVE-15155:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12837971/HIVE-15155.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 10630 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hive.spark.client.TestSparkClient.testJobSubmission (batchId=272)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2028/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2028/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2028/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12837971 - PreCommit-HIVE-Build
> Change Hive version shortname to 2.2.0
> --
>
> Key: HIVE-15155
> URL: https://issues.apache.org/jira/browse/HIVE-15155
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Minor
> Attachments: HIVE-15155.patch
>
>
> Pointing to 2.1.0.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15160) Can't group by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648425#comment-15648425
]
Jesus Camacho Rodriguez commented on HIVE-15160:
Does it complain because of the _group by_ or the _order by_ clause? If it is
because of the _order by_, I think it is correct from SQL point of view, since
_select_ should happen before _order by_?
> Can't group by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-15160) Can't group by an unselected column
[
https://issues.apache.org/jira/browse/HIVE-15160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648421#comment-15648421
]
Vineet Garg commented on HIVE-15160:
[~pxiong] Can you post the error message ?
I tried simple query {code} select sum(l_linenumber) from lineitem group by
l_discount; {code} and I don't get any error
> Can't group by an unselected column
> ---
>
> Key: HIVE-15160
> URL: https://issues.apache.org/jira/browse/HIVE-15160
> Project: Hive
> Issue Type: Bug
>Reporter: Pengcheng Xiong
>Assignee: Pengcheng Xiong
>
> If a grouping key hasn't been selected, Hive complains. For comparison,
> Postgres does not.
> Example. Notice i_item_id is not selected:
> {code}
> select i_item_desc
>,i_category
>,i_class
>,i_current_price
>,sum(cs_ext_sales_price) as itemrevenue
>,sum(cs_ext_sales_price)*100/sum(sum(cs_ext_sales_price)) over
>(partition by i_class) as revenueratio
> from catalog_sales
> ,item
> ,date_dim
> where cs_item_sk = i_item_sk
>and i_category in ('Jewelry', 'Sports', 'Books')
>and cs_sold_date_sk = d_date_sk
> and d_date between cast('2001-01-12' as date)
> and (cast('2001-01-12' as date) + 30 days)
> group by i_item_id
> ,i_item_desc
> ,i_category
> ,i_class
> ,i_current_price
> order by i_category
> ,i_class
> ,i_item_id
> ,i_item_desc
> ,revenueratio
> limit 100;
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-14089) complex type support in LLAP IO is broken
[
https://issues.apache.org/jira/browse/HIVE-14089?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-14089:
Attachment: HIVE-14089.09.patch
Cannot repro those and the logs are gone.
Trying again...
> complex type support in LLAP IO is broken
> --
>
> Key: HIVE-14089
> URL: https://issues.apache.org/jira/browse/HIVE-14089
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Prasanth Jayachandran
>Assignee: Sergey Shelukhin
> Attachments: HIVE-14089.04.patch, HIVE-14089.05.patch,
> HIVE-14089.06.patch, HIVE-14089.07.patch, HIVE-14089.08.patch,
> HIVE-14089.09.patch, HIVE-14089.WIP.2.patch, HIVE-14089.WIP.3.patch,
> HIVE-14089.WIP.patch
>
>
> HIVE-13617 is causing MiniLlapCliDriver following test failures
> {code}
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_all
> org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver_vector_complex_join
> {code}
> Note to self - need to add multi-stripe test, and also test complex types
> with some nulls so that present stream is not suppressed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-13000) Hive returns useless parsing error
[
https://issues.apache.org/jira/browse/HIVE-13000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648396#comment-15648396
]
Hive QA commented on HIVE-13000:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12837969/HIVE-13000.5.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 6 failed/errored test(s), 10631 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=135)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[groupby_struct]
(batchId=83)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[subquery_notexists_implicit_gby]
(batchId=83)
org.apache.hadoop.hive.ql.parse.TestParseNegativeDriver.testCliDriver[nonkey_groupby]
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2027/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2027/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2027/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 6 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12837969 - PreCommit-HIVE-Build
> Hive returns useless parsing error
> ---
>
> Key: HIVE-13000
> URL: https://issues.apache.org/jira/browse/HIVE-13000
> Project: Hive
> Issue Type: Bug
>Affects Versions: 0.13.0, 1.0.0, 1.2.1, 2.2.0
>Reporter: Alina Abramova
>Assignee: Alina Abramova
>Priority: Minor
> Attachments: HIVE-13000.1.patch, HIVE-13000.2.patch,
> HIVE-13000.3.patch, HIVE-13000.4.patch, HIVE-13000.5.patch
>
>
> When I run query like these I receive unclear exception
> hive> SELECT record FROM ctest GROUP BY record.instance_id;
> FAILED: SemanticException Error in parsing
> It will be clearer if it would be like:
> hive> SELECT record FROM ctest GROUP BY record.instance_id;
> FAILED: SemanticException Expression not in GROUP BY key record
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15159) LLAP ContainerRunner should not reduce the available heap while distributing to individual executors
[ https://issues.apache.org/jira/browse/HIVE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-15159: -- Status: Patch Available (was: Open) > LLAP ContainerRunner should not reduce the available heap while distributing > to individual executors > > > Key: HIVE-15159 > URL: https://issues.apache.org/jira/browse/HIVE-15159 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-15159.01.patch > > > hive.llap.daemon.memory.per.instance.mb is set to the heapsize of the process > when llap service driver is used. > ContainerRunner currently cuts this down to 80% of the available. This > reduction is not necessary if the heapsize is used, instead of a container > size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15159) LLAP ContainerRunner should not reduce the available heap while distributing to individual executors
[ https://issues.apache.org/jira/browse/HIVE-15159?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siddharth Seth updated HIVE-15159: -- Attachment: HIVE-15159.01.patch [~gopalv] - could you please take a look. > LLAP ContainerRunner should not reduce the available heap while distributing > to individual executors > > > Key: HIVE-15159 > URL: https://issues.apache.org/jira/browse/HIVE-15159 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Siddharth Seth >Assignee: Siddharth Seth > Attachments: HIVE-15159.01.patch > > > hive.llap.daemon.memory.per.instance.mb is set to the heapsize of the process > when llap service driver is used. > ContainerRunner currently cuts this down to 80% of the available. This > reduction is not necessary if the heapsize is used, instead of a container > size. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15153) Loading data from files having invalid extensions or no extension
[ https://issues.apache.org/jira/browse/HIVE-15153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648329#comment-15648329 ] Sergey Shelukhin commented on HIVE-15153: - Why should there be an error? Hive does not use extension in any way. > Loading data from files having invalid extensions or no extension > - > > Key: HIVE-15153 > URL: https://issues.apache.org/jira/browse/HIVE-15153 > Project: Hive > Issue Type: Bug >Reporter: Pallavi Singh > > When we execute the following queries: > LOAD DATA INPATH 'hdfs://localhost:54310/user/hive/warehouse/file1.csv.csv' > INTO TABLE emp; > LOAD DATA INPATH > 'hdfs://localhost:54310/user/hive/warehouse/file1.csv.csv.csv.csv' INTO TABLE > emp; > LOAD DATA INPATH 'hdfs://localhost:54310/user/hive/warehouse/file1.txttt' > INTO TABLE emp; > LOAD DATA INPATH 'hdfs://localhost:54310/user/hive/warehouse/file1' INTO > TABLE emp; > LOAD DATA INPATH > 'hdfs://localhost:54310/user/hive/warehouse/file1.txt.bat.csv' INTO TABLE emp; > We should get Input File Error, but we are able to load data successfully > into hive table -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15033) Ensure there is only 1 StatsTask in the query plan
[ https://issues.apache.org/jira/browse/HIVE-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648327#comment-15648327 ] Eugene Koifman commented on HIVE-15033: --- I talked to [~ashutoshc] - he doesn't think concurrent StatsTasks will corrupt anything. So the issue here mostly about efficiency. Providing answers to queries based on stats is turned off for Acid tables. > Ensure there is only 1 StatsTask in the query plan > -- > > Key: HIVE-15033 > URL: https://issues.apache.org/jira/browse/HIVE-15033 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > currently there is 1 per WHEN clause -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15033) Ensure there is only 1 StatsTask in the query plan
[ https://issues.apache.org/jira/browse/HIVE-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648301#comment-15648301 ] Eugene Koifman commented on HIVE-15033: --- This prop causes a new node (stage) to be inserted into the plan that acts like a sync point (CyclicBarrier). It doesn't change the number of MoveTask noes. > Ensure there is only 1 StatsTask in the query plan > -- > > Key: HIVE-15033 > URL: https://issues.apache.org/jira/browse/HIVE-15033 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > currently there is 1 per WHEN clause -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-15033) Ensure there is only 1 StatsTask in the query plan
[ https://issues.apache.org/jira/browse/HIVE-15033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648277#comment-15648277 ] Eugene Koifman commented on HIVE-15033: --- Perhaps this needs a different approach altogether. Merge only works on full ACID tables which use MVCC. Stats data, on the other hand, is not versioned. Even if we fix multiple StatsTask issue in Merge, it's perfectly legal in Hive to have to concurrent inserts into an Acid table so you could still end up with stats data for the partition which is not accurate (assuming parallel stats computations simply overwrite each other, rather than corrupt some data structures). The later is a general issue with Acid and stats (and may be for MicroManaged tables as well). Perhaps the right answer is to only compute stats for Acid tables at compactions (or even add another process to trigger stats computation based on number of writes to the partition). As long as stats are used to guide the CBO, not provide exact answers to queries, they would still be approximately accurate and thus useful. cc [~alangates], [~pxiong] > Ensure there is only 1 StatsTask in the query plan > -- > > Key: HIVE-15033 > URL: https://issues.apache.org/jira/browse/HIVE-15033 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > > currently there is 1 per WHEN clause -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HIVE-12891) Hive fails when java.io.tmpdir is set to a relative location
[
https://issues.apache.org/jira/browse/HIVE-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648252#comment-15648252
]
Hive QA commented on HIVE-12891:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12837968/HIVE-12891.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 10635 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_ppd_schema_evol_3a]
(batchId=133)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[columnstats_part_coltype]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[join_acid_non_acid]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[union_fast_stats]
(batchId=145)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_4]
(batchId=91)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=90)
org.apache.hive.service.cli.TestEmbeddedThriftBinaryCLIService.testTaskStatus
(batchId=207)
org.apache.hive.spark.client.TestSparkClient.testJobSubmission (batchId=272)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/2026/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/2026/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-2026/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12837968 - PreCommit-HIVE-Build
> Hive fails when java.io.tmpdir is set to a relative location
>
>
> Key: HIVE-12891
> URL: https://issues.apache.org/jira/browse/HIVE-12891
> Project: Hive
> Issue Type: Bug
>Reporter: Reuben Kuhnert
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12891.01.19.2016.01.patch, HIVE-12891.03.patch,
> HIVE-12891.04.patch, HIVE-12891.5.patch, HIVE-12981.01.22.2016.02.patch
>
>
> The function {{SessionState.createSessionDirs}} fails when trying to create
> directories where {{java.io.tmpdir}} is set to a relative location.
> {code}
> \[SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED:
> IllegalArgumentException java.net.URISyntaxException: Relative path in
> absolute URI:
> file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
> ...
> Minor variations:
> \[SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED: SemanticException
> Exception while processing Exception while writing out the local file
> o.a.h.hive.ql/parse.SemanticException: Exception while processing exception
> while writing out local file
> ...
> caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative path in absolute URI:
> file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
> at o.a.h.fs.Path.initialize (206)
> at o.a.h.fs.Path.(197)...
> at o.a.h.hive.ql.context.getScratchDir(267)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HIVE-12891) Hive fails when java.io.tmpdir is set to a relative location
[
https://issues.apache.org/jira/browse/HIVE-12891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15648211#comment-15648211
]
Chaoyu Tang commented on HIVE-12891:
LGTM, +1
> Hive fails when java.io.tmpdir is set to a relative location
>
>
> Key: HIVE-12891
> URL: https://issues.apache.org/jira/browse/HIVE-12891
> Project: Hive
> Issue Type: Bug
>Reporter: Reuben Kuhnert
>Assignee: Barna Zsombor Klara
> Attachments: HIVE-12891.01.19.2016.01.patch, HIVE-12891.03.patch,
> HIVE-12891.04.patch, HIVE-12891.5.patch, HIVE-12981.01.22.2016.02.patch
>
>
> The function {{SessionState.createSessionDirs}} fails when trying to create
> directories where {{java.io.tmpdir}} is set to a relative location.
> {code}
> \[SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED:
> IllegalArgumentException java.net.URISyntaxException: Relative path in
> absolute URI:
> file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
> ...
> Minor variations:
> \[SubtaskRunner] ERROR o.a.h.hive..ql.Driver - FAILED: SemanticException
> Exception while processing Exception while writing out the local file
> o.a.h.hive.ql/parse.SemanticException: Exception while processing exception
> while writing out local file
> ...
> caused by: java.lang.IllegalArgumentException: java.net.URISyntaxException:
> Relative path in absolute URI:
> file:./tmp///hive_2015_12_11_09-12-25_352_4325234652356-1
> at o.a.h.fs.Path.initialize (206)
> at o.a.h.fs.Path.(197)...
> at o.a.h.hive.ql.context.getScratchDir(267)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HIVE-15154) Fix rest of q test file changes in branch-2.1
[ https://issues.apache.org/jira/browse/HIVE-15154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-15154: --- Attachment: HIVE-15154-branch-2.1.patch > Fix rest of q test file changes in branch-2.1 > - > > Key: HIVE-15154 > URL: https://issues.apache.org/jira/browse/HIVE-15154 > Project: Hive > Issue Type: Sub-task > Components: Test >Affects Versions: 2.1.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Minor > Attachments: HIVE-15154-branch-2.1.patch, HIVE-15154-branch-2.1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-15119) Support standard syntax for ROLLUP & CUBE
[ https://issues.apache.org/jira/browse/HIVE-15119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-15119: --- Attachment: HIVE-15119.5.patch > Support standard syntax for ROLLUP & CUBE > - > > Key: HIVE-15119 > URL: https://issues.apache.org/jira/browse/HIVE-15119 > Project: Hive > Issue Type: Task > Components: Parser, SQL >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-15119.03.patch, HIVE-15119.2.patch, > HIVE-15119.4.patch, HIVE-15119.5.patch, HIVE-15119.patch > > > Standard ROLLUP and CUBE syntax is GROUP BY ROLLUP (expression list)... and > GROUP BY CUBE (expression list) respectively. > Currently HIVE only allows GROUP BY WITH ROLLUP/CUBE syntax. > > We would like HIVE to support standard ROLLUP/CUBE syntax. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HIVE-13966) DbNotificationListener: can loose DDL operation notifications
[ https://issues.apache.org/jira/browse/HIVE-13966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mohit Sabharwal updated HIVE-13966: --- Attachment: HIVE-13966.4.patch > DbNotificationListener: can loose DDL operation notifications > - > > Key: HIVE-13966 > URL: https://issues.apache.org/jira/browse/HIVE-13966 > Project: Hive > Issue Type: Bug > Components: HCatalog >Reporter: Nachiket Vaidya >Assignee: Mohit Sabharwal >Priority: Critical > Attachments: HIVE-13966.1.patch, HIVE-13966.2.patch, > HIVE-13966.3.patch, HIVE-13966.4.patch, HIVE-13966.4.patch, HIVE-13966.pdf > > > The code for each API in HiveMetaStore.java is like this: > 1. openTransaction() > 2. -- operation-- > 3. commit() or rollback() based on result of the operation. > 4. add entry to notification log (unconditionally) > If the operation is failed (in step 2), we still add entry to notification > log. Found this issue in testing. > It is still ok as this is the case of false positive. > If the operation is successful and adding to notification log failed, the > user will get an MetaException. It will not rollback the operation, as it is > already committed. We need to handle this case so that we will not have false > negatives. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
