[jira] [Commented] (HIVE-16827) Merge stats task and column stats task into a single task
[ https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241610#comment-16241610 ] Zoltan Haindrich commented on HIVE-16827: - bucketmapjoin7 have missed the q.out update; other test results look good > Merge stats task and column stats task into a single task > - > > Key: HIVE-16827 > URL: https://issues.apache.org/jira/browse/HIVE-16827 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Zoltan Haindrich > Attachments: HIVE-16827.01.patch, HIVE-16827.02.patch, > HIVE-16827.03.patch, HIVE-16827.04wip03.patch, HIVE-16827.04wip04.patch, > HIVE-16827.04wip05.patch, HIVE-16827.04wip06.patch, HIVE-16827.04wip09.patch, > HIVE-16827.04wip10.patch, HIVE-16827.05.patch, HIVE-16827.05wip01.patch, > HIVE-16827.05wip02.patch, HIVE-16827.05wip03.patch, HIVE-16827.05wip04.patch, > HIVE-16827.05wip05.patch, HIVE-16827.05wip08.patch, HIVE-16827.05wip10.patch, > HIVE-16827.05wip10.patch, HIVE-16827.05wip11.patch, HIVE-16827.05wip12.patch, > HIVE-16827.4.patch > > > Within the task, we can specify whether to compute basic stats only or column > stats only or both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16827) Merge stats task and column stats task into a single task
[
https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241591#comment-16241591
]
Hive QA commented on HIVE-16827:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896286/HIVE-16827.05.patch
{color:green}SUCCESS:{color} +1 due to 17 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11373 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[llap_text] (batchId=73)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
(batchId=173)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin7]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkPerfCliDriver.testCliDriver[query39]
(batchId=245)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7673/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7673/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7673/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896286 - PreCommit-HIVE-Build
> Merge stats task and column stats task into a single task
> -
>
> Key: HIVE-16827
> URL: https://issues.apache.org/jira/browse/HIVE-16827
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Zoltan Haindrich
> Attachments: HIVE-16827.01.patch, HIVE-16827.02.patch,
> HIVE-16827.03.patch, HIVE-16827.04wip03.patch, HIVE-16827.04wip04.patch,
> HIVE-16827.04wip05.patch, HIVE-16827.04wip06.patch, HIVE-16827.04wip09.patch,
> HIVE-16827.04wip10.patch, HIVE-16827.05.patch, HIVE-16827.05wip01.patch,
> HIVE-16827.05wip02.patch, HIVE-16827.05wip03.patch, HIVE-16827.05wip04.patch,
> HIVE-16827.05wip05.patch, HIVE-16827.05wip08.patch, HIVE-16827.05wip10.patch,
> HIVE-16827.05wip10.patch, HIVE-16827.05wip11.patch, HIVE-16827.05wip12.patch,
> HIVE-16827.4.patch
>
>
> Within the task, we can specify whether to compute basic stats only or column
> stats only or both.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Resolution: Fixed Status: Resolved (was: Patch Available) Pushed it to master. Thanks for reviewing [~ashutoshc] > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, > HIVE-17767.6.patch, HIVE-17767.7.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241531#comment-16241531 ] Ashutosh Chauhan commented on HIVE-15016: - +1 > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.10.patch, HIVE-15016.2.patch, > HIVE-15016.3.patch, HIVE-15016.4.patch, HIVE-15016.5.patch, > HIVE-15016.6.patch, HIVE-15016.7.patch, HIVE-15016.8.patch, > HIVE-15016.9.patch, HIVE-15016.patch, Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241527#comment-16241527 ] Ashutosh Chauhan commented on HIVE-17767: - +1 > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, > HIVE-17767.6.patch, HIVE-17767.7.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[
https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241522#comment-16241522
]
Hive QA commented on HIVE-15016:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896278/HIVE-15016.10.patch
{color:green}SUCCESS:{color} +1 due to 16 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11367 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
org.apache.hive.jdbc.TestJdbcWithMiniHS2.testHttpRetryOnServerIdleTimeout
(batchId=233)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7672/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7672/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7672/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896278 - PreCommit-HIVE-Build
> Run tests with Hadoop 3.0.0-beta1
> -
>
> Key: HIVE-15016
> URL: https://issues.apache.org/jira/browse/HIVE-15016
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Sergio Peña
>Assignee: Aihua Xu
> Attachments: HIVE-15016.10.patch, HIVE-15016.2.patch,
> HIVE-15016.3.patch, HIVE-15016.4.patch, HIVE-15016.5.patch,
> HIVE-15016.6.patch, HIVE-15016.7.patch, HIVE-15016.8.patch,
> HIVE-15016.9.patch, HIVE-15016.patch, Hadoop3Upstream.patch
>
>
> Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run
> tests against this new version before GA.
> We should start running tests with Hive to validate compatibility against
> Hadoop 3.0.
> NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0
> GA is released.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17963) Fix for HIVE-17113 can be improved for non-blobstore filesystems
[ https://issues.apache.org/jira/browse/HIVE-17963?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241521#comment-16241521 ] Ashutosh Chauhan commented on HIVE-17963: - +1 Thanks for detailed comments. > Fix for HIVE-17113 can be improved for non-blobstore filesystems > > > Key: HIVE-17963 > URL: https://issues.apache.org/jira/browse/HIVE-17963 > Project: Hive > Issue Type: Bug >Reporter: Jason Dere >Assignee: Jason Dere > Attachments: HIVE-17963.1.patch, HIVE-17963.2.patch > > > HIVE-17113/HIVE-17813 fix the duplicate file issue by performing file moves > on a file-by-file basis. For non-blobstore filesystems this results in many > more filesystem/namenode operations compared to the previous > Utilities.mvFileToFinalPath() behavior (dedup files in src dir, rename src > dir to final dir). > For non-blobstore filesystems, a better solution would be the one described > [here|https://issues.apache.org/jira/browse/HIVE-17113?focusedCommentId=16100564=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16100564]: > 1) Move the temp directory to a new directory name, to prevent additional > files from being added by any runaway processes. > 2) Run removeTempOrDuplicateFiles() on this renamed temp directory > 3) Run renameOrMoveFiles() to move the renamed temp directory to the final > location. > This results in only one additional file operation in non-blobstore FSes > compared to the original Utilities.mvFileToFinalPath() behavior. > The proposal is to do away with the config setting > hive.exec.move.files.from.source.dir and always have behavior that should > take care of the duplicate file issue described in HIVE-17113. For > non-blobstore filesystems we will do steps 1-3 described above. For blobstore > filesystems we will do the solution done in HIVE-17113/HIVE-17813 which does > the file-by-file copy - this should have the same number of file operations > as doing a rename directory on blobstore, which effectively results in file > moves on a file-by-file basis. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241519#comment-16241519 ] Vineet Garg commented on HIVE-17767: Removed those unwanted changes in latest patch. > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, > HIVE-17767.6.patch, HIVE-17767.7.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Status: Patch Available (was: Open) > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, > HIVE-17767.6.patch, HIVE-17767.7.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Attachment: HIVE-17767.7.patch > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, > HIVE-17767.6.patch, HIVE-17767.7.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Status: Open (was: Patch Available) > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, HIVE-17767.6.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241503#comment-16241503 ] Ashutosh Chauhan commented on HIVE-17767: - seems like there are unrelated changes in latest patch. +1 post removing those changes. > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, HIVE-17767.6.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Status: Patch Available (was: Open) > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, HIVE-17767.6.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Attachment: HIVE-17767.6.patch > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, HIVE-17767.6.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Status: Open (was: Patch Available) > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch, HIVE-17767.6.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17961) NPE during initialization of VectorizedParquetRecordReader when input split is null
[
https://issues.apache.org/jira/browse/HIVE-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241470#comment-16241470
]
Hive QA commented on HIVE-17961:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896274/HIVE-17961.02.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11367 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[vectorization_parquet_projection]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7671/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7671/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7671/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896274 - PreCommit-HIVE-Build
> NPE during initialization of VectorizedParquetRecordReader when input split
> is null
> ---
>
> Key: HIVE-17961
> URL: https://issues.apache.org/jira/browse/HIVE-17961
> Project: Hive
> Issue Type: Sub-task
> Components: Hive
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17961.01.patch, HIVE-17961.02.patch
>
>
> HIVE-16465 introduces the regression which causes a NPE during initialize of
> the vectorized reader when input split is null. This was already fixed in
> HIVE-15718 but got exposed again we refactored for HIVE-16465. We should also
> add a test case to catch such regressions in the future.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17229) HiveMetastore HMSHandler locks during initialization, even though its static variable threadPool is not null
[
https://issues.apache.org/jira/browse/HIVE-17229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241438#comment-16241438
]
Rajesh Balamohan commented on HIVE-17229:
-
Patch lgtm in terms of reducing the locking contention when {{threadPool}} is
already initialized.
[~yuan_zac] - When you say "locks during initialization", does it hang during
init phase?. Or is the intent to reduce the locking contention?
> HiveMetastore HMSHandler locks during initialization, even though its static
> variable threadPool is not null
>
>
> Key: HIVE-17229
> URL: https://issues.apache.org/jira/browse/HIVE-17229
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Zac Zhou
>Assignee: Zac Zhou
> Attachments: HIVE-17229.2.patch, HIVE-17229.patch
>
>
> A thread pool is used to accelerate adding partitions operation, since
> [HIVE-13901|https://issues.apache.org/jira/browse/HIVE-13901].
> However, HMSHandler needs a lock during initialization every time, even
> though its static variable threadPool is not null
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17942) HiveAlterHandler not using conf from threadlocal
[
https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241431#comment-16241431
]
Hive QA commented on HIVE-17942:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896264/HIVE-17942.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 8 failed/errored test(s), 11368 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7670/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7670/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7670/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 8 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896264 - PreCommit-HIVE-Build
> HiveAlterHandler not using conf from threadlocal
>
>
> Key: HIVE-17942
> URL: https://issues.apache.org/jira/browse/HIVE-17942
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.1.1
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
> Fix For: 3.0.0
>
> Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch
>
>
> When HiveAlterHandler looks for conf, it is not getting the one from thread
> local. So, local changes are not visible.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17985) When check the partitions size in the partitioned table, it will throw NullPointerException
[
https://issues.apache.org/jira/browse/HIVE-17985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241425#comment-16241425
]
wan kun commented on HIVE-17985:
Hi [~kgyrtkirk] ,thank you for you reply!
I have reproduce this issue with qtest table srcpart.
{code:sql}
set hive.limit.query.max.table.partition=1;
CREATE TABLE srcpart_like (key STRING COMMENT 'default', value STRING COMMENT
'default')
PARTITIONED BY (ds STRING, hr STRING)
STORED AS TEXTFILE;
INSERT OVERWRITE TABLE srcpart_like PARTITION (ds='2008-04-08', hr='12')
select key,value from srcpart;
{code}
> When check the partitions size in the partitioned table, it will throw
> NullPointerException
>
>
> Key: HIVE-17985
> URL: https://issues.apache.org/jira/browse/HIVE-17985
> Project: Hive
> Issue Type: Bug
> Components: Parser, Physical Optimizer
>Affects Versions: 1.2.2, 2.3.0, 3.0.0
>Reporter: wan kun
>Assignee: wan kun
> Fix For: 3.0.0
>
> Attachments: HIVE-17985-branch-1.2.patch,
> HIVE-17985-branch-2.3.patch, HIVE-17985.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> When the hive.limit.query.max.table.partition parameter is set, the
> SemanticAnalyzer will throw NullPointerException.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17902) add a notions of default pool and unmanaged mapping part 1
[ https://issues.apache.org/jira/browse/HIVE-17902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241414#comment-16241414 ] Sergey Shelukhin commented on HIVE-17902: - [~prasanth_j] can you take a look at https://reviews.apache.org/r/63346/diff/2-10/ ? :) > add a notions of default pool and unmanaged mapping part 1 > -- > > Key: HIVE-17902 > URL: https://issues.apache.org/jira/browse/HIVE-17902 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17902.01.patch, HIVE-17902.02.patch, > HIVE-17902.03.patch, HIVE-17902.04.patch, HIVE-17902.05.patch, > HIVE-17902.06.patch, HIVE-17902.07.patch, HIVE-17902.08.patch, > HIVE-17902.patch > > > This is needed to map queries between WM and non-WM execution -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17902) add a notions of default pool and unmanaged mapping part 1
[ https://issues.apache.org/jira/browse/HIVE-17902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17902: Attachment: HIVE-17902.08.patch Rebasing again. > add a notions of default pool and unmanaged mapping part 1 > -- > > Key: HIVE-17902 > URL: https://issues.apache.org/jira/browse/HIVE-17902 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17902.01.patch, HIVE-17902.02.patch, > HIVE-17902.03.patch, HIVE-17902.04.patch, HIVE-17902.05.patch, > HIVE-17902.06.patch, HIVE-17902.07.patch, HIVE-17902.08.patch, > HIVE-17902.patch > > > This is needed to map queries between WM and non-WM execution -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17714) move custom SerDe schema considerations into metastore from QL
[ https://issues.apache.org/jira/browse/HIVE-17714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241398#comment-16241398 ] Vihang Karajgaonkar commented on HIVE-17714: Hi [~sershe] I am still looking at the AvroSerDe (and SerDes/ObjectInspectors/TypeInfos in general), so may be I don't understand the big picture correctly but here are my thoughts: I am not sure if changing all the metastore APIs to fallback on getting the schema from Deserializer is the way forward. This will create strong dependency with the Hive source code and make the metastore separation work largely irrelevant (in the sense, you won't be able to use standalone metastore with having hive jars in the classpath). I like the idea of looking if we can move the Deserializer and friends to some common project (storage-api?) or metastore instead. But I think Alan had investigated that early on in his work and it was not trivial. SerDes, TypeInfo and ObjectInspector are all intertwined such that we cannot move one out without the others if I understand it right. I am not sure how it makes sense from design perspective for metastore to serve something which it doesn't know in the first place and has to go an external source to fetch that information. Its not really a metastore if it doesn't store metadata isn't it. Do you know what was the motivation of using this way to get the fields information from external sources (urls). > move custom SerDe schema considerations into metastore from QL > -- > > Key: HIVE-17714 > URL: https://issues.apache.org/jira/browse/HIVE-17714 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Alan Gates > > Columns in metastore for tables that use external schema don't have the type > information (since HIVE-11985) and may be entirely inconsistent (since > forever, due to issues like HIVE-17713; or for SerDes that allow an URL for > the schema, due to a change in the underlying file). > Currently, if you trace the usage of ConfVars.SERDESUSINGMETASTOREFORSCHEMA, > and to MetaStoreUtils.getFieldsFromDeserializer, you'd see that the code in > QL handles this in Hive. So, for the most part metastore just returns > whatever is stored for columns in the database. > One exception appears to be get_fields_with_environment_context, which is > interesting... so getTable will return incorrect columns (potentially), but > get_fields/get_schema will return correct ones from SerDe as far as I can > tell. > As part of separating the metastore, we should make sure all the APIs return > the correct schema for the columns; it's not a good idea to have everyone > reimplement getFieldsFromDeserializer. > Note: this should also remove a flag introduced in HIVE-17731 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17229) HiveMetastore HMSHandler locks during initialization, even though its static variable threadPool is not null
[ https://issues.apache.org/jira/browse/HIVE-17229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241364#comment-16241364 ] Ashutosh Chauhan commented on HIVE-17229: - [~rajesh.balamohan] Can you please take a look? > HiveMetastore HMSHandler locks during initialization, even though its static > variable threadPool is not null > > > Key: HIVE-17229 > URL: https://issues.apache.org/jira/browse/HIVE-17229 > Project: Hive > Issue Type: Improvement > Components: Metastore >Reporter: Zac Zhou >Assignee: Zac Zhou > Attachments: HIVE-17229.2.patch, HIVE-17229.patch > > > A thread pool is used to accelerate adding partitions operation, since > [HIVE-13901|https://issues.apache.org/jira/browse/HIVE-13901]. > However, HMSHandler needs a lock during initialization every time, even > though its static variable threadPool is not null -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[
https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241362#comment-16241362
]
Hive QA commented on HIVE-17767:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896253/HIVE-17767.5.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 9 failed/errored test(s), 11354 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[spark_explainuser_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query14]
(batchId=243)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query23]
(batchId=243)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7669/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7669/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7669/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 9 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896253 - PreCommit-HIVE-Build
> Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
> ---
>
> Key: HIVE-17767
> URL: https://issues.apache.org/jira/browse/HIVE-17767
> Project: Hive
> Issue Type: Improvement
> Components: Query Planning
>Reporter: Vineet Garg
>Assignee: Vineet Garg
> Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch,
> HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch
>
>
> Currently such queries are written into group by + inner join with value
> generator and is inefficient. Value generator consists of join with outer
> query to fetch all correlated values. This value generator could be
> completely eliminated if such queries are instead rewritten into LEFT SEMI
> JOIN.
> Note that to do this first hive need to support LEFT SEMI JOIN with non-equi
> condition (HIVE-17766).
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17259) Hive JDBC does not recognize UNIONTYPE columns
[
https://issues.apache.org/jira/browse/HIVE-17259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241359#comment-16241359
]
Ashutosh Chauhan commented on HIVE-17259:
-
[~pvillard] Patch is incorrectly generated. It deletes (non-existent) lines
instead of adding them.
> Hive JDBC does not recognize UNIONTYPE columns
> --
>
> Key: HIVE-17259
> URL: https://issues.apache.org/jira/browse/HIVE-17259
> Project: Hive
> Issue Type: Bug
> Components: Beeline, JDBC
> Environment: Hive 1.2.1000.2.6.1.0-129
> Beeline version 1.2.1000.2.6.1.0-129 by Apache Hive
>Reporter: Pierre Villard
>Assignee: Pierre Villard
> Attachments: HIVE-17259.patch
>
>
> Hive JDBC does not recognize UNIONTYPE columns.
> I've an external table backed by an avro schema containing a union type field.
> {noformat}
> "name" : "value",
> "type" : [ "int", "string", "null" ]
> {noformat}
> When describing the table I've:
> {noformat}
> describe test_table;
> +---+---+--+--+
> | col_name | data_type
> | comment |
> +---+---+--+--+
> | description | string
> | |
> | name | string
> | |
> | value | uniontype
> | |
> +---+---+--+--+
> {noformat}
> When doing a select query over the data using the Hive CLI, it works:
> {noformat}
> hive> select value from test_table;
> OK
> {0:10}
> {0:10}
> {0:9}
> {0:9}
> ...
> {noformat}
> But when using beeline, it fails:
> {noformat}
> 0: jdbc:hive2://> select * from test_table;
> Error: Unrecognized column type: UNIONTYPE (state=,code=0)
> {noformat}
> By applying the patch provided with this JIRA, the command succeeds and
> return the expected output.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241344#comment-16241344
]
Steve Yeom edited comment on HIVE-17856 at 11/7/17 2:08 AM:
4) is actually a good comment.
The current patch 1 has not considered IOW union all case yet. So I will check
and possibly add/modify the code. Also the getAcidState() if needed..
was (Author: steveyeom2017):
4) is actually a good comment.
The current patch 1 has not considered IOW union all case yet. So I will check
and possibly add/modify the code.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241344#comment-16241344
]
Steve Yeom commented on HIVE-17856:
---
4) is actually a good comment.
The current patch 1 has not considered IOW union all case yet. So I will check
and possibly add/modify the code.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17376) Upgrade snappy version to 1.1.4
[ https://issues.apache.org/jira/browse/HIVE-17376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241343#comment-16241343 ] Ashutosh Chauhan commented on HIVE-17376: - +1 > Upgrade snappy version to 1.1.4 > --- > > Key: HIVE-17376 > URL: https://issues.apache.org/jira/browse/HIVE-17376 > Project: Hive > Issue Type: Improvement > Components: Hive >Affects Versions: 3.0.0 >Reporter: Aihua Xu >Assignee: Aihua Xu > Attachments: HIVE-17376.1.patch > > > Upgrade the snappy java version to 1.1.4. The older version has some issues > like memory leak (https://github.com/xerial/snappy-java/issues/91). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241341#comment-16241341
]
Eugene Koifman commented on HIVE-17856:
---
2) Everything in Acid relies on AcidUtils.getAcidState() - it has the logic to
skip deltas that are subsumed by a base as well as paying attention to
ValidTxnList. From what I've seen MM usually tries to use it's own logic for
figuring out the list of dirs it should read. One caveat is that for full acid
the structure is always table/partition/delta|base/buckets so it wouldn't
handle the Union subdirs anyway.
4b) good question - I suspect IOW + Union All may not work for full acid
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-17994: --- Priority: Minor (was: Major) > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V >Priority: Minor > Attachments: vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-17994: --- Description: On machines with slower NUMA, the hashmap lookup for TypeInfo::getPrimitiveCategory is the slowest part of the vectorized serialization loops. The static object references run hot with the NUMA access speeds penalizing half the threads. This lookup is done for every column, for every row - though vectorization enforces that this type cannot change at all. !vec-serialize-hashmap.png! was: On machines with slower NUMA, the hashmap lookup for TypeInfo::getPrimitiveCategory is the slowest part of the vectorized serialization loops. The static object references run hot with the NUMA access speeds penalizing half the threads. !vec-serialize-hashmap.png! > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V > Attachments: vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > This lookup is done for every column, for every row - though vectorization > enforces that this type cannot change at all. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17994) Vectorization: Serialization bottlenecked on irrelevant hashmap lookup
[ https://issues.apache.org/jira/browse/HIVE-17994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gopal V updated HIVE-17994: --- Attachment: vec-serialize-hashmap.png > Vectorization: Serialization bottlenecked on irrelevant hashmap lookup > -- > > Key: HIVE-17994 > URL: https://issues.apache.org/jira/browse/HIVE-17994 > Project: Hive > Issue Type: Bug >Reporter: Gopal V > Attachments: vec-serialize-hashmap.png > > > On machines with slower NUMA, the hashmap lookup for > TypeInfo::getPrimitiveCategory is the slowest part of the vectorized > serialization loops. The static object references run hot with the NUMA > access speeds penalizing half the threads. > !vec-serialize-hashmap.png! -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17904) handle internal Tez AM restart in registry and WM
[ https://issues.apache.org/jira/browse/HIVE-17904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17904: Attachment: HIVE-17904.patch > handle internal Tez AM restart in registry and WM > - > > Key: HIVE-17904 > URL: https://issues.apache.org/jira/browse/HIVE-17904 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17904.patch, HIVE-17904.patch > > > After the plan update patch is committed. The current code doesn't account > very well for it; registry may have races, and an event needs to be added to > WM when some AM resets, at least to make sure we discard the update errors > that pertain to the old AM. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16917) HiveServer2 guard rails - Limit concurrent connections from user
[ https://issues.apache.org/jira/browse/HIVE-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16917: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Test failures are unrelated. Committed to master. Thanks for the reviews! > HiveServer2 guard rails - Limit concurrent connections from user > > > Key: HIVE-16917 > URL: https://issues.apache.org/jira/browse/HIVE-16917 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Prasanth Jayachandran > Fix For: 3.0.0 > > Attachments: HIVE-16917.1.patch, HIVE-16917.2.patch, > HIVE-16917.3.patch, HIVE-16917.4.patch, HIVE-16917.5.patch > > > Rogue applications can make HS2 unusable for others by making too many > connections at a time. > HS2 should start rejecting the number of connections from a user, after it > has reached a configurable threshold. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16917) HiveServer2 guard rails - Limit concurrent connections from user
[
https://issues.apache.org/jira/browse/HIVE-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241277#comment-16241277
]
Hive QA commented on HIVE-16917:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896252/HIVE-16917.5.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 7 failed/errored test(s), 11367 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7668/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7668/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7668/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 7 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896252 - PreCommit-HIVE-Build
> HiveServer2 guard rails - Limit concurrent connections from user
>
>
> Key: HIVE-16917
> URL: https://issues.apache.org/jira/browse/HIVE-16917
> Project: Hive
> Issue Type: New Feature
> Components: HiveServer2
>Reporter: Thejas M Nair
>Assignee: Prasanth Jayachandran
> Attachments: HIVE-16917.1.patch, HIVE-16917.2.patch,
> HIVE-16917.3.patch, HIVE-16917.4.patch, HIVE-16917.5.patch
>
>
> Rogue applications can make HS2 unusable for others by making too many
> connections at a time.
> HS2 should start rejecting the number of connections from a user, after it
> has reached a configurable threshold.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17967) Move HiveMetaStore class
[ https://issues.apache.org/jira/browse/HIVE-17967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-17967: -- Attachment: HIVE-17967.3.patch Third patch that addresses one of the earlier unit test failures. > Move HiveMetaStore class > > > Key: HIVE-17967 > URL: https://issues.apache.org/jira/browse/HIVE-17967 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Labels: pull-request-available > Attachments: HIVE-17967.2.patch, HIVE-17967.3.patch, HIVE-17967.patch > > > We need to move HiveMetaStore and a few tightly integrated classes into the > standalone metastore. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16827) Merge stats task and column stats task into a single task
[
https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241209#comment-16241209
]
Hive QA commented on HIVE-16827:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896224/HIVE-16827.05wip12.patch
{color:green}SUCCESS:{color} +1 due to 17 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 37 failed/errored test(s), 11361 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[explain]
(batchId=246)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_default] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_const]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_6]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_if_expr_2]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_like_2]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_reference_windowed]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin2]
(batchId=23)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_11]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_nway_join]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_6]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_expressions]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_streaming]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=101)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin7]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=244)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=244)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=244)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=244)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=244)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=244)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=244)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=244)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7667/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7667/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7667/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 37 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896224 - PreCommit-HIVE-Build
> Merge stats task and column stats task into a single task
> -
>
> Key: HIVE-16827
> URL: https://issues.apache.org/jira/browse/HIVE-16827
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>
[jira] [Updated] (HIVE-17904) handle internal Tez AM restart in registry and WM
[ https://issues.apache.org/jira/browse/HIVE-17904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17904: Attachment: HIVE-17904.patch The patch. Actually, onDestroy was doing an entirely wrong thing - ID->session mapping should be maintained for open sessions, it's unrelated to registry. Registry callback is now propagated to the session and everything handles the changing endpoint info. No need to have a WM event for that, WM can just get version from the time of the error and compare with current one to discard errors for old versions. > handle internal Tez AM restart in registry and WM > - > > Key: HIVE-17904 > URL: https://issues.apache.org/jira/browse/HIVE-17904 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Attachments: HIVE-17904.patch > > > After the plan update patch is committed. The current code doesn't account > very well for it; registry may have races, and an event needs to be added to > WM when some AM resets, at least to make sure we discard the update errors > that pertain to the old AM. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17979) Tez: Improve ReduceRecordSource passDownKey copying
[
https://issues.apache.org/jira/browse/HIVE-17979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gopal V updated HIVE-17979:
---
Attachment: HIVE-17979.2.patch
> Tez: Improve ReduceRecordSource passDownKey copying
> ---
>
> Key: HIVE-17979
> URL: https://issues.apache.org/jira/browse/HIVE-17979
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Gopal V
>Assignee: Gopal V
> Attachments: HIVE-17979.1.patch, HIVE-17979.2.patch
>
>
> Tez does not use a single Key stream for both sides of the join, so each
> input gets its own ReduceRecordSource
> {code}
> sources[tag] = new ReduceRecordSource();
> {code}
> And this means for each input stream, there's a deserialized key (because the
> tag is not part of the Key byte stream), this means for a 2-table join there
> are 2 ReduceRecordSource objects.
> This means that the passDownKey is only an optimization when the Key,
> List has more than 1 value in it. Otherwise the copy is entirely
> wasted CPU cycles, because it deserializes the entire row to extract the key
> and discards the row.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-16827) Merge stats task and column stats task into a single task
[ https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16827: Attachment: (was: HIVE-16827.05wip12.patch) > Merge stats task and column stats task into a single task > - > > Key: HIVE-16827 > URL: https://issues.apache.org/jira/browse/HIVE-16827 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Zoltan Haindrich > Attachments: HIVE-16827.01.patch, HIVE-16827.02.patch, > HIVE-16827.03.patch, HIVE-16827.04wip03.patch, HIVE-16827.04wip04.patch, > HIVE-16827.04wip05.patch, HIVE-16827.04wip06.patch, HIVE-16827.04wip09.patch, > HIVE-16827.04wip10.patch, HIVE-16827.05.patch, HIVE-16827.05wip01.patch, > HIVE-16827.05wip02.patch, HIVE-16827.05wip03.patch, HIVE-16827.05wip04.patch, > HIVE-16827.05wip05.patch, HIVE-16827.05wip08.patch, HIVE-16827.05wip10.patch, > HIVE-16827.05wip10.patch, HIVE-16827.05wip11.patch, HIVE-16827.05wip12.patch, > HIVE-16827.4.patch > > > Within the task, we can specify whether to compute basic stats only or column > stats only or both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16827) Merge stats task and column stats task into a single task
[ https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16827: Attachment: HIVE-16827.05.patch > Merge stats task and column stats task into a single task > - > > Key: HIVE-16827 > URL: https://issues.apache.org/jira/browse/HIVE-16827 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Zoltan Haindrich > Attachments: HIVE-16827.01.patch, HIVE-16827.02.patch, > HIVE-16827.03.patch, HIVE-16827.04wip03.patch, HIVE-16827.04wip04.patch, > HIVE-16827.04wip05.patch, HIVE-16827.04wip06.patch, HIVE-16827.04wip09.patch, > HIVE-16827.04wip10.patch, HIVE-16827.05.patch, HIVE-16827.05wip01.patch, > HIVE-16827.05wip02.patch, HIVE-16827.05wip03.patch, HIVE-16827.05wip04.patch, > HIVE-16827.05wip05.patch, HIVE-16827.05wip08.patch, HIVE-16827.05wip10.patch, > HIVE-16827.05wip10.patch, HIVE-16827.05wip11.patch, HIVE-16827.05wip12.patch, > HIVE-16827.4.patch > > > Within the task, we can specify whether to compute basic stats only or column > stats only or both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16827) Merge stats task and column stats task into a single task
[ https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16827: Attachment: HIVE-16827.05wip12.patch > Merge stats task and column stats task into a single task > - > > Key: HIVE-16827 > URL: https://issues.apache.org/jira/browse/HIVE-16827 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Zoltan Haindrich > Attachments: HIVE-16827.01.patch, HIVE-16827.02.patch, > HIVE-16827.03.patch, HIVE-16827.04wip03.patch, HIVE-16827.04wip04.patch, > HIVE-16827.04wip05.patch, HIVE-16827.04wip06.patch, HIVE-16827.04wip09.patch, > HIVE-16827.04wip10.patch, HIVE-16827.05wip01.patch, HIVE-16827.05wip02.patch, > HIVE-16827.05wip03.patch, HIVE-16827.05wip04.patch, HIVE-16827.05wip05.patch, > HIVE-16827.05wip08.patch, HIVE-16827.05wip10.patch, HIVE-16827.05wip10.patch, > HIVE-16827.05wip11.patch, HIVE-16827.05wip12.patch, HIVE-16827.05wip12.patch, > HIVE-16827.4.patch > > > Within the task, we can specify whether to compute basic stats only or column > stats only or both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[
https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241137#comment-16241137
]
Hive QA commented on HIVE-17954:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896183/HIVE-17954.01.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7666/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7666/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7666/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-11-06 22:52:07.518
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-7666/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-11-06 22:52:07.521
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
ddce801..d7d9665 master -> origin/master
e2d5d00..307f582 branch-2 -> origin/branch-2
8af7b86..25bc3c6 standalone-metastore -> origin/standalone-metastore
+ git reset --hard HEAD
HEAD is now at ddce801 HIVE-17907 : enable and apply resource plan commands in
HS2 (Sergey Shelukhin, reviewed by Prasanth Jayachandran)
+ git clean -f -d
Removing
standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/ObjectStore.java.orig
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at d7d9665 HIVE-17953: Metrics should move to destination
atomically (Alexander Kolbasov, reviewed by Sahil Takiar, Barna Zsombor Klara)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-11-06 22:52:15.502
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed: metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql:2
error: metastore/scripts/upgrade/derby/046-HIVE-17566.derby.sql: patch does not
apply
error: patch failed:
metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql:112
error: metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql: patch does
not apply
error: patch failed: metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql:16
error: metastore/scripts/upgrade/mssql/031-HIVE-17566.mssql.sql: patch does not
apply
error: patch failed:
metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql:612
error: metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql: patch does
not apply
error: patch failed: metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql:12
error: metastore/scripts/upgrade/mysql/046-HIVE-17566.mysql.sql: patch does not
apply
error: patch failed:
metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql:863
error: metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql: patch does
not apply
error: patch failed:
metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql:16
error: metastore/scripts/upgrade/oracle/046-HIVE-17566.oracle.sql: patch does
not apply
error: patch failed:
metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql:593
error: metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql: patch
does not apply
error: patch failed:
metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql:16
error: metastore/scripts/upgrade/postgres/045-HIVE-17566.postgres.sql: patch
does not apply
error: patch failed:
metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql:630
error: metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql: patch
does not apply
error: patch failed:
[jira] [Commented] (HIVE-17947) Concurrent inserts might fail for ACID table since HIVE-17526 on branch-1
[
https://issues.apache.org/jira/browse/HIVE-17947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241136#comment-16241136
]
Hive QA commented on HIVE-17947:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12895378/HIVE-17947.3-branch-1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 165 failed/errored test(s), 8123 tests
executed
*Failed tests:*
{noformat}
TestAcidOnTez - did not produce a TEST-*.xml file (likely timed out)
(batchId=376)
TestAdminUser - did not produce a TEST-*.xml file (likely timed out)
(batchId=358)
TestAuthorizationPreEventListener - did not produce a TEST-*.xml file (likely
timed out) (batchId=391)
TestAuthzApiEmbedAuthorizerInEmbed - did not produce a TEST-*.xml file (likely
timed out) (batchId=368)
TestAuthzApiEmbedAuthorizerInRemote - did not produce a TEST-*.xml file (likely
timed out) (batchId=374)
TestBeeLineWithArgs - did not produce a TEST-*.xml file (likely timed out)
(batchId=398)
TestCLIAuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=416)
TestClearDanglingScratchDir - did not produce a TEST-*.xml file (likely timed
out) (batchId=383)
TestClientSideAuthorizationProvider - did not produce a TEST-*.xml file (likely
timed out) (batchId=390)
TestCompactor - did not produce a TEST-*.xml file (likely timed out)
(batchId=379)
TestCreateUdfEntities - did not produce a TEST-*.xml file (likely timed out)
(batchId=378)
TestCustomAuthentication - did not produce a TEST-*.xml file (likely timed out)
(batchId=399)
TestDBTokenStore - did not produce a TEST-*.xml file (likely timed out)
(batchId=342)
TestDDLWithRemoteMetastoreSecondNamenode - did not produce a TEST-*.xml file
(likely timed out) (batchId=377)
TestDynamicSerDe - did not produce a TEST-*.xml file (likely timed out)
(batchId=345)
TestEmbeddedHiveMetaStore - did not produce a TEST-*.xml file (likely timed
out) (batchId=355)
TestEmbeddedThriftBinaryCLIService - did not produce a TEST-*.xml file (likely
timed out) (batchId=402)
TestFilterHooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=350)
TestFolderPermissions - did not produce a TEST-*.xml file (likely timed out)
(batchId=385)
TestHS2AuthzContext - did not produce a TEST-*.xml file (likely timed out)
(batchId=419)
TestHS2AuthzSessionContext - did not produce a TEST-*.xml file (likely timed
out) (batchId=420)
TestHS2ClearDanglingScratchDir - did not produce a TEST-*.xml file (likely
timed out) (batchId=406)
TestHS2ImpersonationWithRemoteMS - did not produce a TEST-*.xml file (likely
timed out) (batchId=407)
TestHiveAuthorizerCheckInvocation - did not produce a TEST-*.xml file (likely
timed out) (batchId=394)
TestHiveAuthorizerShowFilters - did not produce a TEST-*.xml file (likely timed
out) (batchId=393)
TestHiveHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=396)
TestHiveMetaStoreTxns - did not produce a TEST-*.xml file (likely timed out)
(batchId=370)
TestHiveMetaStoreWithEnvironmentContext - did not produce a TEST-*.xml file
(likely timed out) (batchId=360)
TestHiveMetaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=373)
TestHiveServer2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=422)
TestHiveServer2SessionTimeout - did not produce a TEST-*.xml file (likely timed
out) (batchId=423)
TestHiveSessionImpl - did not produce a TEST-*.xml file (likely timed out)
(batchId=403)
TestHs2Hooks - did not produce a TEST-*.xml file (likely timed out)
(batchId=375)
TestHs2HooksWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=451)
TestJdbcDriver2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=410)
TestJdbcMetadataApiAuth - did not produce a TEST-*.xml file (likely timed out)
(batchId=421)
TestJdbcWithLocalClusterSpark - did not produce a TEST-*.xml file (likely timed
out) (batchId=415)
TestJdbcWithMiniHS2 - did not produce a TEST-*.xml file (likely timed out)
(batchId=412)
TestJdbcWithMiniKdc - did not produce a TEST-*.xml file (likely timed out)
(batchId=448)
TestJdbcWithMiniKdcCookie - did not produce a TEST-*.xml file (likely timed
out) (batchId=447)
TestJdbcWithMiniKdcSQLAuthBinary - did not produce a TEST-*.xml file (likely
timed out) (batchId=445)
TestJdbcWithMiniKdcSQLAuthHttp - did not produce a TEST-*.xml file (likely
timed out) (batchId=450)
TestJdbcWithMiniMr - did not produce a TEST-*.xml file (likely timed out)
(batchId=411)
TestJdbcWithSQLAuthUDFBlacklist - did not produce a TEST-*.xml file (likely
timed out) (batchId=417)
TestJdbcWithSQLAuthorization - did not produce a TEST-*.xml file (likely timed
out) (batchId=418)
TestLocationQueries - did not produce a TEST-*.xml file (likely timed out)
(batchId=382)
TestMTQueries - did not produce a TEST-*.xml file (likely timed out)
[jira] [Commented] (HIVE-17983) Make the standalone metastore generate tarballs etc.
[ https://issues.apache.org/jira/browse/HIVE-17983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241135#comment-16241135 ] Thejas M Nair commented on HIVE-17983: -- bq. On the installation and upgrade scripts I have only copied the 2.3 and 3.0 installation scripts and 2.3->3.0 upgrade. My assumption is that the standalone metastore will be used with Hive 3.0 or later, so it doesn't make sense to copy all the older scripts. I copied in the 2.3 installation scripts so that we could test the upgrade procedure. While install scripts of old versions are not needed, I think it would be useful to have the upgrade scripts from old versions. We could limit it to starting from say 2.0.0 or 1.0.0, but given the 0 maintenance cost of older version upgrade files, we could go all the way. > Make the standalone metastore generate tarballs etc. > > > Key: HIVE-17983 > URL: https://issues.apache.org/jira/browse/HIVE-17983 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates > > In order to be separately installable the standalone metastore needs its own > tarballs, startup scripts, etc. All of the SQL installation and upgrade > scripts also need to move from metastore to standalone-metastore. > I also plan to create Dockerfiles for different database types so that > developers can test the SQL installation and upgrade scripts. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17874) Parquet vectorization fails on tables with complex columns when there are no projected columns
[
https://issues.apache.org/jira/browse/HIVE-17874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17874:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Parquet vectorization fails on tables with complex columns when there are no
> projected columns
> --
>
> Key: HIVE-17874
> URL: https://issues.apache.org/jira/browse/HIVE-17874
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17874.01-branch-2.patch, HIVE-17874.01.patch,
> HIVE-17874.02.patch, HIVE-17874.03.patch, HIVE-17874.04.patch,
> HIVE-17874.05.patch, HIVE-17874.06.patch, HIVE-17874.07-branch-2.patch,
> HIVE-17874.08-branch-2.patch
>
>
> When a parquet table contains an unsupported type like {{Map}}, {{LIST}} or
> {{UNION}} simple queries like {{select count(*) from table}} fails with
> {{unsupported type exception}} even though vectorized reader doesn't really
> need read the complex type into batches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17874) Parquet vectorization fails on tables with complex columns when there are no projected columns
[
https://issues.apache.org/jira/browse/HIVE-17874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241009#comment-16241009
]
Vihang Karajgaonkar commented on HIVE-17874:
Failures are unrelated. Committed to branch-2
> Parquet vectorization fails on tables with complex columns when there are no
> projected columns
> --
>
> Key: HIVE-17874
> URL: https://issues.apache.org/jira/browse/HIVE-17874
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17874.01-branch-2.patch, HIVE-17874.01.patch,
> HIVE-17874.02.patch, HIVE-17874.03.patch, HIVE-17874.04.patch,
> HIVE-17874.05.patch, HIVE-17874.06.patch, HIVE-17874.07-branch-2.patch,
> HIVE-17874.08-branch-2.patch
>
>
> When a parquet table contains an unsupported type like {{Map}}, {{LIST}} or
> {{UNION}} simple queries like {{select count(*) from table}} fails with
> {{unsupported type exception}} even though vectorized reader doesn't really
> need read the complex type into batches.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17908) LLAP External client not correctly handling killTask for pending requests
[
https://issues.apache.org/jira/browse/HIVE-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16241001#comment-16241001
]
Jason Dere commented on HIVE-17908:
---
I believe that was the case - cc [~sseth] to confirm. I'll try to test with
this patch on the cluster.
> LLAP External client not correctly handling killTask for pending requests
> -
>
> Key: HIVE-17908
> URL: https://issues.apache.org/jira/browse/HIVE-17908
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-17908.1.patch, HIVE-17908.2.patch,
> HIVE-17908.3.patch, HIVE-17908.4.patch, HIVE-17908.5.patch, HIVE-17908.6.patch
>
>
> Hitting "Timed out waiting for heartbeat for task ID" errors with the LLAP
> external client.
> HIVE-17393 fixed some of these errors, however it is also occurring because
> the client is not correctly handling the killTask notification when the
> request is accepted but still waiting for the first task heartbeat. In this
> situation the client should retry the request, similar to what the LLAP AM
> does. Current logic is ignoring the killTask in this situation, which results
> in a heartbeat timeout - no heartbeats are sent by LLAP because of the
> killTask notification.
> {noformat}
> 17/08/09 05:36:02 WARN TaskSetManager: Lost task 10.0 in stage 4.0 (TID 14,
> cn114-10.l42scl.hortonworks.com, executor 5): java.io.IOException: Received
> reader event error: Timed out waiting for heartbeat for task ID
> attempt_7739111832518812959_0005_0_00_10_0
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:178)
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:50)
> at
> org.apache.hadoop.hive.llap.LlapRowRecordReader.next(LlapRowRecordReader.java:121)
> at
> org.apache.hadoop.hive.llap.LlapRowRecordReader.next(LlapRowRecordReader.java:68)
> at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:266)
> at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:211)
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithKeys$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException:
> LlapTaskUmbilicalExternalClient(attempt_7739111832518812959_0005_0_00_10_0):
> Error while attempting to read chunk length
> at
> org.apache.hadoop.hive.llap.io.ChunkedInputStream.read(ChunkedInputStream.java:82)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.hasInput(LlapBaseRecordReader.java:267)
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:142)
> ... 22 more
> Caused by: java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects
[ https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240991#comment-16240991 ] Alan Gates commented on HIVE-17990: --- Committed patch for this to standalone-metastore branch. > Add Thrift and DB storage for Schema Registry objects > - > > Key: HIVE-17990 > URL: https://issues.apache.org/jira/browse/HIVE-17990 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: Adding-Schema-Registry-to-Metastore.pdf > > > This JIRA tracks changes to Thrift, RawStore, and DB scripts to support > objects in the Schema Registry. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17993) llap_acid_fast seems to be sensitive to something
[
https://issues.apache.org/jira/browse/HIVE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240966#comment-16240966
]
Sergey Shelukhin commented on HIVE-17993:
-
cc [~teddy.choi]
I think I already opened a JIRA for this somewhere...
> llap_acid_fast seems to be sensitive to something
> -
>
> Key: HIVE-17993
> URL: https://issues.apache.org/jira/browse/HIVE-17993
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>
> from time-to-time it seems like this test is failing...
> https://builds.apache.org/job/PreCommit-HIVE-Build/7663/testReport/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_llap_acid_fast_/history/
> from seemingly unrelated tickets: HIVE-17898 HIVE-17953 HIVE-17907 HIVE-16827
> this failure seems to be pretty bad; it looks like the results are all
> changed ; the last column's value is changed to {{-1645852809}} or
> {{1864027286}} instead of the random values.
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing llap_acid_fast.q
> 171,176c171,174
> < -838810013 1 1864027286
> < -595277064 1 -1645852809
> < -334595454 1 -1645852809
> < 185212032 1 -1645852809
> < 186967185 1 -1645852809
> < 241008004 1 -1645852809
> ---
> > -285355633 1 -1241163445
> > -109813638 1 -58941842
> > 164554497 1 1161977292
> > 199879534 1 123351087
> 178,179c176,178
> < 518213127 1 -1645852809
> < 584923170 1 -1645852809
> ---
> > 354670578 1 562841852
> > 455419170 1 1108177470
> > 665801232 1 480783141
> 422,424c421,422
> < -838810013 1 1864027286
> < -595277064 1 -1645852809
> < -334595454 1 -1645852809
> ---
> > -285355633 1 -1241163445
> > -109813638 1 -58941842
> 426,428c424,425
> < 185212032 1 -1645852809
> < 186967185 1 -1645852809
> < 241008004 1 -1645852809
> ---
> > 164554497 1 1161977292
> > 199879534 1 123351087
> 430,431c427,429
> < 518213127 1 -1645852809
> < 584923170 1 -1645852809
> ---
> > 354670578 1 562841852
> > 455419170 1 1108177470
> > 665801232 1 480783141
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15016: Attachment: HIVE-15016.10.patch patch-10: fix more test failures. > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.10.patch, HIVE-15016.2.patch, > HIVE-15016.3.patch, HIVE-15016.4.patch, HIVE-15016.5.patch, > HIVE-15016.6.patch, HIVE-15016.7.patch, HIVE-15016.8.patch, > HIVE-15016.9.patch, HIVE-15016.patch, Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15016: Status: Patch Available (was: In Progress) > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.10.patch, HIVE-15016.2.patch, > HIVE-15016.3.patch, HIVE-15016.4.patch, HIVE-15016.5.patch, > HIVE-15016.6.patch, HIVE-15016.7.patch, HIVE-15016.8.patch, > HIVE-15016.9.patch, HIVE-15016.patch, Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17921) Aggregation with struct in LLAP produces wrong result
[
https://issues.apache.org/jira/browse/HIVE-17921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-17921:
--
Description:
Consider
{noformat}
select ROW__ID, count(*) from over10k_orc_bucketed group by ROW__ID having
count(*) > 1;
{noformat}
in acid_vectorization_original.q (available since HIVE-17458)
when run using TestMiniLlapCliDriver produces "NULL, N" where N varies from run
to run.
The right answer is empty results set as can be seen by running
{noformat}
select ROW__ID, * from over10k_orc_bucketed where ROW__ID is null
{noformat}
in the same test.
This is with
{noformat}
set hive.vectorized.execution.enabled=true;
set hive.vectorized.row.identifier.enabled=true;
{noformat}
It fails with TestMiniLlapCliDriver but not TestMiniTezCliDriver. See
acid_vectorization_original_tez.q which has identical query.
was:
Consider
{noformat}
select ROW__ID, count(*) from over10k_orc_bucketed group by ROW__ID having
count(*) > 1;
{noformat}
in acid_vectorization_original.q (available since HIVE-17458)
when run using TestMiniLlapCliDriver produces "NULL, N" where N varies from run
to run.
The right answer is empty results set as can be seen by running
{noformat}
select ROW__ID, * from over10k_orc_bucketed where ROW__ID is null
{noformat}
in the same test.
> Aggregation with struct in LLAP produces wrong result
> -
>
> Key: HIVE-17921
> URL: https://issues.apache.org/jira/browse/HIVE-17921
> Project: Hive
> Issue Type: Sub-task
> Components: llap, Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>
> Consider
> {noformat}
> select ROW__ID, count(*) from over10k_orc_bucketed group by ROW__ID having
> count(*) > 1;
> {noformat}
> in acid_vectorization_original.q (available since HIVE-17458)
> when run using TestMiniLlapCliDriver produces "NULL, N" where N varies from
> run to run.
> The right answer is empty results set as can be seen by running
> {noformat}
> select ROW__ID, * from over10k_orc_bucketed where ROW__ID is null
> {noformat}
> in the same test.
> This is with
> {noformat}
> set hive.vectorized.execution.enabled=true;
> set hive.vectorized.row.identifier.enabled=true;
> {noformat}
> It fails with TestMiniLlapCliDriver but not TestMiniTezCliDriver. See
> acid_vectorization_original_tez.q which has identical query.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-15016) Run tests with Hadoop 3.0.0-beta1
[ https://issues.apache.org/jira/browse/HIVE-15016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-15016: Status: In Progress (was: Patch Available) > Run tests with Hadoop 3.0.0-beta1 > - > > Key: HIVE-15016 > URL: https://issues.apache.org/jira/browse/HIVE-15016 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0 >Reporter: Sergio Peña >Assignee: Aihua Xu > Attachments: HIVE-15016.2.patch, HIVE-15016.3.patch, > HIVE-15016.4.patch, HIVE-15016.5.patch, HIVE-15016.6.patch, > HIVE-15016.7.patch, HIVE-15016.8.patch, HIVE-15016.9.patch, HIVE-15016.patch, > Hadoop3Upstream.patch > > > Hadoop 3.0.0-alpha1 was released back on Sep/16 to allow other components run > tests against this new version before GA. > We should start running tests with Hive to validate compatibility against > Hadoop 3.0. > NOTE: The patch used to test must not be committed to Hive until Hadoop 3.0 > GA is released. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17969) Metastore to alter table in batches of partitions when renaming table
[
https://issues.apache.org/jira/browse/HIVE-17969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240953#comment-16240953
]
Hive QA commented on HIVE-17969:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896208/HIVE-17969.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 12 failed/errored test(s), 11355 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testAlterViewParititon
(batchId=211)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testAlterViewParititon
(batchId=213)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testAlterViewParititon
(batchId=209)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testAlterViewParititon
(batchId=208)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testAlterViewParititon
(batchId=218)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7664/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7664/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7664/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 12 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896208 - PreCommit-HIVE-Build
> Metastore to alter table in batches of partitions when renaming table
> -
>
> Key: HIVE-17969
> URL: https://issues.apache.org/jira/browse/HIVE-17969
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-17969.0.patch, HIVE-17969.1.patch, batched.png,
> hive9447OptimizationOnly.png, original.png
>
>
> I'm currently trying to speed up the {{alter table rename to}} feature of
> HMS. The recently submitted change (HIVE-9447) already helps a lot especially
> on Oracle HMS DBs.
> This time I intend to gain throughput independently of DB types by enabling
> HMS to execute this alter table command on batches of partitions (rather than
> 1by1)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17993) llap_acid_fast seems to be sensitive to something
[
https://issues.apache.org/jira/browse/HIVE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240949#comment-16240949
]
Zoltan Haindrich commented on HIVE-17993:
-
FYI: [~sershe]
> llap_acid_fast seems to be sensitive to something
> -
>
> Key: HIVE-17993
> URL: https://issues.apache.org/jira/browse/HIVE-17993
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>
> from time-to-time it seems like this test is failing...
> https://builds.apache.org/job/PreCommit-HIVE-Build/7663/testReport/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_llap_acid_fast_/history/
> from seemingly unrelated tickets: HIVE-17898 HIVE-17953 HIVE-17907 HIVE-16827
> this failure seems to be pretty bad; it looks like the results are all
> changed ; the last column's value is changed to {{-1645852809}} or
> {{1864027286}} instead of the random values.
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing llap_acid_fast.q
> 171,176c171,174
> < -838810013 1 1864027286
> < -595277064 1 -1645852809
> < -334595454 1 -1645852809
> < 185212032 1 -1645852809
> < 186967185 1 -1645852809
> < 241008004 1 -1645852809
> ---
> > -285355633 1 -1241163445
> > -109813638 1 -58941842
> > 164554497 1 1161977292
> > 199879534 1 123351087
> 178,179c176,178
> < 518213127 1 -1645852809
> < 584923170 1 -1645852809
> ---
> > 354670578 1 562841852
> > 455419170 1 1108177470
> > 665801232 1 480783141
> 422,424c421,422
> < -838810013 1 1864027286
> < -595277064 1 -1645852809
> < -334595454 1 -1645852809
> ---
> > -285355633 1 -1241163445
> > -109813638 1 -58941842
> 426,428c424,425
> < 185212032 1 -1645852809
> < 186967185 1 -1645852809
> < 241008004 1 -1645852809
> ---
> > 164554497 1 1161977292
> > 199879534 1 123351087
> 430,431c427,429
> < 518213127 1 -1645852809
> < 584923170 1 -1645852809
> ---
> > 354670578 1 562841852
> > 455419170 1 1108177470
> > 665801232 1 480783141
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17993) llap_acid_fast seems to be sensitive to something
[
https://issues.apache.org/jira/browse/HIVE-17993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zoltan Haindrich updated HIVE-17993:
Description:
from time-to-time it seems like this test is failing...
https://builds.apache.org/job/PreCommit-HIVE-Build/7663/testReport/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_llap_acid_fast_/history/
from seemingly unrelated tickets: HIVE-17898 HIVE-17953 HIVE-17907 HIVE-16827
this failure seems to be pretty bad; it looks like the results are all changed
; the last column's value is changed to {{-1645852809}} or {{1864027286}}
instead of the random values.
{code}
Client Execution succeeded but contained differences (error code = 1) after
executing llap_acid_fast.q
171,176c171,174
< -838810013 1 1864027286
< -595277064 1 -1645852809
< -334595454 1 -1645852809
< 185212032 1 -1645852809
< 186967185 1 -1645852809
< 241008004 1 -1645852809
---
> -285355633 1 -1241163445
> -109813638 1 -58941842
> 164554497 1 1161977292
> 199879534 1 123351087
178,179c176,178
< 518213127 1 -1645852809
< 584923170 1 -1645852809
---
> 354670578 1 562841852
> 455419170 1 1108177470
> 665801232 1 480783141
422,424c421,422
< -838810013 1 1864027286
< -595277064 1 -1645852809
< -334595454 1 -1645852809
---
> -285355633 1 -1241163445
> -109813638 1 -58941842
426,428c424,425
< 185212032 1 -1645852809
< 186967185 1 -1645852809
< 241008004 1 -1645852809
---
> 164554497 1 1161977292
> 199879534 1 123351087
430,431c427,429
< 518213127 1 -1645852809
< 584923170 1 -1645852809
---
> 354670578 1 562841852
> 455419170 1 1108177470
> 665801232 1 480783141
{code}
was:
from time-to-time it seems like this test is failing...
{code}
Client Execution succeeded but contained differences (error code = 1) after
executing llap_acid_fast.q
171,176c171,174
< -838810013 1 1864027286
< -595277064 1 -1645852809
< -334595454 1 -1645852809
< 185212032 1 -1645852809
< 186967185 1 -1645852809
< 241008004 1 -1645852809
---
> -285355633 1 -1241163445
> -109813638 1 -58941842
> 164554497 1 1161977292
> 199879534 1 123351087
178,179c176,178
< 518213127 1 -1645852809
< 584923170 1 -1645852809
---
> 354670578 1 562841852
> 455419170 1 1108177470
> 665801232 1 480783141
422,424c421,422
< -838810013 1 1864027286
< -595277064 1 -1645852809
< -334595454 1 -1645852809
---
> -285355633 1 -1241163445
> -109813638 1 -58941842
426,428c424,425
< 185212032 1 -1645852809
< 186967185 1 -1645852809
< 241008004 1 -1645852809
---
> 164554497 1 1161977292
> 199879534 1 123351087
430,431c427,429
< 518213127 1 -1645852809
< 584923170 1 -1645852809
---
> 354670578 1 562841852
> 455419170 1 1108177470
> 665801232 1 480783141
{code}
Issue Type: Bug (was: Improvement)
> llap_acid_fast seems to be sensitive to something
> -
>
> Key: HIVE-17993
> URL: https://issues.apache.org/jira/browse/HIVE-17993
> Project: Hive
> Issue Type: Bug
>Reporter: Zoltan Haindrich
>
> from time-to-time it seems like this test is failing...
> https://builds.apache.org/job/PreCommit-HIVE-Build/7663/testReport/org.apache.hadoop.hive.cli/TestMiniLlapLocalCliDriver/testCliDriver_llap_acid_fast_/history/
> from seemingly unrelated tickets: HIVE-17898 HIVE-17953 HIVE-17907 HIVE-16827
> this failure seems to be pretty bad; it looks like the results are all
> changed ; the last column's value is changed to {{-1645852809}} or
> {{1864027286}} instead of the random values.
> {code}
> Client Execution succeeded but contained differences (error code = 1) after
> executing llap_acid_fast.q
> 171,176c171,174
> < -838810013 1 1864027286
> < -595277064 1 -1645852809
> < -334595454 1 -1645852809
> < 185212032 1 -1645852809
> < 186967185 1 -1645852809
> < 241008004 1 -1645852809
> ---
> > -285355633 1 -1241163445
> > -109813638 1 -58941842
> > 164554497 1 1161977292
> > 199879534 1 123351087
> 178,179c176,178
> < 518213127 1 -1645852809
> < 584923170 1 -1645852809
> ---
> > 354670578 1 562841852
> > 455419170 1 1108177470
> > 665801232 1 480783141
> 422,424c421,422
> < -838810013 1 1864027286
> < -595277064 1 -1645852809
> < -334595454 1 -1645852809
> ---
> > -285355633 1 -1241163445
> > -109813638 1 -58941842
> 426,428c424,425
> < 185212032 1 -1645852809
> < 186967185 1 -1645852809
> < 241008004 1 -1645852809
> ---
> > 164554497 1 1161977292
> > 199879534 1 123351087
> 430,431c427,429
> < 518213127 1 -1645852809
> < 584923170 1 -1645852809
> ---
> > 354670578 1 562841852
> > 455419170 1 1108177470
> > 665801232 1 480783141
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17961) NPE during initialization of VectorizedParquetRecordReader when input split is null
[ https://issues.apache.org/jira/browse/HIVE-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vihang Karajgaonkar updated HIVE-17961: --- Attachment: HIVE-17961.02.patch The test failures were caused due to a separate constructor which had some duplicate code. The fix went in one constructor but not other. The second version of the patch removes the redundant constructor for tests. > NPE during initialization of VectorizedParquetRecordReader when input split > is null > --- > > Key: HIVE-17961 > URL: https://issues.apache.org/jira/browse/HIVE-17961 > Project: Hive > Issue Type: Sub-task > Components: Hive >Affects Versions: 3.0.0, 2.4.0 >Reporter: Vihang Karajgaonkar >Assignee: Vihang Karajgaonkar > Attachments: HIVE-17961.01.patch, HIVE-17961.02.patch > > > HIVE-16465 introduces the regression which causes a NPE during initialize of > the vectorized reader when input split is null. This was already fixed in > HIVE-15718 but got exposed again we refactored for HIVE-16465. We should also > add a test case to catch such regressions in the future. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240929#comment-16240929
]
Steve Yeom commented on HIVE-17856:
---
I am also carefully checking all the test failures to find any possible code
problem.
1. mm_all qtest was successful from my environment.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240925#comment-16240925
]
Steve Yeom commented on HIVE-17856:
---
Hey [~sershe] Thank you for the great review.
I can create a review board with the changes for 1) (that you suggested).
I have added comments for each item.
1) I will check and change if relevant for this patch. Esp., patch version 2.
2) The processForWriteIds() with the "// Find the abse, created for IOW"
has a call to AcidUtils.getAcidState(), which returns "dirInfo" and this
class
returns only valid deltas via "dirinfo.getCurrentDirectories()".
Obsolete ones are in "obsolete" list from dirInfo, which are cleaned by the
Compactor cleaner later.
3) I have run mm_all in my environment and it was clear and successful.
4) I will check further the items and come back with the possible code changes
for item 1).
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240887#comment-16240887
]
Sergey Shelukhin edited comment on HIVE-17856 at 11/6/17 9:02 PM:
--
Is it possible to post a review board review
(https://reviews.apache.org/r/new/, project hive-git) or a pull request
(whichever you prefer)?
I have some minor comments that I can post there. The ones though,
1) It should be possible to remove the code that deletes "other" deltas with
this patch, and probably to remove "isMatch" argument from IdPathFilter, since
it was only used for that.
2) On the read side, where it adds the base (" // Find the base, created for
IOW."), shouldn't it also do something to skip the deltas before the last
committed base? Maybe there's logic in ACID somewhere; it may be hard to reuse
directly given a different layout, but perhaps it can be duplicated or modified
cc [~ekoifman]
3) Tests appear to have failed (mm_all).
4) getValidPartitionsInPath passes in false for the new flag value, although it
is possible to have dynamic partitioning with IOW.
4a) In general though, one txn can only create either base or delta in each
given location (or multiple of one type), as far as I understand. So it may be
possible to change the filter/etc. to find both by txn number, instead of
passing the flag and only finding one type. Then, we can error out if both are
present at once.
4b) Actually [~ekoifman], that makes me wonder about IOW with union in full
ACID. ACID union insert creates two deltas; does ACID IOW with union create two
base directories?
was (Author: sershe):
Is it possible to post a review board review
(https://reviews.apache.org/r/new/, project hive-git) or a pull request
(whichever you prefer)?
I have some minor comments that I can post there. The ones though,
1) It should be possible to remove the code that deletes "other" deltas with
this patch, and probably to remove "isMatch" argument from IdPathFilter, since
it was only used for that.
2) On the read side, where it adds the base (" // Find the base, created for
IOW."), shouldn't it also do something to skip the deltas before the last
committed base? Maybe there's logic in ACID somewhere; it may be hard to reuse
directly given a different layout, but perhaps it can be duplicated or modified
cc [~ekoifman]
3) Tests appear to have failed (mm_all).
4) getValidPartitionsInPath passes in false for the new flag value, although it
is possible to have dynamic partitioning with IOW.
4a) In general though, one txn can only create either base or delta in each
given location (or multiple of one type), as far as I understand. So it may be
possible to change the filter/etc. to find both by txn number, instead of
passing the flag and only finding one type. Then, we can error out if both are
present at once.
4b) Actually [~ekoifman], that makes me wonder, about IOW with union work for
ACID? ACID union insert creates two deltas; does ACID IOW with union create two
base directories?
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3
[jira] [Comment Edited] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240887#comment-16240887
]
Sergey Shelukhin edited comment on HIVE-17856 at 11/6/17 9:01 PM:
--
Is it possible to post a review board review
(https://reviews.apache.org/r/new/, project hive-git) or a pull request
(whichever you prefer)?
I have some minor comments that I can post there. The ones though,
1) It should be possible to remove the code that deletes "other" deltas with
this patch, and probably to remove "isMatch" argument from IdPathFilter, since
it was only used for that.
2) On the read side, where it adds the base (" // Find the base, created for
IOW."), shouldn't it also do something to skip the deltas before the last
committed base? Maybe there's logic in ACID somewhere; it may be hard to reuse
directly given a different layout, but perhaps it can be duplicated or modified
cc [~ekoifman]
3) Tests appear to have failed (mm_all).
4) getValidPartitionsInPath passes in false for the new flag value, although it
is possible to have dynamic partitioning with IOW.
4a) In general though, one txn can only create either base or delta in each
given location (or multiple of one type), as far as I understand. So it may be
possible to change the filter/etc. to find both by txn number, instead of
passing the flag and only finding one type. Then, we can error out if both are
present at once.
4b) Actually [~ekoifman], that makes me wonder, about IOW with union work for
ACID? ACID union insert creates two deltas; does ACID IOW with union create two
base directories?
was (Author: sershe):
Is it possible to post a review board review
(https://reviews.apache.org/r/new/, project hive-git) or a pull request
(whichever you prefer)?
I have some minor comments that I can post there. The ones though,
1) It should be possible to remove the code that deletes "other" deltas with
this patch, and probably to remove "isMatch" argument from IdPathFilter, since
it was only used for that.
2) On the read side, where it adds the base (" // Find the base, created for
IOW."), shouldn't it also do something to skip the deltas before the last
committed base? Maybe there's logic in ACID somewhere; it may be hard to reuse
directly given a different layout, but perhaps it can be duplicated or modified
cc [~ekoifman]
3) Tests appear to have failed (mm_all).
4) getValidPartitionsInPath passes in false, although it is possible to have
dynamic partitioning with IOW.
4a) In general though, one txn can only create either base or delta in each
given location (or multiple of one type), as far as I understand. So it may be
possible to change the filter/etc. to find both by txn number, instead of
passing the flag and only finding one type. Then, we can error out if both are
present at once.
4b) Actually [~ekoifman], that makes me wonder, about IOW with union work for
ACID? ACID union insert creates two deltas; does ACID IOW with union create two
base directories?
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240887#comment-16240887
]
Sergey Shelukhin commented on HIVE-17856:
-
Is it possible to post a review board review
(https://reviews.apache.org/r/new/, project hive-git) or a pull request
(whichever you prefer)?
I have some minor comments that I can post there. The ones though,
1) It should be possible to remove the code that deletes "other" deltas with
this patch, and probably to remove "isMatch" argument from IdPathFilter, since
it was only used for that.
2) On the read side, where it adds the base (" // Find the base, created for
IOW."), shouldn't it also do something to skip the deltas before the last
committed base? Maybe there's logic in ACID somewhere; it may be hard to reuse
directly given a different layout, but perhaps it can be duplicated or modified
cc [~ekoifman]
3) Tests appear to have failed (mm_all).
4) getValidPartitionsInPath passes in false, although it is possible to have
dynamic partitioning with IOW.
4a) In general though, one txn can only create either base or delta in each
given location (or multiple of one type), as far as I understand. So it may be
possible to change the filter/etc. to find both by txn number, instead of
passing the flag and only finding one type. Then, we can error out if both are
present at once.
4b) Actually [~ekoifman], that makes me wonder, about IOW with union work for
ACID? ACID union insert creates two deltas; does ACID IOW with union create two
base directories?
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17953) Metrics should move to destination atomically
[
https://issues.apache.org/jira/browse/HIVE-17953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-17953:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master.
> Metrics should move to destination atomically
> -
>
> Key: HIVE-17953
> URL: https://issues.apache.org/jira/browse/HIVE-17953
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Alexander Kolbasov
>Assignee: Alexander Kolbasov
>Priority: Trivial
> Fix For: 3.0.0
>
> Attachments: HIVE-17953.03.patch
>
>
> HIVE-17563 reimplemented metrics using native nio interfaces. It used the
> assumption that{{Files.move()}} is atomic operation. It turns out that by
> default it isn't, unless {{ATOMIC_MOVE}} option is specified. Otherwise the
> destination file is unlinked and then the source file is copied.
> This may cause test failure since the file may be temporarily unavailable.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17942) HiveAlterHandler not using conf from threadlocal
[ https://issues.apache.org/jira/browse/HIVE-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-17942: --- Attachment: HIVE-17942.2.patch Incorporated comments from Vihang. > HiveAlterHandler not using conf from threadlocal > > > Key: HIVE-17942 > URL: https://issues.apache.org/jira/browse/HIVE-17942 > Project: Hive > Issue Type: Bug >Affects Versions: 2.1.1 >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani > Fix For: 3.0.0 > > Attachments: HIVE-17942.1.patch, HIVE-17942.2.patch > > > When HiveAlterHandler looks for conf, it is not getting the one from thread > local. So, local changes are not visible. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16827) Merge stats task and column stats task into a single task
[
https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240838#comment-16240838
]
Hive QA commented on HIVE-16827:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896224/HIVE-16827.05wip12.patch
{color:green}SUCCESS:{color} +1 due to 17 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 30 failed/errored test(s), 11361 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[explain]
(batchId=246)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_default] (batchId=81)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_const]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_6]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_if_expr_2]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_like_2]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_reference_windowed]
(batchId=31)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vectorized_mapjoin2]
(batchId=23)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_no_buckets]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_11]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_nway_join]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_6]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_expressions]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_streaming]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketmapjoin7]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=101)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[bucketmapjoin7]
(batchId=116)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7663/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7663/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7663/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 30 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896224 - PreCommit-HIVE-Build
> Merge stats task and column stats task into a single task
> -
>
> Key: HIVE-16827
> URL: https://issues.apache.org/jira/browse/HIVE-16827
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Zoltan Haindrich
> Attachments: HIVE-16827.01.patch, HIVE-16827.02.patch,
> HIVE-16827.03.patch, HIVE-16827.04wip03.patch, HIVE-16827.04wip04.patch,
> HIVE-16827.04wip05.patch, HIVE-16827.04wip06.patch, HIVE-16827.04wip09.patch,
> HIVE-16827.04wip10.patch, HIVE-16827.05wip01.patch, HIVE-16827.05wip02.patch,
> HIVE-16827.05wip03.patch, HIVE-16827.05wip04.patch, HIVE-16827.05wip05.patch,
> HIVE-16827.05wip08.patch, HIVE-16827.05wip10.patch, HIVE-16827.05wip10.patch,
> HIVE-16827.05wip11.patch, HIVE-16827.05wip12.patch, HIVE-16827.4.patch
>
>
> Within the task, we can specify
[jira] [Commented] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[ https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240841#comment-16240841 ] Sergey Shelukhin commented on HIVE-17954: - Can you post a RB of the patch without generated changes? Also, is it possible to change resource_plan.user/group_name to not use a nested structure? perhaps create ... mapping in (plan) (user) to (pool)? Also ordering should be optional. What is the error you are hitting? > Implement pool, user, group and trigger to pool management API's. > - > > Key: HIVE-17954 > URL: https://issues.apache.org/jira/browse/HIVE-17954 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-17954.01.patch > > > Implement the following commands: > -- Pool management. > CREATE POOL `resource_plan`.`pool_path` WITH > ALLOC_FRACTION `fraction` > QUERY_PARALLELISM `parallelism` > SCHEDULING_POLICY `policy`; > ALTER POOL `resource_plan`.`pool_path` SET > PATH = `new_path`, > ALLOC_FRACTION = `fraction`, > QUERY_PARALLELISM = `parallelism`, > SCHEDULING_POLICY = `policy`; > DROP POOL `resource_plan`.`pool_path`; > -- Trigger to pool mappings. > ALTER RESOURCE PLAN `resource_plan` > ADD TRIGGER `trigger_name` TO `pool_path`; > ALTER RESOURCE PLAN `resource_plan` > DROP TRIGGER `trigger_name` TO `pool_path`; > -- User/Group to pool mappings. > CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name` > TO `pool_path` WITH ORDERING `order_no`; > DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17908) LLAP External client not correctly handling killTask for pending requests
[
https://issues.apache.org/jira/browse/HIVE-17908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240793#comment-16240793
]
Sergey Shelukhin commented on HIVE-17908:
-
Hmm... new retry is only called from taskKilled. Is taskKilled going to be
called repeatedly for every failed attempt if the call fails multiple times? If
so +1
> LLAP External client not correctly handling killTask for pending requests
> -
>
> Key: HIVE-17908
> URL: https://issues.apache.org/jira/browse/HIVE-17908
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Jason Dere
>Assignee: Jason Dere
> Attachments: HIVE-17908.1.patch, HIVE-17908.2.patch,
> HIVE-17908.3.patch, HIVE-17908.4.patch, HIVE-17908.5.patch, HIVE-17908.6.patch
>
>
> Hitting "Timed out waiting for heartbeat for task ID" errors with the LLAP
> external client.
> HIVE-17393 fixed some of these errors, however it is also occurring because
> the client is not correctly handling the killTask notification when the
> request is accepted but still waiting for the first task heartbeat. In this
> situation the client should retry the request, similar to what the LLAP AM
> does. Current logic is ignoring the killTask in this situation, which results
> in a heartbeat timeout - no heartbeats are sent by LLAP because of the
> killTask notification.
> {noformat}
> 17/08/09 05:36:02 WARN TaskSetManager: Lost task 10.0 in stage 4.0 (TID 14,
> cn114-10.l42scl.hortonworks.com, executor 5): java.io.IOException: Received
> reader event error: Timed out waiting for heartbeat for task ID
> attempt_7739111832518812959_0005_0_00_10_0
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:178)
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:50)
> at
> org.apache.hadoop.hive.llap.LlapRowRecordReader.next(LlapRowRecordReader.java:121)
> at
> org.apache.hadoop.hive.llap.LlapRowRecordReader.next(LlapRowRecordReader.java:68)
> at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:266)
> at org.apache.spark.rdd.HadoopRDD$$anon$1.getNext(HadoopRDD.scala:211)
> at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithKeys$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.io.IOException:
> LlapTaskUmbilicalExternalClient(attempt_7739111832518812959_0005_0_00_10_0):
> Error while attempting to read chunk length
> at
> org.apache.hadoop.hive.llap.io.ChunkedInputStream.read(ChunkedInputStream.java:82)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
> at java.io.FilterInputStream.read(FilterInputStream.java:83)
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.hasInput(LlapBaseRecordReader.java:267)
> at
> org.apache.hadoop.hive.llap.LlapBaseRecordReader.next(LlapBaseRecordReader.java:142)
> ... 22 more
> Caused by: java.net.SocketException: Socket closed
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Attachment: HIVE-17767.5.patch > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Status: Patch Available (was: Open) > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch, HIVE-17767.5.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17767) Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN
[ https://issues.apache.org/jira/browse/HIVE-17767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-17767: --- Status: Open (was: Patch Available) > Rewrite correlated EXISTS/IN subqueries into LEFT SEMI JOIN > --- > > Key: HIVE-17767 > URL: https://issues.apache.org/jira/browse/HIVE-17767 > Project: Hive > Issue Type: Improvement > Components: Query Planning >Reporter: Vineet Garg >Assignee: Vineet Garg > Attachments: HIVE-17767.1.patch, HIVE-17767.2.patch, > HIVE-17767.3.patch, HIVE-17767.4.patch > > > Currently such queries are written into group by + inner join with value > generator and is inefficient. Value generator consists of join with outer > query to fetch all correlated values. This value generator could be > completely eliminated if such queries are instead rewritten into LEFT SEMI > JOIN. > Note that to do this first hive need to support LEFT SEMI JOIN with non-equi > condition (HIVE-17766). -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16917) HiveServer2 guard rails - Limit concurrent connections from user
[ https://issues.apache.org/jira/browse/HIVE-16917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-16917: - Attachment: HIVE-16917.5.patch Fixes test > HiveServer2 guard rails - Limit concurrent connections from user > > > Key: HIVE-16917 > URL: https://issues.apache.org/jira/browse/HIVE-16917 > Project: Hive > Issue Type: New Feature > Components: HiveServer2 >Reporter: Thejas M Nair >Assignee: Prasanth Jayachandran > Attachments: HIVE-16917.1.patch, HIVE-16917.2.patch, > HIVE-16917.3.patch, HIVE-16917.4.patch, HIVE-16917.5.patch > > > Rogue applications can make HS2 unusable for others by making too many > connections at a time. > HS2 should start rejecting the number of connections from a user, after it > has reached a configurable threshold. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17907) enable and apply resource plan commands in HS2
[ https://issues.apache.org/jira/browse/HIVE-17907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-17907: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > enable and apply resource plan commands in HS2 > -- > > Key: HIVE-17907 > URL: https://issues.apache.org/jira/browse/HIVE-17907 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin > Fix For: 3.0.0 > > Attachments: HIVE-17907.01.patch, HIVE-17907.02.patch, > HIVE-17907.02.patch, HIVE-17907.only.nogen.patch, HIVE-17907.patch > > > Enabling and applying the RP should only be runnable in HS2 with active WM. > Both should validate the full resource plan (or at least enable should; users > cannot modify the RP via normal means once enabled, but it might be worth > double checking since we have to fetch it anyway to apply). > Then, apply should propagate the resource plan to the WM instance. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17926) Support triggers for non-pool sessions
[ https://issues.apache.org/jira/browse/HIVE-17926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-17926: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed patch to master. Test failures are not related and are handled in HIVE-17841 addendum commit. > Support triggers for non-pool sessions > -- > > Key: HIVE-17926 > URL: https://issues.apache.org/jira/browse/HIVE-17926 > Project: Hive > Issue Type: Sub-task >Affects Versions: 3.0.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran > Fix For: 3.0.0 > > Attachments: HIVE-17926.1.patch, HIVE-17926.1.patch, > HIVE-17926.2.patch, HIVE-17926.3.patch, HIVE-17926.3.patch, HIVE-17926.4.patch > > > Current trigger implementation works only with tez session pools. In case > when tez sessions pools are not used, a new session gets created for every > query in which case trigger validation does not happen. It will be good to > support such one-off session case as well. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17948) Hive 2.3.2 Release Planning
[ https://issues.apache.org/jira/browse/HIVE-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240741#comment-16240741 ] Sahil Takiar commented on HIVE-17948: - I've cherry-picked them all onto branch-2.3 locally, put haven't pushed any of them yet. This patch was generated by combining all the cherry-picks together. So I'll apply the cherry-pick so that the git history is cleaner. > Hive 2.3.2 Release Planning > --- > > Key: HIVE-17948 > URL: https://issues.apache.org/jira/browse/HIVE-17948 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.2 >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Fix For: 2.3.2 > > Attachments: HIVE-17948-branch-2.3.patch, > HIVE-17948.2-branch-2.3.patch, HIVE-17948.3-branch-2.3.patch > > > Release planning for Hive 2.3.2 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17934) Merging Statistics are promoted to COMPLETE (most of the time)
[
https://issues.apache.org/jira/browse/HIVE-17934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240729#comment-16240729
]
Hive QA commented on HIVE-17934:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896207/HIVE-17934.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 60 failed/errored test(s), 11354 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[select_dummy_source]
(batchId=243)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_part]
(batchId=15)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer10]
(batchId=75)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer11]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer14]
(batchId=34)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer15]
(batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer5]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer7]
(batchId=21)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer8]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer9]
(batchId=6)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[select_dummy_source]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[stats_ppr_all]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_exists_having]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[subquery_in_having]
(batchId=57)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_view] (batchId=14)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[dynamic_semijoin_user_level]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_nullscan]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mapreduce1]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mapreduce2]
(batchId=144)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[reduce_deduplicate]
(batchId=148)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[auto_sortmerge_join_12]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketmapjoin1]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketpruning1]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[constprog_dpp]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_partition_pruning]
(batchId=155)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynamic_semijoin_reduction]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mapjoin_hint]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[metadataonly1]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[optimize_nullscan]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[ppd_union_view]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[semijoin_hint]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_in]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_multi]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_scalar]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[subquery_select]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_smb_empty]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[tez_union_group_by]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_gby2]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_windowing_streaming]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorized_dynamic_partition_pruning]
(batchId=155)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[optimize_nullscan]
(batchId=138)
[jira] [Updated] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects
[ https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-17990: -- Attachment: Adding-Schema-Registry-to-Metastore.pdf A design doc for the proposed approach. > Add Thrift and DB storage for Schema Registry objects > - > > Key: HIVE-17990 > URL: https://issues.apache.org/jira/browse/HIVE-17990 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates > Attachments: Adding-Schema-Registry-to-Metastore.pdf > > > This JIRA tracks changes to Thrift, RawStore, and DB scripts to support > objects in the Schema Registry. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17990) Add Thrift and DB storage for Schema Registry objects
[ https://issues.apache.org/jira/browse/HIVE-17990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-17990: - > Add Thrift and DB storage for Schema Registry objects > - > > Key: HIVE-17990 > URL: https://issues.apache.org/jira/browse/HIVE-17990 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates > > This JIRA tracks changes to Thrift, RawStore, and DB scripts to support > objects in the Schema Registry. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17989) Adding support for Schema Registry objects to standalone
[ https://issues.apache.org/jira/browse/HIVE-17989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates reassigned HIVE-17989: - > Adding support for Schema Registry objects to standalone > > > Key: HIVE-17989 > URL: https://issues.apache.org/jira/browse/HIVE-17989 > Project: Hive > Issue Type: New Feature > Components: Standalone Metastore >Reporter: Alan Gates >Assignee: Alan Gates > > The team building the [Hortonworks Schema > Registry|https://github.com/hortonworks/registry/] has expressed interest in > integrating it into the standalone metastore. This will allow users to have > one data catalog for table based and stream based data. This JIRA will serve > as an umbrella JIRA for tracking related issues. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16827) Merge stats task and column stats task into a single task
[ https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zoltan Haindrich updated HIVE-16827: Attachment: HIVE-16827.05wip12.patch > Merge stats task and column stats task into a single task > - > > Key: HIVE-16827 > URL: https://issues.apache.org/jira/browse/HIVE-16827 > Project: Hive > Issue Type: Sub-task >Reporter: Pengcheng Xiong >Assignee: Zoltan Haindrich > Attachments: HIVE-16827.01.patch, HIVE-16827.02.patch, > HIVE-16827.03.patch, HIVE-16827.04wip03.patch, HIVE-16827.04wip04.patch, > HIVE-16827.04wip05.patch, HIVE-16827.04wip06.patch, HIVE-16827.04wip09.patch, > HIVE-16827.04wip10.patch, HIVE-16827.05wip01.patch, HIVE-16827.05wip02.patch, > HIVE-16827.05wip03.patch, HIVE-16827.05wip04.patch, HIVE-16827.05wip05.patch, > HIVE-16827.05wip08.patch, HIVE-16827.05wip10.patch, HIVE-16827.05wip10.patch, > HIVE-16827.05wip11.patch, HIVE-16827.05wip12.patch, HIVE-16827.4.patch > > > Within the task, we can specify whether to compute basic stats only or column > stats only or both. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-16827) Merge stats task and column stats task into a single task
[
https://issues.apache.org/jira/browse/HIVE-16827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240640#comment-16240640
]
Hive QA commented on HIVE-16827:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896206/HIVE-16827.05wip11.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7661/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7661/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7661/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2017-11-06 18:06:41.568
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-7661/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2017-11-06 18:06:41.571
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
993a16c..31f547f master -> origin/master
+ git reset --hard HEAD
HEAD is now at 993a16c HIVE-17945: Support column projection for index access
when using Parquet Vectorization (Ferdinand Xu via Vihang Karajgaonkar)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 2 commits, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at 31f547f HIVE-17965 : Remove HIVELIMITTABLESCANPARTITION support
(Zoltan Haindrich via Ashutosh Chauhan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2017-11-06 18:06:46.278
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:7003
error: ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java: patch
does not apply
The patch does not appear to apply with p0, p1, or p2
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896206 - PreCommit-HIVE-Build
> Merge stats task and column stats task into a single task
> -
>
> Key: HIVE-16827
> URL: https://issues.apache.org/jira/browse/HIVE-16827
> Project: Hive
> Issue Type: Sub-task
>Reporter: Pengcheng Xiong
>Assignee: Zoltan Haindrich
> Attachments: HIVE-16827.01.patch, HIVE-16827.02.patch,
> HIVE-16827.03.patch, HIVE-16827.04wip03.patch, HIVE-16827.04wip04.patch,
> HIVE-16827.04wip05.patch, HIVE-16827.04wip06.patch, HIVE-16827.04wip09.patch,
> HIVE-16827.04wip10.patch, HIVE-16827.05wip01.patch, HIVE-16827.05wip02.patch,
> HIVE-16827.05wip03.patch, HIVE-16827.05wip04.patch, HIVE-16827.05wip05.patch,
> HIVE-16827.05wip08.patch, HIVE-16827.05wip10.patch, HIVE-16827.05wip10.patch,
> HIVE-16827.05wip11.patch, HIVE-16827.4.patch
>
>
> Within the task, we can specify whether to compute basic stats only or column
> stats only or both.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17948) Hive 2.3.2 Release Planning
[ https://issues.apache.org/jira/browse/HIVE-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240634#comment-16240634 ] Sergio Peña commented on HIVE-17948: I Build 7655 run with the patch on this jira. I triggered manually during the weekend and use this jira as a parameter. So based on flaky tests and the testNonAcidToAcidConversion02 failure that has been since 2.3.1, then we should continue forward on commiting this patch and continue with the release. Btw, are you going to apply the big patch or cherry-pick each of the commits on this patch? > Hive 2.3.2 Release Planning > --- > > Key: HIVE-17948 > URL: https://issues.apache.org/jira/browse/HIVE-17948 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.2 >Reporter: Sahil Takiar >Assignee: Sahil Takiar > Fix For: 2.3.2 > > Attachments: HIVE-17948-branch-2.3.patch, > HIVE-17948.2-branch-2.3.patch, HIVE-17948.3-branch-2.3.patch > > > Release planning for Hive 2.3.2 -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Resolved] (HIVE-17944) OrcSplit.canUseLlapIo() disables LLAP IO for non-vectorized acid reads
[ https://issues.apache.org/jira/browse/HIVE-17944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman resolved HIVE-17944. --- Resolution: Won't Fix subsumed by HIVE-17915 > OrcSplit.canUseLlapIo() disables LLAP IO for non-vectorized acid reads > -- > > Key: HIVE-17944 > URL: https://issues.apache.org/jira/browse/HIVE-17944 > Project: Hive > Issue Type: Sub-task > Components: llap, Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman > > As of HIVE-17458, the logic for Acid reads is enable LLAP IO for vectorized > read if VectorizedOrcAcidRowBatchReader can handle it. > and disable it for any non-vectorized acid read. This last part is on par > with HIVE-12631. > The reason is that LlapRecordReader creates VectorizedOrcAcidRowBatchReader > for acid + vectorization, but doesn't create any Acid aware reader w/o > vectorization. > cc [~sershe], [~teddy.choi] > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240628#comment-16240628
]
Steve Yeom commented on HIVE-17856:
---
Hi [~sershe] can you review the 01.patch?
Thank you,
Steve.
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key from
> intermediate;
> insert into table iow1_mm partition (key2)
> select key + 1 as k1, key from intermediate union all select key as k1, key
> from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key from intermediate union all select key + 4 as k1,
> key from intermediate;
> select * from iow1_mm order by key, key2;
> insert overwrite table iow1_mm partition (key2)
> select key + 3 as k1, key + 3 from intermediate union all select key + 2 as
> k1, key + 2 from intermediate;
> select * from iow1_mm order by key, key2;
> drop table iow1_mm;
> {noformat}
> {noformat}
> drop table simple_mm;
> create table simple_mm(key int) stored as orc tblproperties
> ("transactional"="true", "transactional_properties"="insert_only");
> insert into table simple_mm select key from intermediate;
> -insert overwrite table simple_mm select key from intermediate;
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17948) Hive 2.3.2 Release Planning
[
https://issues.apache.org/jira/browse/HIVE-17948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240626#comment-16240626
]
Sahil Takiar commented on HIVE-17948:
-
Thanks [~spena]. Was https://builds.apache.org/job/PreCommit-HIVE-Build/7655/
run with the patch attached to this JIRA?
I've seen {{TestSparkCliDriver.org.apache.hadoop.hive.cli.TestSparkCliDriver}}
fail in {{createSources}} in other places, with the same error - HIVE-15165,
HIVE-15289, HIVE-16561 - we've never been able to get to the root cause. So I
think its probably fine.
Thanks for cherry-picking HIVE-16312.
Since it looks like there are no new tests failures, do you think its safe to
commit this patch?
> Hive 2.3.2 Release Planning
> ---
>
> Key: HIVE-17948
> URL: https://issues.apache.org/jira/browse/HIVE-17948
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.3.2
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
> Fix For: 2.3.2
>
> Attachments: HIVE-17948-branch-2.3.patch,
> HIVE-17948.2-branch-2.3.patch, HIVE-17948.3-branch-2.3.patch
>
>
> Release planning for Hive 2.3.2
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Assigned] (HIVE-17915) Enable VectorizedOrcAcidRowBatchReader to be used with LLAP IO elevator over original acid files
[ https://issues.apache.org/jira/browse/HIVE-17915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-17915: - Assignee: Teddy Choi > Enable VectorizedOrcAcidRowBatchReader to be used with LLAP IO elevator over > original acid files > > > Key: HIVE-17915 > URL: https://issues.apache.org/jira/browse/HIVE-17915 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Teddy Choi >Priority: Critical > > Since HIVE-12631, LLAP IO can support Acid tables but when reading "original" > files. > HIVE-17458 enables VectorizedOrcAcidRowBatchReader to vectorize reads over > "original" files but not with LLAP IO. > Current implementation of _OrcSplit.canUseLlapIo()_ is the same as in > HIVE-12631. > This can/should be improved. There are 2 parts to this: > When a read of "original" file is performed such that data doesn't need to be > decorated with ROW__ID (see > __VectorizedOrcAcidRowBatchReader.canUseLlapForAcid()_) then > VectorizedOrcAcidRowBatchReader as of HIVE-17458 should be usable with LLAP > IO but when I tried it I got _ArrayIndexOutOfBoundsException_ in various > places of the stack. > This is the more important one. > The 2nd issue is that reading "original" acid files (when ROW__IDs are > needed) requires using > _org.apache.hadoop.hive.ql.io.orc.RecordReader.getRowNumber()_ in > __VectorizedOrcAcidRowBatchReader_ > This API is not available on the reader that _LlapRecordReader_ provides. > It would be better if getRowNumber() was available for performance as well as > simpler logic in the code. > cc [~sershe], [~teddy.choi] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17915) Enable VectorizedOrcAcidRowBatchReader to be used with LLAP IO elevator over original acid files
[ https://issues.apache.org/jira/browse/HIVE-17915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17915: -- Priority: Critical (was: Minor) > Enable VectorizedOrcAcidRowBatchReader to be used with LLAP IO elevator over > original acid files > > > Key: HIVE-17915 > URL: https://issues.apache.org/jira/browse/HIVE-17915 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Priority: Critical > > Since HIVE-12631, LLAP IO can support Acid tables but when reading "original" > files. > HIVE-17458 enables VectorizedOrcAcidRowBatchReader to vectorize reads over > "original" files but not with LLAP IO. > Current implementation of _OrcSplit.canUseLlapIo()_ is the same as in > HIVE-12631. > This can/should be improved. There are 2 parts to this: > When a read of "original" file is performed such that data doesn't need to be > decorated with ROW__ID (see > __VectorizedOrcAcidRowBatchReader.canUseLlapForAcid()_) then > VectorizedOrcAcidRowBatchReader as of HIVE-17458 should be usable with LLAP > IO but when I tried it I got _ArrayIndexOutOfBoundsException_ in various > places of the stack. > This is the more important one. > The 2nd issue is that reading "original" acid files (when ROW__IDs are > needed) requires using > _org.apache.hadoop.hive.ql.io.orc.RecordReader.getRowNumber()_ in > __VectorizedOrcAcidRowBatchReader_ > This API is not available on the reader that _LlapRecordReader_ provides. > It would be better if getRowNumber() was available for performance as well as > simpler logic in the code. > cc [~sershe], [~teddy.choi] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17915) Enable VectorizedOrcAcidRowBatchReader to be used with LLAP IO elevator over original acid files
[ https://issues.apache.org/jira/browse/HIVE-17915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-17915: -- Description: Since HIVE-12631, LLAP IO can support Acid tables but when reading "original" files. HIVE-17458 enables VectorizedOrcAcidRowBatchReader to vectorize reads over "original" files but not with LLAP IO. Current implementation of _OrcSplit.canUseLlapIo()_ is the same as in HIVE-12631. This can/should be improved. There are 2 parts to this: When a read of "original" file is performed such that data doesn't need to be decorated with ROW__ID (see __VectorizedOrcAcidRowBatchReader.canUseLlapForAcid()_) then VectorizedOrcAcidRowBatchReader as of HIVE-17458 should be usable with LLAP IO but when I tried it I got _ArrayIndexOutOfBoundsException_ in various places of the stack. This is the more important one. The 2nd issue is that reading "original" acid files (when ROW__IDs are needed) requires using _org.apache.hadoop.hive.ql.io.orc.RecordReader.getRowNumber()_ in __VectorizedOrcAcidRowBatchReader_ This API is not available on the reader that _LlapRecordReader_ provides. It would be better if getRowNumber() was available for performance as well as simpler logic in the code. cc [~sershe], [~teddy.choi] was: Reading "original" acid files requires using _org.apache.hadoop.hive.ql.io.orc.RecordReader.getRowNumber()_ in __VectorizedOrcAcidRowBatchReader_ This API is not available on the reader that _LlapRecordReader_ provides so _VectorizedOrcAcidRowBatchReader.canUseLlapForAcid()_ is used to disable LLAP IO in some corner cases. It would be better if getRowNumber() was available for performance as well as simpler logic in the code. This needs HIVE-17458 to be committed to make sense. cc [~sershe], [~teddy.choi] > Enable VectorizedOrcAcidRowBatchReader to be used with LLAP IO elevator over > original acid files > > > Key: HIVE-17915 > URL: https://issues.apache.org/jira/browse/HIVE-17915 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Priority: Minor > > Since HIVE-12631, LLAP IO can support Acid tables but when reading "original" > files. > HIVE-17458 enables VectorizedOrcAcidRowBatchReader to vectorize reads over > "original" files but not with LLAP IO. > Current implementation of _OrcSplit.canUseLlapIo()_ is the same as in > HIVE-12631. > This can/should be improved. There are 2 parts to this: > When a read of "original" file is performed such that data doesn't need to be > decorated with ROW__ID (see > __VectorizedOrcAcidRowBatchReader.canUseLlapForAcid()_) then > VectorizedOrcAcidRowBatchReader as of HIVE-17458 should be usable with LLAP > IO but when I tried it I got _ArrayIndexOutOfBoundsException_ in various > places of the stack. > This is the more important one. > The 2nd issue is that reading "original" acid files (when ROW__IDs are > needed) requires using > _org.apache.hadoop.hive.ql.io.orc.RecordReader.getRowNumber()_ in > __VectorizedOrcAcidRowBatchReader_ > This API is not available on the reader that _LlapRecordReader_ provides. > It would be better if getRowNumber() was available for performance as well as > simpler logic in the code. > cc [~sershe], [~teddy.choi] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-17856) MM tables - IOW is not ACID compliant
[
https://issues.apache.org/jira/browse/HIVE-17856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240483#comment-16240483
]
Hive QA commented on HIVE-17856:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12896162/HIVE-17856.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 26 failed/errored test(s), 11361 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[index_auto_mult_tables]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=62)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mm_all] (batchId=66)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[mm_all]
(batchId=147)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[unionDistinct_1]
(batchId=146)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_2]
(batchId=102)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[ct_noperm_loc]
(batchId=94)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[subquery_multi]
(batchId=111)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion02
(batchId=274)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
(batchId=274)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion02
(batchId=284)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3
(batchId=284)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAmPoolInteractions
(batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testApplyPlanQpChanges
(batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testApplyPlanUserMapping
(batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testAsyncSessionInitFailures
(batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testClusterFractions
(batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testDestroyAndReturn
(batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testQueueing
(batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReopen (batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuse (batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithDifferentPool
(batchId=281)
org.apache.hadoop.hive.ql.exec.tez.TestWorkloadManager.testReuseWithQueueing
(batchId=281)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=223)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/7660/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/7660/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-7660/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 26 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12896162 - PreCommit-HIVE-Build
> MM tables - IOW is not ACID compliant
> -
>
> Key: HIVE-17856
> URL: https://issues.apache.org/jira/browse/HIVE-17856
> Project: Hive
> Issue Type: Sub-task
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Steve Yeom
> Labels: mm-gap-1
> Attachments: HIVE-17856.1.patch
>
>
> The following tests were removed from mm_all during "integration"... I should
> have never allowed such manner of intergration.
> MM logic should have been kept intact until ACID logic could catch up. Alas,
> here we are.
> {noformat}
> drop table iow0_mm;
> create table iow0_mm(key int) tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow0_mm select key from intermediate;
> insert into table iow0_mm select key + 1 from intermediate;
> select * from iow0_mm order by key;
> insert overwrite table iow0_mm select key + 2 from intermediate;
> select * from iow0_mm order by key;
> drop table iow0_mm;
> drop table iow1_mm;
> create table iow1_mm(key int) partitioned by (key2 int)
> tblproperties("transactional"="true",
> "transactional_properties"="insert_only");
> insert overwrite table iow1_mm partition (key2)
> select key as k1, key from intermediate union all select key as k1, key
[jira] [Updated] (HIVE-17965) Remove HIVELIMITTABLESCANPARTITION support
[ https://issues.apache.org/jira/browse/HIVE-17965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-17965: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master. Thanks, Zoltan! > Remove HIVELIMITTABLESCANPARTITION support > -- > > Key: HIVE-17965 > URL: https://issues.apache.org/jira/browse/HIVE-17965 > Project: Hive > Issue Type: Improvement >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Trivial > Fix For: 3.0.0 > > Attachments: HIVE-17965.01.patch > > > HIVE-13884 marked it as deprecated -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17954) Implement pool, user, group and trigger to pool management API's.
[ https://issues.apache.org/jira/browse/HIVE-17954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-17954: - Status: Patch Available (was: Open) > Implement pool, user, group and trigger to pool management API's. > - > > Key: HIVE-17954 > URL: https://issues.apache.org/jira/browse/HIVE-17954 > Project: Hive > Issue Type: Sub-task >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash > Attachments: HIVE-17954.01.patch > > > Implement the following commands: > -- Pool management. > CREATE POOL `resource_plan`.`pool_path` WITH > ALLOC_FRACTION `fraction` > QUERY_PARALLELISM `parallelism` > SCHEDULING_POLICY `policy`; > ALTER POOL `resource_plan`.`pool_path` SET > PATH = `new_path`, > ALLOC_FRACTION = `fraction`, > QUERY_PARALLELISM = `parallelism`, > SCHEDULING_POLICY = `policy`; > DROP POOL `resource_plan`.`pool_path`; > -- Trigger to pool mappings. > ALTER RESOURCE PLAN `resource_plan` > ADD TRIGGER `trigger_name` TO `pool_path`; > ALTER RESOURCE PLAN `resource_plan` > DROP TRIGGER `trigger_name` TO `pool_path`; > -- User/Group to pool mappings. > CREATE USER|GROUP MAPPING `resource_plan`.`group_or_user_name` > TO `pool_path` WITH ORDERING `order_no`; > DROP USER|GROUP MAPPING `resource_plan`.`group_or_user_name`; -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-16736) General Improvements to BufferedRows
[
https://issues.apache.org/jira/browse/HIVE-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan updated HIVE-16736:
Resolution: Fixed
Fix Version/s: 3.0.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks, Beluga!
> General Improvements to BufferedRows
>
>
> Key: HIVE-16736
> URL: https://issues.apache.org/jira/browse/HIVE-16736
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HIVE-16736.1.patch, HIVE-16736.1.patch
>
>
> General improvements for {{BufferedRows.java}}. Use {{ArrayList}} instead of
> {{LinkedList}} to conserve memory for large data sets, prevent having to loop
> through the entire data set twice in {{normalizeWidths}} method, some
> simplifications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-16736) General Improvements to BufferedRows
[
https://issues.apache.org/jira/browse/HIVE-16736?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240455#comment-16240455
]
Ashutosh Chauhan commented on HIVE-16736:
-
+1
> General Improvements to BufferedRows
>
>
> Key: HIVE-16736
> URL: https://issues.apache.org/jira/browse/HIVE-16736
> Project: Hive
> Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Minor
> Attachments: HIVE-16736.1.patch, HIVE-16736.1.patch
>
>
> General improvements for {{BufferedRows.java}}. Use {{ArrayList}} instead of
> {{LinkedList}} to conserve memory for large data sets, prevent having to loop
> through the entire data set twice in {{normalizeWidths}} method, some
> simplifications.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-17985) When check the partitions size in the partitioned table, it will throw NullPointerException
[ https://issues.apache.org/jira/browse/HIVE-17985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16240441#comment-16240441 ] Zoltan Haindrich commented on HIVE-17985: - HIVE-17965 will remove this option from master for other branches it looks good +1 could you given an example how to reproduce this issue? I've tried to reproduce itbut only legit errors were outputted - I've not seen any NPE > When check the partitions size in the partitioned table, it will throw > NullPointerException > > > Key: HIVE-17985 > URL: https://issues.apache.org/jira/browse/HIVE-17985 > Project: Hive > Issue Type: Bug > Components: Parser, Physical Optimizer >Affects Versions: 1.2.2, 2.3.0, 3.0.0 >Reporter: wan kun >Assignee: wan kun > Fix For: 3.0.0 > > Attachments: HIVE-17985-branch-1.2.patch, > HIVE-17985-branch-2.3.patch, HIVE-17985.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > When the hive.limit.query.max.table.partition parameter is set, the > SemanticAnalyzer will throw NullPointerException. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17987) Certain metastore operation should use iterators of partitions over lists
[ https://issues.apache.org/jira/browse/HIVE-17987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-17987: -- Description: In HS2 side we have a PartitionIterable class to reduce memory load and use iterators of partitions instead of whole lists when querying them from HMS. Inside HMS there is no such feature and we should create a similar one. (e.g alter table calls that have to apply a modification to each and every partition of the table) > Certain metastore operation should use iterators of partitions over lists > - > > Key: HIVE-17987 > URL: https://issues.apache.org/jira/browse/HIVE-17987 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita > > In HS2 side we have a PartitionIterable class to reduce memory load and use > iterators of partitions instead of whole lists when querying them from HMS. > Inside HMS there is no such feature and we should create a similar one. (e.g > alter table calls that have to apply a modification to each and every > partition of the table) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17987) Certain metastore operation should use iterators of partitions over lists
[ https://issues.apache.org/jira/browse/HIVE-17987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita updated HIVE-17987: -- Environment: (was: In HS2 side we have a PartitionIterable class to reduce memory load and use iterators of partitions instead of whole lists when querying them from HMS. Inside HMS there is no such feature and we should create a similar one. (e.g alter table calls that have to apply a modification to each and every partition of the table)) > Certain metastore operation should use iterators of partitions over lists > - > > Key: HIVE-17987 > URL: https://issues.apache.org/jira/browse/HIVE-17987 > Project: Hive > Issue Type: Improvement >Reporter: Adam Szita >Assignee: Adam Szita > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-17987) Certain metastore operation should use iterators of partitions over lists
[ https://issues.apache.org/jira/browse/HIVE-17987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Szita reassigned HIVE-17987: - Assignee: Adam Szita > Certain metastore operation should use iterators of partitions over lists > - > > Key: HIVE-17987 > URL: https://issues.apache.org/jira/browse/HIVE-17987 > Project: Hive > Issue Type: Improvement > Environment: In HS2 side we have a PartitionIterable class to reduce > memory load and use iterators of partitions instead of whole lists when > querying them from HMS. Inside HMS there is no such feature and we should > create a similar one. (e.g alter table calls that have to apply a > modification to each and every partition of the table) >Reporter: Adam Szita >Assignee: Adam Szita > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-17969) Metastore to alter table in batches of partitions when renaming table
[
https://issues.apache.org/jira/browse/HIVE-17969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-17969:
--
Status: Patch Available (was: In Progress)
> Metastore to alter table in batches of partitions when renaming table
> -
>
> Key: HIVE-17969
> URL: https://issues.apache.org/jira/browse/HIVE-17969
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-17969.0.patch, HIVE-17969.1.patch, batched.png,
> hive9447OptimizationOnly.png, original.png
>
>
> I'm currently trying to speed up the {{alter table rename to}} feature of
> HMS. The recently submitted change (HIVE-9447) already helps a lot especially
> on Oracle HMS DBs.
> This time I intend to gain throughput independently of DB types by enabling
> HMS to execute this alter table command on batches of partitions (rather than
> 1by1)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17969) Metastore to alter table in batches of partitions when renaming table
[
https://issues.apache.org/jira/browse/HIVE-17969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-17969:
--
Attachment: HIVE-17969.1.patch
> Metastore to alter table in batches of partitions when renaming table
> -
>
> Key: HIVE-17969
> URL: https://issues.apache.org/jira/browse/HIVE-17969
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-17969.0.patch, HIVE-17969.1.patch, batched.png,
> hive9447OptimizationOnly.png, original.png
>
>
> I'm currently trying to speed up the {{alter table rename to}} feature of
> HMS. The recently submitted change (HIVE-9447) already helps a lot especially
> on Oracle HMS DBs.
> This time I intend to gain throughput independently of DB types by enabling
> HMS to execute this alter table command on batches of partitions (rather than
> 1by1)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-17969) Metastore to alter table in batches of partitions when renaming table
[
https://issues.apache.org/jira/browse/HIVE-17969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Adam Szita updated HIVE-17969:
--
Status: In Progress (was: Patch Available)
> Metastore to alter table in batches of partitions when renaming table
> -
>
> Key: HIVE-17969
> URL: https://issues.apache.org/jira/browse/HIVE-17969
> Project: Hive
> Issue Type: Improvement
> Components: Metastore
>Reporter: Adam Szita
>Assignee: Adam Szita
> Attachments: HIVE-17969.0.patch, HIVE-17969.1.patch, batched.png,
> hive9447OptimizationOnly.png, original.png
>
>
> I'm currently trying to speed up the {{alter table rename to}} feature of
> HMS. The recently submitted change (HIVE-9447) already helps a lot especially
> on Oracle HMS DBs.
> This time I intend to gain throughput independently of DB types by enabling
> HMS to execute this alter table command on batches of partitions (rather than
> 1by1)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
