[jira] [Commented] (HIVE-22150) HS2 allows setting system properties
[
https://issues.apache.org/jira/browse/HIVE-22150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967301#comment-16967301
]
Hui An commented on HIVE-22150:
---
[~thejas], [~vaibhgup] May you please review this? Thanks in advance.
> HS2 allows setting system properties
>
>
> Key: HIVE-22150
> URL: https://issues.apache.org/jira/browse/HIVE-22150
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Affects Versions: 4.0.0, 3.1.1
>Reporter: Craig Condit
>Assignee: Hui An
>Priority: Major
> Attachments: HIVE-22150.patch.1, HIVE-22150.patch.2
>
>
> HiveServer2 currently allows setting system properties, which is a problem
> when used in a multi-user environment.
> Connecting via beeline and executing the following demonstrates the issue:
> {noformat}
> 0: jdbc:hive2://serv1000.example.com:2181,serv> SET system:java.io.tmpdir;
> +-+
> | set |
> +-+
> | system:java.io.tmpdir=/tmp |
> +-+
> 1 row selected (0.018 seconds)
> 0: jdbc:hive2://serv1000.example.com:2181,serv> SET
> system:java.io.tmpdir=/tmp/attacker-dir;
> No rows affected (0.013 seconds)
> 0: jdbc:hive2://serv1000.example.com:2181,serv> SET system:java.io.tmpdir;
> +--+
> | set|
> +--+
> | system:java.io.tmpdir=/tmp/attacker-dir |
> +--+
> 1 row selected (0.019 seconds)
> {noformat}
> Any changes persist until HS2 is restarted, and affect all connected users.
> At the very least, this is a denial-of-service vector (verified by setting
> line.separator to a random string).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22444) Clean up Project POM Files
[
https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967182#comment-16967182
]
Hive QA commented on HIVE-22444:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984837/HIVE-22444.6.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 62 failed/errored test(s), 17519 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=299)
TestMiniHiveKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=299)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[buckets]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_hdfs]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_hdfs_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[explain]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[having]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore_nonpart]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse_nonpart]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_blobstore_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_empty_into_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_move]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_only]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_move_only]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join2]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[load_data]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[multiple_agg]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[multiple_db]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[nested_outer_join]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_buckets]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_format_nonpart]
(batchId=302)
[jira] [Commented] (HIVE-22444) Clean up Project POM Files
[
https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967180#comment-16967180
]
Hive QA commented on HIVE-22444:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
58s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
24s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} storage-api generated 0 new + 0 unchanged - 2 fixed
= 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
20s{color} | {color:green} serde in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} upgrade-acid in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
21s{color} | {color:green} root generated 0 new + 32 unchanged - 2 fixed = 32
total (was 34) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} custom-serde in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} custom-udfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} udf-classloader-udf1 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} udf-classloader-udf2 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} udf-vectorized-badexample in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} hcatalog-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} hive-blobstore in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} hive-jmh in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} hive-minikdc in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} hive-unit-hadoop2 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} qtest-accumulo in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} qtest-druid in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} qtest-kudu in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} qtest-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} test-serde in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
31s{color} | {color:green} util in the patch passed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
23s{color} |
[jira] [Commented] (HIVE-22455) Union branch removal rule does not kick in.
[ https://issues.apache.org/jira/browse/HIVE-22455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967155#comment-16967155 ] Julian Hyde commented on HIVE-22455: There are a bunch of rules in Calcite's class PruneEmptyRules that recognize empty relational expressions and simplify accordingly. > Union branch removal rule does not kick in. > --- > > Key: HIVE-22455 > URL: https://issues.apache.org/jira/browse/HIVE-22455 > Project: Hive > Issue Type: Improvement >Reporter: Steve Carlin >Priority: Major > > After the Calcite upgrade to 1.21, there is a rule where 2 branches of a > union have limit 0. This can be simplified. > This can be found in: union_assertion_type.q.out -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[
https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967148#comment-16967148
]
Hive QA commented on HIVE-21304:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12967940/HIVE-21304.05.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19291/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19291/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19291/
Messages:
{noformat}
This message was trimmed, see log for full details
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/optimize_nullscan.q.out' with
conflicts.
error: patch failed: ql/src/test/results/clientpositive/spark/sample10.q.out:319
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/spark/sample10.q.out' with
conflicts.
error: patch failed: ql/src/test/results/clientpositive/spark/sample8.q.out:45
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/spark/sample8.q.out'
cleanly.
error: patch failed:
ql/src/test/results/clientpositive/spark/smb_mapjoin_11.q.out:94
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/smb_mapjoin_11.q.out' with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/spark/transform_ppr1.q.out:69
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/spark/transform_ppr1.q.out' cleanly.
error: patch failed: ql/src/test/results/clientpositive/transform_ppr1.q.out:64
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/transform_ppr1.q.out'
cleanly.
Going to apply patch with: git apply -p0
/data/hiveptest/working/scratch/build.patch:426: trailing whitespace.
totalSize 4508
/data/hiveptest/working/scratch/build.patch:435: trailing whitespace.
totalSize 4508
/data/hiveptest/working/scratch/build.patch:444: trailing whitespace.
totalSize 4508
/data/hiveptest/working/scratch/build.patch:453: trailing whitespace.
totalSize 9015
/data/hiveptest/working/scratch/build.patch:462: trailing whitespace.
totalSize 9015
error: patch failed:
itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out:107
Falling back to three-way merge...
Applied patch to
'itests/hive-blobstore/src/test/results/clientpositive/insert_into_dynamic_partitions.q.out'
with conflicts.
error: patch failed:
itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out:125
Falling back to three-way merge...
Applied patch to
'itests/hive-blobstore/src/test/results/clientpositive/insert_overwrite_dynamic_partitions.q.out'
with conflicts.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java:135
Falling back to three-way merge...
Applied patch to
'ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java'
cleanly.
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java:1012
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/parse/TezCompiler.java'
cleanly.
error: patch failed: ql/src/test/results/clientpositive/acid_nullscan.q.out:62
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/acid_nullscan.q.out' with
conflicts.
error: patch failed:
ql/src/test/results/clientpositive/acid_table_stats.q.out:97
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/acid_table_stats.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/autoColumnStats_4.q.out:252
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/autoColumnStats_4.q.out'
with conflicts.
error: patch failed:
ql/src/test/results/clientpositive/beeline/smb_mapjoin_11.q.out:88
Falling back to three-way merge...
Applied patch to
'ql/src/test/results/clientpositive/beeline/smb_mapjoin_11.q.out' with
conflicts.
error: patch failed: ql/src/test/results/clientpositive/bucket1.q.out:39
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/bucket1.q.out' with
conflicts.
error: patch failed: ql/src/test/results/clientpositive/bucket2.q.out:38
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/bucket2.q.out' with
conflicts.
error: patch failed: ql/src/test/results/clientpositive/bucket3.q.out:39
Falling back to three-way merge...
[jira] [Commented] (HIVE-21485) Hive desc operation takes more than 100 seconds after upgrading from Hive 1.2.1 to 2.3.4
[
https://issues.apache.org/jira/browse/HIVE-21485?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967147#comment-16967147
]
Hive QA commented on HIVE-21485:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12963994/HIVE-21485.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19290/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19290/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19290/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-11-05 01:37:24.321
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-19290/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-11-05 01:37:24.324
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 164b78b HIVE-22429: Migrated clustered tables using
bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts (Ramesh
Kumar Thangarajan, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing ${project.basedir}/
Removing itests/${project.basedir}/
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 164b78b HIVE-22429: Migrated clustered tables using
bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts (Ramesh
Kumar Thangarajan, reviewed by Jesus Camacho Rodriguez)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-11-05 01:37:26.794
+ rm -rf ../yetus_PreCommit-HIVE-Build-19290
+ mkdir ../yetus_PreCommit-HIVE-Build-19290
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-19290
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-19290/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: ql/src/java/org/apache/hadoop/hive/ql/ddl/table/DescTableOperation.java:
does not exist in index
error: patch failed: ql/src/test/results/clientpositive/describe_table.q.out:388
Falling back to three-way merge...
Applied patch to 'ql/src/test/results/clientpositive/describe_table.q.out' with
conflicts.
error: src/java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in
index
error: src/java/org/apache/hadoop/hive/ql/ddl/table/DescTableOperation.java:
does not exist in index
error: src/test/queries/clientpositive/describe_table.q: does not exist in index
error: src/test/results/clientpositive/describe_table.q.out: does not exist in
index
error: java/org/apache/hadoop/hive/conf/HiveConf.java: does not exist in index
error: java/org/apache/hadoop/hive/ql/ddl/table/DescTableOperation.java: does
not exist in index
error: test/queries/clientpositive/describe_table.q: does not exist in index
error: test/results/clientpositive/describe_table.q.out: does not exist in index
The patch does not appear to apply with p0, p1, or p2
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-19290
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12963994 - PreCommit-HIVE-Build
> Hive desc operation takes more than 100 seconds after upgrading from Hive
> 1.2.1 to 2.3.4
>
>
> Key: HIVE-21485
> URL: https://issues.apache.org/jira/browse/HIVE-21485
> Project: Hive
> Issue Type: Bug
> Components: CLI, Hive
>Affects Versions: 2.3.4
>Reporter: Qingxin Wu
>Priority: Major
>
[jira] [Commented] (HIVE-22453) Describe table unnecessarily fetches partitions
[
https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967145#comment-16967145
]
Hive QA commented on HIVE-22453:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984829/HIVE-22453.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17566 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.exec.spark.TestSparkSessionTimeout.testMultiSessionSparkSessionTimeout
(batchId=276)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19289/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19289/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19289/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984829 - PreCommit-HIVE-Build
> Describe table unnecessarily fetches partitions
> ---
>
> Key: HIVE-22453
> URL: https://issues.apache.org/jira/browse/HIVE-22453
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Toshihiko Uchida
>Assignee: Toshihiko Uchida
>Priority: Minor
> Attachments: HIVE-22453.patch
>
>
> The simple describe table command without EXTENDED and FORMATTED (i.e.,
> DESCRIBE table_name) fetches all partitions when no partition is specified,
> although it does not display partition statistics in nature.
> The command should not fetch partitions since it can take a long time for a
> large amount of partitions.
> For instance, in our environment, the command takes around 8 seconds for a
> table with 8760 (24 * 365) partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22397) "describe table" statement for the table backed by custom storage handler fail with CNF
[
https://issues.apache.org/jira/browse/HIVE-22397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967134#comment-16967134
]
Naveen Gangam commented on HIVE-22397:
--
[~Rajkumar Singh] Are you running this reproducer from Hive CLI or from
beeline? The {{ADD JAR}} is a carry-over from the hive CLI fat client era. Its
not really ideal to be used in the beeline/hiveserver2 model.
The recommended way to configure custom UDFs and SerDes with HiveServer to is
to use {{hive.aux.jars.path}} or {{hive.reloadable.aux.jars.path}} to have a
user-configurable location for such central resources on the server side. Once
configured, the users can just drop jars into this directory/path, without
having to restart the HS2. Then there would not be a need for {{add jars}} from
any of the beeline sessions. Please refer to the documentation below.
https://docs.cloudera.com/documentation/enterprise/5-9-x/topics/cm_mc_hive_udf.html#concept_t1x_srm_2r
> "describe table" statement for the table backed by custom storage handler
> fail with CNF
>
>
> Key: HIVE-22397
> URL: https://issues.apache.org/jira/browse/HIVE-22397
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.1.2
>Reporter: Rajkumar Singh
>Assignee: Rajkumar Singh
>Priority: Major
> Attachments: HIVE-22397.patch
>
>
> Steps to Repro:
> {code:java}
> 1) describe customsdtable;
> 2) ADD JAR hdfs:///user/hive/customsdtable.jar;
> 3) describe customsdtable;
> CNF is expected for #1 but even adding the custome serde, hive fail with
> following exception for statement #3
> Error: Error while processing statement: FAILED: Execution Error, return code
> 1 from org.apache.hadoop.hive.ql.exec.DDLTask.
> java.lang.ClassNotFoundException
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22453) Describe table unnecessarily fetches partitions
[
https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967120#comment-16967120
]
Hive QA commented on HIVE-22453:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
11s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
41s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
2s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19289/dev-support/hive-personality.sh
|
| git revision | master / 164b78b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19289/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Describe table unnecessarily fetches partitions
> ---
>
> Key: HIVE-22453
> URL: https://issues.apache.org/jira/browse/HIVE-22453
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.2, 2.3.6
>Reporter: Toshihiko Uchida
>Assignee: Toshihiko Uchida
>Priority: Minor
> Attachments: HIVE-22453.patch
>
>
> The simple describe table command without EXTENDED and FORMATTED (i.e.,
> DESCRIBE table_name) fetches all partitions when no partition is specified,
> although it does not display partition statistics in nature.
> The command should not fetch partitions since it can take a long time for a
> large amount of partitions.
> For instance, in our environment, the command takes around 8 seconds for a
> table with 8760 (24 * 365) partitions.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[
https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967102#comment-16967102
]
Hive QA commented on HIVE-22327:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984824/HIVE-22327.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 13 failed/errored test(s), 17567 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesBootstrapWithJsonMessage.testRetryAcidTablesBootstrapFromDifferentDump
(batchId=252)
org.apache.hadoop.hive.ql.parse.TestReplAcidTablesWithJsonMessage.testOpenTxnEvent
(batchId=269)
org.apache.hadoop.hive.ql.parse.TestReplicationOfHiveStreaming.testHiveStreamingDynamicPartitionWithTxnBatchSizeAsOne
(batchId=249)
org.apache.hadoop.hive.ql.parse.TestReplicationOfHiveStreaming.testHiveStreamingStaticPartitionWithTxnBatchSizeAsOne
(batchId=249)
org.apache.hadoop.hive.ql.parse.TestReplicationOfHiveStreaming.testHiveStreamingUnpartitionedWithTxnBatchSizeAsOne
(batchId=249)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTables.testOpenTxnEvent
(batchId=271)
org.apache.hadoop.hive.ql.parse.TestReplicationScenariosAcidTablesBootstrap.testRetryAcidTablesBootstrapFromDifferentDump
(batchId=250)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosACID.testRetryFailure
(batchId=254)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosACIDNoAutogather.testRetryFailure
(batchId=255)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMM.testRetryFailure
(batchId=259)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMMNoAutogather.testRetryFailure
(batchId=251)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMigration.testRetryFailure
(batchId=258)
org.apache.hadoop.hive.ql.parse.TestStatsReplicationScenariosMigrationNoAutogather.testRetryFailure
(batchId=268)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19288/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19288/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19288/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 13 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984824 - PreCommit-HIVE-Build
> Repl: Ignore read-only transactions in notification log
> ---
>
> Key: HIVE-22327
> URL: https://issues.apache.org/jira/browse/HIVE-22327
> Project: Hive
> Issue Type: Improvement
> Components: repl
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22327.1.patch
>
>
> Read txns need not be replicated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[
https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967080#comment-16967080
]
Hive QA commented on HIVE-22327:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
8s{color} | {color:blue} standalone-metastore/metastore-server in master has
171 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
24s{color} | {color:red} standalone-metastore/metastore-server: The patch
generated 8 new + 551 unchanged - 1 fixed = 559 total (was 552) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
14s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 30s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19288/dev-support/hive-personality.sh
|
| git revision | master / 164b78b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19288/yetus/diff-checkstyle-standalone-metastore_metastore-server.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19288/yetus/patch-asflicense-problems.txt
|
| modules | C: standalone-metastore/metastore-server U:
standalone-metastore/metastore-server |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19288/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Repl: Ignore read-only transactions in notification log
> ---
>
> Key: HIVE-22327
> URL: https://issues.apache.org/jira/browse/HIVE-22327
> Project: Hive
> Issue Type: Improvement
> Components: repl
>Reporter: Gopal Vijayaraghavan
>Assignee: Denys Kuzmenko
>Priority: Major
> Attachments: HIVE-22327.1.patch
>
>
> Read txns need not be replicated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967069#comment-16967069
]
Hive QA commented on HIVE-22292:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984819/HIVE-22292.7.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17566 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19287/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19287/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19287/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984819 - PreCommit-HIVE-Build
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22444: -- Attachment: HIVE-22444.6.patch > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.2.patch, > HIVE-22444.3.patch, HIVE-22444.4.patch, HIVE-22444.5.patch, HIVE-22444.6.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22444: -- Status: Open (was: Patch Available) > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.2.patch, > HIVE-22444.3.patch, HIVE-22444.4.patch, HIVE-22444.5.patch, HIVE-22444.6.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22444: -- Status: Patch Available (was: Open) > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.2.patch, > HIVE-22444.3.patch, HIVE-22444.4.patch, HIVE-22444.5.patch, HIVE-22444.6.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967026#comment-16967026
]
Hive QA commented on HIVE-22292:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} ql: The patch generated 0 new + 509 unchanged - 4
fixed = 509 total (was 513) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 19s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19287/dev-support/hive-personality.sh
|
| git revision | master / 164b78b |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19287/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967012#comment-16967012 ] David Mollitor commented on HIVE-22444: --- Hmm. This is passing locally for me. Not sure why it keeps failing here. > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.2.patch, > HIVE-22444.3.patch, HIVE-22444.4.patch, HIVE-22444.5.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22444) Clean up Project POM Files
[
https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967006#comment-16967006
]
Hive QA commented on HIVE-22444:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 13m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 11m
56s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 15m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 14m
0s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
11s{color} | {color:green} storage-api generated 0 new + 0 unchanged - 2 fixed
= 0 total (was 2) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
19s{color} | {color:green} serde in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} upgrade-acid in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m
35s{color} | {color:green} root generated 0 new + 32 unchanged - 2 fixed = 32
total (was 34) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} custom-serde in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} custom-udfs in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} udf-classloader-udf1 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} udf-classloader-udf2 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
15s{color} | {color:green} udf-vectorized-badexample in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
23s{color} | {color:green} hcatalog-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} hive-blobstore in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} hive-jmh in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} hive-minikdc in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} hive-unit in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} hive-unit-hadoop2 in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | {color:green} qtest-accumulo in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
14s{color} | {color:green} qtest-druid in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} qtest-kudu in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
33s{color} | {color:green} qtest-spark in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
13s{color} | {color:green} test-serde in the patch passed. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
32s{color} | {color:green} util in the patch passed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
21s{color} |
[jira] [Commented] (HIVE-22444) Clean up Project POM Files
[
https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967005#comment-16967005
]
Hive QA commented on HIVE-22444:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984808/HIVE-22444.5.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 62 failed/errored test(s), 17519 tests
executed
*Failed tests:*
{noformat}
TestMiniDruidKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=299)
TestMiniHiveKafkaCliDriver - did not produce a TEST-*.xml file (likely timed
out) (batchId=299)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[buckets]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[create_like]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_blobstore_to_hdfs]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[ctas_hdfs_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[explain]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[having]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_local]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_blobstore_to_warehouse]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_addpartition_local_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_blobstore_nonpart]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_local]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_blobstore_to_warehouse_nonpart]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[import_local_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_blobstore_to_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_empty_into_blobstore]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_dynamic_partitions]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_into_table]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_directory]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_move]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_merge_only]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_dynamic_partitions_move_only]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[insert_overwrite_table]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join2]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[join]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[load_data]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[map_join]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[multiple_agg]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[multiple_db]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[nested_outer_join]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_buckets]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_nonpart]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_format_part]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[orc_nonstd_partitions_loc]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_buckets]
(batchId=302)
org.apache.hadoop.hive.cli.TestBlobstoreCliDriver.testCliDriver[parquet_format_nonpart]
(batchId=302)
[jira] [Updated] (HIVE-22429) Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts
[ https://issues.apache.org/jira/browse/HIVE-22429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22429: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~rameshkumar]. Please, upload a patch that applies cleanly on branch-3 and branch-3.1 if backport applies there too. > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts > -- > > Key: HIVE-22429 > URL: https://issues.apache.org/jira/browse/HIVE-22429 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22429.1.patch, HIVE-22429.2.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-22429) Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts
[ https://issues.apache.org/jira/browse/HIVE-22429?focusedWorklogId=338349=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338349 ] ASF GitHub Bot logged work on HIVE-22429: - Author: ASF GitHub Bot Created on: 04/Nov/19 20:44 Start Date: 04/Nov/19 20:44 Worklog Time Spent: 10m Work Description: asfgit commented on pull request #836: HIVE-22429 use right bucketing_version for inserts for tables with un… URL: https://github.com/apache/hive/pull/836 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 338349) Time Spent: 20m (was: 10m) > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts > -- > > Key: HIVE-22429 > URL: https://issues.apache.org/jira/browse/HIVE-22429 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22429.1.patch, HIVE-22429.2.patch > > Time Spent: 20m > Remaining Estimate: 0h > > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts
[
https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966956#comment-16966956
]
Hive QA commented on HIVE-22411:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984803/HIVE-22411.1.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 2 failed/errored test(s), 17547 tests
executed
*Failed tests:*
{noformat}
TestMiniLlapCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=157)
[unionDistinct_1.q,table_nonprintable.q,file_with_header_footer_aggregation.q,vector_offset_limit.q,orc_llap_counters1.q,mm_cttas.q,whroot_external1.q,global_limit.q,rcfile_createas1.q,dynamic_partition_pruning_2.q,intersect_merge.q,results_cache_diff_fs.q,cttl.q,parallel_colstats.q,load_hdfs_file_with_space_in_the_name.q]
TestStatsReplicationScenariosACIDNoAutogather - did not produce a TEST-*.xml
file (likely timed out) (batchId=255)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19285/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19285/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19285/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 2 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984803 - PreCommit-HIVE-Build
> Performance degradation on single row inserts
> -
>
> Key: HIVE-22411
> URL: https://issues.apache.org/jira/browse/HIVE-22411
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50
> PM.png
>
>
> Executing single insert statements on a transactional table effects write
> performance on a s3 file system. Each insert creates a new delta directory.
> After each insert hive calculates statistics like number of file in the table
> and total size of the table. In order to calculate these, it traverses the
> directory recursively. During the recursion for each path a separate
> listStatus call is executed. In the end the more delta directory you have the
> more time it takes to calculate the statistics.
> Therefore insertion time goes up linearly:
> !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436!
> The fix is to use fs.listFiles(path, /**recursive**/ true) instead the
> handcrafter recursive method/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966948#comment-16966948 ] Ramesh Kumar Thangarajan edited comment on HIVE-21304 at 11/4/19 7:32 PM: -- Hi [~kgyrtkirk], I was fixing one of the bugs in buketing_version during inserts in https://issues.apache.org/jira/browse/HIVE-22429 and I found this Jira could be useful for adding tests and verifying the issue. It looks like the patch is ready, do you think I can look into this patch and get this in upstream? I am thinking it is only test failures we need to fix. Please let me know your thoughts? was (Author: rameshkumar): Hi [~kgyrtkirk], It was fixing one of the bugs in buketing_version during inserts and I found this Jira could be useful for adding tests and verifying the issue. It looks like the patch is ready, do you think I can look into this patch and get this in upstream? I am thinking it is only test failures we need to fix. Please let me know your thoughts? > Show Bucketing version for ReduceSinkOp in explain extended plan > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch > > > Show Bucketing version for ReduceSinkOp in explain extended plan. > This helps identify what hashing algorithm is being used by by ReduceSinkOp. > > cc [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-21304) Show Bucketing version for ReduceSinkOp in explain extended plan
[ https://issues.apache.org/jira/browse/HIVE-21304?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966948#comment-16966948 ] Ramesh Kumar Thangarajan commented on HIVE-21304: - Hi [~kgyrtkirk], It was fixing one of the bugs in buketing_version during inserts and I found this Jira could be useful for adding tests and verifying the issue. It looks like the patch is ready, do you think I can look into this patch and get this in upstream? I am thinking it is only test failures we need to fix. Please let me know your thoughts? > Show Bucketing version for ReduceSinkOp in explain extended plan > > > Key: HIVE-21304 > URL: https://issues.apache.org/jira/browse/HIVE-21304 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-21304.01.patch, HIVE-21304.02.patch, > HIVE-21304.03.patch, HIVE-21304.04.patch, HIVE-21304.05.patch > > > Show Bucketing version for ReduceSinkOp in explain extended plan. > This helps identify what hashing algorithm is being used by by ReduceSinkOp. > > cc [~vgarg] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22429) Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts
[ https://issues.apache.org/jira/browse/HIVE-22429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966944#comment-16966944 ] Jesus Camacho Rodriguez commented on HIVE-22429: Sounds good. +1 > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts > -- > > Key: HIVE-22429 > URL: https://issues.apache.org/jira/browse/HIVE-22429 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22429.1.patch, HIVE-22429.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22429) Migrated clustered tables using bucketing_version 1 on hive 3 uses bucketing_version 2 for inserts
[ https://issues.apache.org/jira/browse/HIVE-22429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966937#comment-16966937 ] Ramesh Kumar Thangarajan commented on HIVE-22429: - [~jcamachorodriguez], I am not able to get a test case that fails in my local with the current code. So I have verified the fix in the cluster with help from Rohit. There is another way to verify the right bucketing_version. The patch in https://issues.apache.org/jira/browse/HIVE-21304 will help us to check if the insert query is using bucketing_version 1 or 2, using the query plan. I will speak with Zoltan and continue working on the patch. I am planning to add the test after getting the 21304 done. Do you think this makes sense? > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts > -- > > Key: HIVE-22429 > URL: https://issues.apache.org/jira/browse/HIVE-22429 > Project: Hive > Issue Type: Bug >Reporter: Ramesh Kumar Thangarajan >Assignee: Ramesh Kumar Thangarajan >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22429.1.patch, HIVE-22429.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Migrated clustered tables using bucketing_version 1 on hive 3 uses > bucketing_version 2 for inserts -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22453) Describe table unnecessarily fetches partitions
[ https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Toshihiko Uchida updated HIVE-22453: Attachment: HIVE-22453.patch Assignee: Toshihiko Uchida Status: Patch Available (was: Open) > Describe table unnecessarily fetches partitions > --- > > Key: HIVE-22453 > URL: https://issues.apache.org/jira/browse/HIVE-22453 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.6, 3.1.2 >Reporter: Toshihiko Uchida >Assignee: Toshihiko Uchida >Priority: Minor > Attachments: HIVE-22453.patch > > > The simple describe table command without EXTENDED and FORMATTED (i.e., > DESCRIBE table_name) fetches all partitions when no partition is specified, > although it does not display partition statistics in nature. > The command should not fetch partitions since it can take a long time for a > large amount of partitions. > For instance, in our environment, the command takes around 8 seconds for a > table with 8760 (24 * 365) partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22453) Describe table unnecessarily fetches partitions
[ https://issues.apache.org/jira/browse/HIVE-22453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966935#comment-16966935 ] Toshihiko Uchida commented on HIVE-22453: - HIVE-21485 also reports a performance issue on the describe table command, and tries to resolve it by introducing a runtime parameter that determines whether partition statistics are displayed or not. In the case of the describe table command without EXTENDED and FORMATTED, however, partitions should not be fetched regardless of the parameter. > Describe table unnecessarily fetches partitions > --- > > Key: HIVE-22453 > URL: https://issues.apache.org/jira/browse/HIVE-22453 > Project: Hive > Issue Type: Bug >Affects Versions: 3.1.2, 2.3.6 >Reporter: Toshihiko Uchida >Assignee: Toshihiko Uchida >Priority: Minor > Attachments: HIVE-22453.patch > > > The simple describe table command without EXTENDED and FORMATTED (i.e., > DESCRIBE table_name) fetches all partitions when no partition is specified, > although it does not display partition statistics in nature. > The command should not fetch partitions since it can take a long time for a > large amount of partitions. > For instance, in our environment, the command takes around 8 seconds for a > table with 8760 (24 * 365) partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-17350) metrics errors when retrying HS2 startup
[
https://issues.apache.org/jira/browse/HIVE-17350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966923#comment-16966923
]
Sergey Shelukhin commented on HIVE-17350:
-
I don't recall, I think it wouldn't start anyway in this case because it would
fail again for the same reason it failed for the first time.
I just noticed this was broken so even if retry succeeded, metrics might be
broken.
> metrics errors when retrying HS2 startup
>
>
> Key: HIVE-17350
> URL: https://issues.apache.org/jira/browse/HIVE-17350
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Priority: Major
>
> Looks like there are some sort of retries that happen when HS2 init fails.
> When HS2 startup fails for an unrelated reason and is retried, the metrics
> source initialization fails on subsequent attempts.
> {noformat}
> 2017-08-15T23:31:47,650 WARN [main]: impl.MetricsSystemImpl
> (MetricsSystemImpl.java:init(152)) - hiveserver2 metrics system already
> initialized!
> 2017-08-15T23:31:47,650 ERROR [main]: metastore.HiveMetaStore
> (HiveMetaStore.java:init(438)) - error in Metrics init:
> java.lang.reflect.InvocationTargetException null
> java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.common.metrics.common.MetricsFactory.init(MetricsFactory.java:42)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.init(HiveMetaStore.java:435)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:148)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:107)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.(RetryingHMSHandler.java:79)
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.getProxy(RetryingHMSHandler.java:92)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore.newRetryingHMSHandler(HiveMetaStore.java:6892)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:140)
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:74)
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1653)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3612)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3664)
> at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3644)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:582)
> at
> org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:545)
> at
> org.apache.hive.service.cli.CLIService.applyAuthorizationConfigPolicy(CLIService.java:128)
> at org.apache.hive.service.cli.CLIService.init(CLIService.java:113)
> at
> org.apache.hive.service.CompositeService.init(CompositeService.java:59)
> at org.apache.hive.service.server.HiveServer2.init(HiveServer2.java:139)
> at
> org.apache.hive.service.server.HiveServer2.startHiveServer2(HiveServer2.java:595)
> at
> org.apache.hive.service.server.HiveServer2.access$700(HiveServer2.java:97)
> at
> org.apache.hive.service.server.HiveServer2$StartOptionExecutor.execute(HiveServer2.java:843)
> at org.apache.hive.service.server.HiveServer2.main(HiveServer2.java:712)
> at
[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts
[
https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966903#comment-16966903
]
Hive QA commented on HIVE-22411:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
32s{color} | {color:blue} standalone-metastore/metastore-common in master has
37 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} standalone-metastore/metastore-common: The patch
generated 5 new + 36 unchanged - 0 fixed = 41 total (was 36) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19285/dev-support/hive-personality.sh
|
| git revision | master / 6af5e6a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19285/yetus/diff-checkstyle-standalone-metastore_metastore-common.txt
|
| modules | C: standalone-metastore/metastore-common U:
standalone-metastore/metastore-common |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19285/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Performance degradation on single row inserts
> -
>
> Key: HIVE-22411
> URL: https://issues.apache.org/jira/browse/HIVE-22411
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50
> PM.png
>
>
> Executing single insert statements on a transactional table effects write
> performance on a s3 file system. Each insert creates a new delta directory.
> After each insert hive calculates statistics like number of file in the table
> and total size of the table. In order to calculate these, it traverses the
> directory recursively. During the recursion for each path a separate
> listStatus call is executed. In the end the more delta directory you have the
> more time it takes to calculate the statistics.
> Therefore insertion time goes up linearly:
> !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436!
> The fix is to use fs.listFiles(path, /**recursive**/ true) instead the
> handcrafter recursive method/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22414) Make LLAP CacheTags more memory efficient
[
https://issues.apache.org/jira/browse/HIVE-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966885#comment-16966885
]
Hive QA commented on HIVE-22414:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984799/HIVE-22414.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17555 tests
executed
*Failed tests:*
{noformat}
TestJdbcWithMiniLlapArrow - did not produce a TEST-*.xml file (likely timed
out) (batchId=284)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19284/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19284/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19284/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984799 - PreCommit-HIVE-Build
> Make LLAP CacheTags more memory efficient
> -
>
> Key: HIVE-22414
> URL: https://issues.apache.org/jira/browse/HIVE-22414
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22414.0.patch, HIVE-22414.1.patch,
> HIVE-22414.2.patch, HIVE-22414.3.patch, HIVE-22414.4.patch
>
>
> MultiPartitionCacheTag relies on LinkedLists.
> A LinkedList object that holds e.g. 2 nodes consumes 112 bytes roughly in
> this composition:
> * 16 bytes for LinkedList object header
> * 8 bytes for referring head
> * 8 bytes for referring tail
> * 80 = 2 x (16 bytes for LinkedList$Node header, 3 x 8 bytes for referring
> prev, next, item)
> This is a lot, so I propose to replace LinkedList in MultiPartitionCacheTag
> with a simple String array. (For a similar scenario an array would take 16 +
> 8 + 2 x 8 = 40 bytes, as per header, count, and 2 references for our actual
> objects).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-22311) Propagate min/max column values from statistics to the optimizer for timestamp type
[ https://issues.apache.org/jira/browse/HIVE-22311?focusedWorklogId=338247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338247 ] ASF GitHub Bot logged work on HIVE-22311: - Author: ASF GitHub Bot Created on: 04/Nov/19 17:57 Start Date: 04/Nov/19 17:57 Worklog Time Spent: 10m Work Description: jcamachor commented on pull request #821: HIVE-22311 URL: https://github.com/apache/hive/pull/821#discussion_r342187541 ## File path: ql/src/java/org/apache/hadoop/hive/ql/metadata/formatting/JsonMetaDataFormatter.java ## @@ -338,6 +338,19 @@ private ColumnStatisticsData getStatistics(FieldSchema column, List Propagate min/max column values from statistics to the optimizer for > timestamp type > --- > > Key: HIVE-22311 > URL: https://issues.apache.org/jira/browse/HIVE-22311 > Project: Hive > Issue Type: Bug > Components: Statistics >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22311.01.patch, HIVE-22311.01.patch, > HIVE-22311.02.patch, HIVE-22311.03.patch, HIVE-22311.patch > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently stats annotation does not consider timestamp type e.g. for > estimates with range predicates. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[ https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22327: -- Attachment: (was: HIVE-22327.patch.1) > Repl: Ignore read-only transactions in notification log > --- > > Key: HIVE-22327 > URL: https://issues.apache.org/jira/browse/HIVE-22327 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22327.1.patch > > > Read txns need not be replicated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[ https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22327: -- Status: Patch Available (was: Open) > Repl: Ignore read-only transactions in notification log > --- > > Key: HIVE-22327 > URL: https://issues.apache.org/jira/browse/HIVE-22327 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22327.1.patch > > > Read txns need not be replicated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[ https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22327: -- Attachment: HIVE-22327.1.patch > Repl: Ignore read-only transactions in notification log > --- > > Key: HIVE-22327 > URL: https://issues.apache.org/jira/browse/HIVE-22327 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22327.1.patch > > > Read txns need not be replicated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[ https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22327: -- Attachment: (was: HIVE-22327.patch) > Repl: Ignore read-only transactions in notification log > --- > > Key: HIVE-22327 > URL: https://issues.apache.org/jira/browse/HIVE-22327 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > Attachments: HIVE-22327.patch.1 > > > Read txns need not be replicated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22327) Repl: Ignore read-only transactions in notification log
[ https://issues.apache.org/jira/browse/HIVE-22327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Denys Kuzmenko updated HIVE-22327: -- Attachment: HIVE-22327.patch.1 > Repl: Ignore read-only transactions in notification log > --- > > Key: HIVE-22327 > URL: https://issues.apache.org/jira/browse/HIVE-22327 > Project: Hive > Issue Type: Improvement > Components: repl >Reporter: Gopal Vijayaraghavan >Assignee: Denys Kuzmenko >Priority: Major > > Read txns need not be replicated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22292:
--
Status: Patch Available (was: Open)
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22292:
--
Attachment: HIVE-22292.7.patch
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22292:
--
Status: Open (was: Patch Available)
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22414) Make LLAP CacheTags more memory efficient
[
https://issues.apache.org/jira/browse/HIVE-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966834#comment-16966834
]
Hive QA commented on HIVE-22414:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
55s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
24s{color} | {color:blue} storage-api in master has 48 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
44s{color} | {color:blue} llap-server in master has 90 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
25s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
35s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19284/dev-support/hive-personality.sh
|
| git revision | master / 6af5e6a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: storage-api llap-server U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19284/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Make LLAP CacheTags more memory efficient
> -
>
> Key: HIVE-22414
> URL: https://issues.apache.org/jira/browse/HIVE-22414
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22414.0.patch, HIVE-22414.1.patch,
> HIVE-22414.2.patch, HIVE-22414.3.patch, HIVE-22414.4.patch
>
>
> MultiPartitionCacheTag relies on LinkedLists.
> A LinkedList object that holds e.g. 2 nodes consumes 112 bytes roughly in
> this composition:
> * 16 bytes for LinkedList object header
> * 8 bytes for referring head
> * 8 bytes for referring tail
> * 80 = 2 x (16 bytes for LinkedList$Node header, 3 x 8 bytes for referring
> prev, next, item)
> This is a lot, so I propose to replace LinkedList in MultiPartitionCacheTag
> with a simple String array. (For a similar scenario an array would take 16 +
> 8 + 2 x 8 = 40 bytes, as per header, count, and 2 references for our actual
> objects).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966812#comment-16966812
]
Hive QA commented on HIVE-22292:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984788/HIVE-22292.7.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17566 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestPartitionManagement.testPartitionDiscoveryTransactionalTable
(batchId=223)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19283/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19283/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19283/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984788 - PreCommit-HIVE-Build
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (HIVE-22392) Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source
[ https://issues.apache.org/jira/browse/HIVE-22392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez reopened HIVE-22392: > Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source > --- > > Key: HIVE-22392 > URL: https://issues.apache.org/jira/browse/HIVE-22392 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22392.01.patch, HIVE-22392.02.patch > > > JDBC Storage Handler supports reading from JDBC data source in Hive. > Currently writing to a JDBC data source is not supported. Hence adding > support for simple insert query so that the data can be written back to JDBC > data source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22433) Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and TIMESTAMP DataType From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jesus Camacho Rodriguez updated HIVE-22433:
---
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master, thanks [~srahman]!
> Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and
> TIMESTAMP DataType From JDBC Data Source
> --
>
> Key: HIVE-22433
> URL: https://issues.apache.org/jira/browse/HIVE-22433
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Fix For: 4.0.0
>
> Attachments: HIVE-22433.01.patch, HIVE-22433.02.patch,
> HIVE-22433.03.patch, HIVE-22433.04.patch, HIVE-22433.05.patch
>
>
> Steps to Reproduce:
> {code:java}
> //Derby table:
> create table testtbl(a BOOLEAN, b TIMESTAMP);
> // Insert to table via mysql connector
> // data in db
> true 2019-11-11 12:00:00
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a BOOLEAN, b TIMESTAMP
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='DERBY',
> 'hive.sql.dbcp.password'='',
> 'hive.sql.dbcp.username'='',
> 'hive.sql.jdbc.driver'='',
> 'hive.sql.jdbc.url'='',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> // result from select query
> false 2019-11-11 20:00:00
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (HIVE-22392) Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source
[ https://issues.apache.org/jira/browse/HIVE-22392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22392: --- Comment: was deleted (was: Pushed to master, thanks [~srahman]!) > Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source > --- > > Key: HIVE-22392 > URL: https://issues.apache.org/jira/browse/HIVE-22392 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22392.01.patch, HIVE-22392.02.patch > > > JDBC Storage Handler supports reading from JDBC data source in Hive. > Currently writing to a JDBC data source is not supported. Hence adding > support for simple insert query so that the data can be written back to JDBC > data source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22392) Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source
[ https://issues.apache.org/jira/browse/HIVE-22392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesus Camacho Rodriguez updated HIVE-22392: --- Fix Version/s: 4.0.0 Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master, thanks [~srahman]! > Hive JDBC Storage Handler: Support For Writing Data to JDBC Data Source > --- > > Key: HIVE-22392 > URL: https://issues.apache.org/jira/browse/HIVE-22392 > Project: Hive > Issue Type: New Feature > Components: StorageHandler >Reporter: Syed Shameerur Rahman >Assignee: Syed Shameerur Rahman >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: HIVE-22392.01.patch, HIVE-22392.02.patch > > > JDBC Storage Handler supports reading from JDBC data source in Hive. > Currently writing to a JDBC data source is not supported. Hence adding > support for simple insert query so that the data can be written back to JDBC > data source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-21894) Hadoop credential password storage for the Kafka Storage handler when security is SSL
[ https://issues.apache.org/jira/browse/HIVE-21894?focusedWorklogId=338174=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-338174 ] ASF GitHub Bot logged work on HIVE-21894: - Author: ASF GitHub Bot Created on: 04/Nov/19 16:02 Start Date: 04/Nov/19 16:02 Worklog Time Spent: 10m Work Description: justinleet commented on pull request #839: HIVE-21894: Hadoop credential password storage for the Kafka Storage handler when security is SSL URL: https://github.com/apache/hive/pull/839 [HIVE-21894](https://issues.apache.org/jira/browse/HIVE-21894) Allows for the KafkaStorageHandler to be configured with SSL properties, where the passwords aren't in plaintext in the table configs. This has been tested on an actual Hadoop cluster against an actual Kafka cluster, but in a pretty limited manner and primarily for the consumer side of things (full disclosure, my use case is pretty exclusively read from). I've done some basic testing to make sure both queries that aren't spinning up jobs (e.g. simple `SELECT *` type queries) and queries that do spin up jobs (e.g. some basic `GROUP BY`) all runs to success. There's a couple things that probably need some feedback and possibly iteration. - Distribution of the key/trust stores. Kafka can only work with these stores locally, but they need to be distributed for jobs, so HDFS seems like the right place to keep them. Right now, it's an HDFS file that is being pulled via the standard HDFS APIs into `DOWNLOADED_RESOURCES_DIR`. There are other StorageHandlers (see: [HIVE-21894](https://issues.apache.org/jira/browse/HIVE-21894?focusedCommentId=16869476=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16869476)) do some dealing with files, but they seem to do with jars and go through the `tmpjars` config (which I believe is just essentially `-libjars`). - Is this the right place to put the files? - Is this a more reasonable way to get them? - Right now, producer / consumer SSL configs are assumed to be the same (i.e. `hive.kafka.ssl.keystore.password` instead of `hive.kafka.consumer.ssl ...` and `hive.kafka.producer.ssl ...` - This could fairly easily be split out if there's a need. I'm not honestly sure how much configuring a producer and consumer separately would be used in practice. - Naming of the configs. If there are any particular conventions I should follow, let me know and I'll test and update. - Automated testing. Given the need for HDFS and Kafka, I've just added some tests that the configs end up reasonable, but we may want more and I'm not familiar enough with Hive's testing utilities to know if there are better options. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 338174) Remaining Estimate: 0h Time Spent: 10m > Hadoop credential password storage for the Kafka Storage handler when > security is SSL > - > > Key: HIVE-21894 > URL: https://issues.apache.org/jira/browse/HIVE-21894 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Affects Versions: 4.0.0 >Reporter: Kristopher Kane >Assignee: Kristopher Kane >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > Time Spent: 10m > Remaining Estimate: 0h > > The Kafka storage handler assumes that if the Hive service is configured with > Kerberos then the destination Kafka cluster is also secured with the same > Kerberos realm or trust of realms. The security configuration of the Kafka > client can be overwritten due to the additive operations of the Kafka client > configs, but, the only way to specify SSL and the keystore/truststore > user/pass is via plain text table properties. > This ticket proposes adding Hadoop credential security to the Kafka storage > handler in support of SSL secured Kafka clusters. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-21894) Hadoop credential password storage for the Kafka Storage handler when security is SSL
[ https://issues.apache.org/jira/browse/HIVE-21894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-21894: -- Labels: pull-request-available (was: ) > Hadoop credential password storage for the Kafka Storage handler when > security is SSL > - > > Key: HIVE-21894 > URL: https://issues.apache.org/jira/browse/HIVE-21894 > Project: Hive > Issue Type: Improvement > Components: kafka integration >Affects Versions: 4.0.0 >Reporter: Kristopher Kane >Assignee: Kristopher Kane >Priority: Minor > Labels: pull-request-available > Fix For: 4.0.0 > > > The Kafka storage handler assumes that if the Hive service is configured with > Kerberos then the destination Kafka cluster is also secured with the same > Kerberos realm or trust of realms. The security configuration of the Kafka > client can be overwritten due to the additive operations of the Kafka client > configs, but, the only way to specify SSL and the keystore/truststore > user/pass is via plain text table properties. > This ticket proposes adding Hadoop credential security to the Kafka storage > handler in support of SSL secured Kafka clusters. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966769#comment-16966769
]
Hive QA commented on HIVE-22292:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
15s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
15s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} ql: The patch generated 0 new + 509 unchanged - 4
fixed = 509 total (was 513) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 26m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19283/dev-support/hive-personality.sh
|
| git revision | master / 136e0eb |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19283/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22417) Remove stringifyException from MetaStore
[ https://issues.apache.org/jira/browse/HIVE-22417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966766#comment-16966766 ] David Mollitor commented on HIVE-22417: --- [~abstractdog] Are you able to take a look at this one? Thanks! > Remove stringifyException from MetaStore > > > Key: HIVE-22417 > URL: https://issues.apache.org/jira/browse/HIVE-22417 > Project: Hive > Issue Type: Sub-task > Components: Metastore, Standalone Metastore >Affects Versions: 3.2.0 >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22417.1.patch, HIVE-22417.2.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22424) Use PerfLogger in MetastoreDirectSqlUtils.java
[ https://issues.apache.org/jira/browse/HIVE-22424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22424: -- Summary: Use PerfLogger in MetastoreDirectSqlUtils.java (was: User PerfLogger in MetastoreDirectSqlUtils.java) > Use PerfLogger in MetastoreDirectSqlUtils.java > -- > > Key: HIVE-22424 > URL: https://issues.apache.org/jira/browse/HIVE-22424 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore >Affects Versions: 3.2.0 >Reporter: David Mollitor >Priority: Minor > Fix For: 4.0.0 > > > Currently has its own way of doing it: > https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/MetastoreDirectSqlUtils.java#L100 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22451) Secure LLAP configurations are still deemed unsecure in Tez AM processes
[
https://issues.apache.org/jira/browse/HIVE-22451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966739#comment-16966739
]
Hive QA commented on HIVE-22451:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984789/HIVE-22451.0.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17567 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19282/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19282/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19282/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984789 - PreCommit-HIVE-Build
> Secure LLAP configurations are still deemed unsecure in Tez AM processes
>
>
> Key: HIVE-22451
> URL: https://issues.apache.org/jira/browse/HIVE-22451
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22451.0.patch
>
>
> Due to the change in HIVE-22354 and HIVE-22195 Zookeeper discovery of LLAP
> workers is not working when invoked from within a Tez AM process: a Tez AM
> process does not log on using Kerberos even in secure environments, hence
> {code:java}
> UserGroupInformation.getLoginUser().hasKerberosCredentials() {code}
> will return false for security-enabled clusters too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22427) PersistenceManagerProvider Logs a Warning About datanucleus.autoStartMechanismMode
[
https://issues.apache.org/jira/browse/HIVE-22427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
David Mollitor updated HIVE-22427:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks [~abstractdog] for the review and [~mgergely] for getting this Jira into
the right hands. Committed to master.
> PersistenceManagerProvider Logs a Warning About
> datanucleus.autoStartMechanismMode
> --
>
> Key: HIVE-22427
> URL: https://issues.apache.org/jira/browse/HIVE-22427
> Project: Hive
> Issue Type: Improvement
> Components: Standalone Metastore
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: David Mollitor
>Priority: Minor
> Fix For: 4.0.0
>
> Attachments: HIVE-22427.1.patch
>
>
> {code:none}
> WARN [pool-6-thread-2] metastore.PersistenceManagerProvider:
> datanucleus.autoStartMechanismMode is set to unsupported value null . Setting
> it to value: ignored
> {code}
> This does not need to be a WARN level logging for this scenario. Perhaps if
> user configures the value to some non-null value, then emit a warning,
> otherwise, simply emit an INFO level stating that the configuration is not
> set and that a reasonable default value will be used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22451) Secure LLAP configurations are still deemed unsecure in Tez AM processes
[
https://issues.apache.org/jira/browse/HIVE-22451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966695#comment-16966695
]
Hive QA commented on HIVE-22451:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
57s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
31s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} llap-client in master has 27 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 1m
20s{color} | {color:blue} standalone-metastore/metastore-server in master has
171 extant Findbugs warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
13s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
50s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
11s{color} | {color:red} llap-client: The patch generated 1 new + 6 unchanged -
0 fixed = 7 total (was 6) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19282/dev-support/hive-personality.sh
|
| git revision | master / cdad75a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19282/yetus/diff-checkstyle-llap-client.txt
|
| modules | C: llap-client standalone-metastore/metastore-server ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19282/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Secure LLAP configurations are still deemed unsecure in Tez AM processes
>
>
> Key: HIVE-22451
> URL: https://issues.apache.org/jira/browse/HIVE-22451
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22451.0.patch
>
>
> Due to the change in HIVE-22354 and HIVE-22195 Zookeeper discovery of LLAP
> workers is not working when invoked from within a Tez AM process: a Tez AM
> process does not log on using Kerberos even in secure
[jira] [Updated] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22444: -- Status: Open (was: Patch Available) > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.2.patch, > HIVE-22444.3.patch, HIVE-22444.4.patch, HIVE-22444.5.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22444: -- Attachment: HIVE-22444.5.patch > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.2.patch, > HIVE-22444.3.patch, HIVE-22444.4.patch, HIVE-22444.5.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22444) Clean up Project POM Files
[ https://issues.apache.org/jira/browse/HIVE-22444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Mollitor updated HIVE-22444: -- Status: Patch Available (was: Open) > Clean up Project POM Files > -- > > Key: HIVE-22444 > URL: https://issues.apache.org/jira/browse/HIVE-22444 > Project: Hive > Issue Type: Improvement >Reporter: David Mollitor >Assignee: David Mollitor >Priority: Major > Attachments: HIVE-22444.1.patch, HIVE-22444.2.patch, > HIVE-22444.3.patch, HIVE-22444.4.patch, HIVE-22444.5.patch > > > # Address warnings in the build process > # Use DependencyManagement in Root POM for ITest (see HIVE-22426) > # General POM cleanup -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (HIVE-22452) CTAS query failure at DDL task stage doesn't clean out the target directory
[ https://issues.apache.org/jira/browse/HIVE-22452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-22452: --- Assignee: Marta Kuczora (was: Riju Trivedi) > CTAS query failure at DDL task stage doesn't clean out the target directory > --- > > Key: HIVE-22452 > URL: https://issues.apache.org/jira/browse/HIVE-22452 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.0, 3.1.2 >Reporter: Riju Trivedi >Assignee: Marta Kuczora >Priority: Major > > CTAS query failure at DDL task stage due to HMS connection issue leaves the > output file in target directory. Since DDL task stage happens after Tez DAG > completion and MOVE Task , output file gets already moved to target > directory and does not get cleaned up after the query failure. > Re-executing the same query causes a duplicate file under table location > hence duplicate data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22420) DbTxnManager.stopHeartbeat() should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966675#comment-16966675
]
Peter Vary commented on HIVE-22420:
---
[~hamvas.aron]: Let's wait for the 1 day grace period, and tomorrow I can
commit.
Thanks,
Peter
> DbTxnManager.stopHeartbeat() should be thread-safe
> --
>
> Key: HIVE-22420
> URL: https://issues.apache.org/jira/browse/HIVE-22420
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
> Attachments: HIVE-22420.1.patch, HIVE-22420.2.patch
>
>
> When a transactional query is being executed and interrupted via HS2 close
> operation request, both the background pool thread executing the query and
> the HttpHandler thread running the close operation logic will eventually call
> the below method:
> {noformat}
> Driver.releaseLocksAndCommitOrRollback(commit boolean)
> {noformat}
> Since this method is invoked several times in both threads, it can happen
> that the two threads invoke it at the same time, and due to a race condition,
> the txnId field of the DbTxnManager used by both threads could be set to 0
> without actually successfully aborting the transaction.
> The root cause is stopHeartbeat() method in DbTxnManager not being thread
> safe:
> When Thread-1 and Thread-2 enter stopHeartbeat() with very little time
> difference, Thread-1 might successfully cancel the heartbeat task and set the
> heartbeatTask field to null, while Thread-2 is trying to observe its state.
> Thread-1 will return to the calling rollbackTxn() method and continue
> execution there, while Thread-2 wis thrown back to the same method with a
> NullPointerException. Thread-2 will then set txnId to 0, and Thread-1 is
> sending this 0 value to HMS. So, the txn will not be aborted, and the locks
> cannot be released later on either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22420) DbTxnManager.stopHeartbeat() should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966673#comment-16966673
]
Aron Hamvas commented on HIVE-22420:
Thanks for the review, [~pvary]! The requested change is in patch 2. If ok, can
you commit this change for me?
> DbTxnManager.stopHeartbeat() should be thread-safe
> --
>
> Key: HIVE-22420
> URL: https://issues.apache.org/jira/browse/HIVE-22420
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
> Attachments: HIVE-22420.1.patch, HIVE-22420.2.patch
>
>
> When a transactional query is being executed and interrupted via HS2 close
> operation request, both the background pool thread executing the query and
> the HttpHandler thread running the close operation logic will eventually call
> the below method:
> {noformat}
> Driver.releaseLocksAndCommitOrRollback(commit boolean)
> {noformat}
> Since this method is invoked several times in both threads, it can happen
> that the two threads invoke it at the same time, and due to a race condition,
> the txnId field of the DbTxnManager used by both threads could be set to 0
> without actually successfully aborting the transaction.
> The root cause is stopHeartbeat() method in DbTxnManager not being thread
> safe:
> When Thread-1 and Thread-2 enter stopHeartbeat() with very little time
> difference, Thread-1 might successfully cancel the heartbeat task and set the
> heartbeatTask field to null, while Thread-2 is trying to observe its state.
> Thread-1 will return to the calling rollbackTxn() method and continue
> execution there, while Thread-2 wis thrown back to the same method with a
> NullPointerException. Thread-2 will then set txnId to 0, and Thread-1 is
> sending this 0 value to HMS. So, the txn will not be aborted, and the locks
> cannot be released later on either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22420) DbTxnManager.stopHeartbeat() should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aron Hamvas updated HIVE-22420:
---
Target Version/s: 4.0.0 (was: 4.0.0, 3.1.3)
> DbTxnManager.stopHeartbeat() should be thread-safe
> --
>
> Key: HIVE-22420
> URL: https://issues.apache.org/jira/browse/HIVE-22420
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
> Attachments: HIVE-22420.1.patch, HIVE-22420.2.patch
>
>
> When a transactional query is being executed and interrupted via HS2 close
> operation request, both the background pool thread executing the query and
> the HttpHandler thread running the close operation logic will eventually call
> the below method:
> {noformat}
> Driver.releaseLocksAndCommitOrRollback(commit boolean)
> {noformat}
> Since this method is invoked several times in both threads, it can happen
> that the two threads invoke it at the same time, and due to a race condition,
> the txnId field of the DbTxnManager used by both threads could be set to 0
> without actually successfully aborting the transaction.
> The root cause is stopHeartbeat() method in DbTxnManager not being thread
> safe:
> When Thread-1 and Thread-2 enter stopHeartbeat() with very little time
> difference, Thread-1 might successfully cancel the heartbeat task and set the
> heartbeatTask field to null, while Thread-2 is trying to observe its state.
> Thread-1 will return to the calling rollbackTxn() method and continue
> execution there, while Thread-2 wis thrown back to the same method with a
> NullPointerException. Thread-2 will then set txnId to 0, and Thread-1 is
> sending this 0 value to HMS. So, the txn will not be aborted, and the locks
> cannot be released later on either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (HIVE-22452) CTAS query failure at DDL task stage doesn't clean out the target directory
[ https://issues.apache.org/jira/browse/HIVE-22452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Riju Trivedi reassigned HIVE-22452: --- > CTAS query failure at DDL task stage doesn't clean out the target directory > --- > > Key: HIVE-22452 > URL: https://issues.apache.org/jira/browse/HIVE-22452 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.1.2, 3.1.0 >Reporter: Riju Trivedi >Assignee: Riju Trivedi >Priority: Major > > CTAS query failure at DDL task stage due to HMS connection issue leaves the > output file in target directory. Since DDL task stage happens after Tez DAG > completion and MOVE Task , output file gets already moved to target > directory and does not get cleaned up after the query failure. > Re-executing the same query causes a duplicate file under table location > hence duplicate data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22420) DbTxnManager.stopHeartbeat() should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1694#comment-1694
]
Hive QA commented on HIVE-22420:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984784/HIVE-22420.2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17566 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19281/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19281/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19281/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984784 - PreCommit-HIVE-Build
> DbTxnManager.stopHeartbeat() should be thread-safe
> --
>
> Key: HIVE-22420
> URL: https://issues.apache.org/jira/browse/HIVE-22420
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
> Attachments: HIVE-22420.1.patch, HIVE-22420.2.patch
>
>
> When a transactional query is being executed and interrupted via HS2 close
> operation request, both the background pool thread executing the query and
> the HttpHandler thread running the close operation logic will eventually call
> the below method:
> {noformat}
> Driver.releaseLocksAndCommitOrRollback(commit boolean)
> {noformat}
> Since this method is invoked several times in both threads, it can happen
> that the two threads invoke it at the same time, and due to a race condition,
> the txnId field of the DbTxnManager used by both threads could be set to 0
> without actually successfully aborting the transaction.
> The root cause is stopHeartbeat() method in DbTxnManager not being thread
> safe:
> When Thread-1 and Thread-2 enter stopHeartbeat() with very little time
> difference, Thread-1 might successfully cancel the heartbeat task and set the
> heartbeatTask field to null, while Thread-2 is trying to observe its state.
> Thread-1 will return to the calling rollbackTxn() method and continue
> execution there, while Thread-2 wis thrown back to the same method with a
> NullPointerException. Thread-2 will then set txnId to 0, and Thread-1 is
> sending this 0 value to HMS. So, the txn will not be aborted, and the locks
> cannot be released later on either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Status: Patch Available (was: Open) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Attachment: (was: HIVE-22411.1.patch) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Attachment: HIVE-22411.1.patch > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Status: Open (was: Patch Available) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Status: Patch Available (was: Open) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Attachment: HIVE-22411.1.patch > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Attachment: (was: HIVE-22411.1.patch) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Status: Open (was: Patch Available) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22420) DbTxnManager.stopHeartbeat() should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966639#comment-16966639
]
Hive QA commented on HIVE-22420:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
7s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19281/dev-support/hive-personality.sh
|
| git revision | master / cdad75a |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19281/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> DbTxnManager.stopHeartbeat() should be thread-safe
> --
>
> Key: HIVE-22420
> URL: https://issues.apache.org/jira/browse/HIVE-22420
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
> Attachments: HIVE-22420.1.patch, HIVE-22420.2.patch
>
>
> When a transactional query is being executed and interrupted via HS2 close
> operation request, both the background pool thread executing the query and
> the HttpHandler thread running the close operation logic will eventually call
> the below method:
> {noformat}
> Driver.releaseLocksAndCommitOrRollback(commit boolean)
> {noformat}
> Since this method is invoked several times in both threads, it can happen
> that the two threads invoke it at the same time, and due to a race condition,
> the txnId field of the DbTxnManager used by both threads could be set to 0
> without actually successfully aborting the transaction.
> The root cause is stopHeartbeat() method in DbTxnManager not being thread
> safe:
> When Thread-1 and Thread-2 enter stopHeartbeat() with very little time
> difference, Thread-1 might successfully cancel the heartbeat task and set the
> heartbeatTask field to null, while Thread-2 is trying to observe its state.
> Thread-1 will return to the calling rollbackTxn() method and continue
> execution there, while Thread-2 wis thrown back to the same method with a
> NullPointerException. Thread-2 will then set txnId to 0, and Thread-1 is
> sending this 0 value to HMS. So, the txn will not be aborted, and the locks
>
[jira] [Commented] (HIVE-21114) Create read-only transactions
[
https://issues.apache.org/jira/browse/HIVE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966633#comment-16966633
]
Denys Kuzmenko commented on HIVE-21114:
---
Thank you, [~pvary] and [~ashutosh.bapat] !
> Create read-only transactions
> -
>
> Key: HIVE-21114
> URL: https://issues.apache.org/jira/browse/HIVE-21114
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21114.1.patch, HIVE-21114.10.patch,
> HIVE-21114.11.patch, HIVE-21114.2.patch, HIVE-21114.3.patch,
> HIVE-21114.4.patch, HIVE-21114.5.patch, HIVE-21114.6.patch,
> HIVE-21114.7.patch, HIVE-21114.8.patch, HIVE-21114.9.patch
>
>
> With HIVE-21036 we have a way to indicate that a txn is read only.
> We should (at least in auto-commit mode) determine if the single stmt is a
> read and mark the txn accordingly.
> Then we can optimize {{TxnHandler.commitTxn()}} so that it doesn't do any
> checks in write_set etc.
> {{TxnHandler.commitTxn()}} already starts with {{lockTransactionRecord(stmt,
> txnid, TXN_OPEN)}} so it can read the txn type in the same SQL stmt.
> HiveOperation only has QUERY, which includes Insert and Select, so this
> requires figuring out how to determine if a query is a SELECT. By the time
> {{Driver.openTransaction();}} is called, we have already parsed the query so
> there should be a way to know if the statement only reads.
> For multi-stmt txns (once these are supported) we should allow user to
> indicate that a txn is read-only and then not allow any statements that can
> make modifications in this txn. This should be a different jira.
> cc [~ikryvenko]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22414) Make LLAP CacheTags more memory efficient
[ https://issues.apache.org/jira/browse/HIVE-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22414: -- Attachment: HIVE-22414.4.patch > Make LLAP CacheTags more memory efficient > - > > Key: HIVE-22414 > URL: https://issues.apache.org/jira/browse/HIVE-22414 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22414.0.patch, HIVE-22414.1.patch, > HIVE-22414.2.patch, HIVE-22414.3.patch, HIVE-22414.4.patch > > > MultiPartitionCacheTag relies on LinkedLists. > A LinkedList object that holds e.g. 2 nodes consumes 112 bytes roughly in > this composition: > * 16 bytes for LinkedList object header > * 8 bytes for referring head > * 8 bytes for referring tail > * 80 = 2 x (16 bytes for LinkedList$Node header, 3 x 8 bytes for referring > prev, next, item) > This is a lot, so I propose to replace LinkedList in MultiPartitionCacheTag > with a simple String array. (For a similar scenario an array would take 16 + > 8 + 2 x 8 = 40 bytes, as per header, count, and 2 references for our actual > objects). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22411) Performance degradation on single row inserts
[
https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966615#comment-16966615
]
Hive QA commented on HIVE-22411:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984775/HIVE-22411.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19280/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19280/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19280/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2019-11-04 12:19:04.940
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-19280/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2019-11-04 12:19:04.943
+ cd apache-github-source-source
+ git fetch origin
>From https://github.com/apache/hive
2a7bbba..cdad75a master -> origin/master
+ git reset --hard HEAD
HEAD is now at 2a7bbba HIVE-22274: Upgrade Calcite version to 1.21.0 (Steve
Carlin, reviewed by Jesus Camacho Rodriguez)
+ git clean -f -d
Removing standalone-metastore/metastore-server/src/gen/
+ git checkout master
Already on 'master'
Your branch is behind 'origin/master' by 1 commit, and can be fast-forwarded.
(use "git pull" to update your local branch)
+ git reset --hard origin/master
HEAD is now at cdad75a HIVE-21114: Create read-only transactions (Denys
Kuzmenko, reviewed by Ashutosh Bapat and Peter Vary)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2019-11-04 12:19:06.492
+ rm -rf ../yetus_PreCommit-HIVE-Build-19280
+ mkdir ../yetus_PreCommit-HIVE-Build-19280
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-19280
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-19280/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error:
a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/Warehouse.java:
does not exist in index
error:
a/standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java:
does not exist in index
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java:17
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java:17
Falling back to three-way merge...
Applied patch to
'standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java'
with conflicts.
U
standalone-metastore/metastore-common/src/main/java/org/apache/hadoop/hive/metastore/utils/FileUtils.java
+ result=1
+ '[' 1 -ne 0 ']'
+ rm -rf yetus_PreCommit-HIVE-Build-19280
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984775 - PreCommit-HIVE-Build
> Performance degradation on single row inserts
> -
>
> Key: HIVE-22411
> URL: https://issues.apache.org/jira/browse/HIVE-22411
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: Attila Magyar
>Assignee: Attila Magyar
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50
> PM.png
>
>
> Executing single insert statements on a
[jira] [Commented] (HIVE-22414) Make LLAP CacheTags more memory efficient
[
https://issues.apache.org/jira/browse/HIVE-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966612#comment-16966612
]
Hive QA commented on HIVE-22414:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984772/HIVE-22414.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_overwrite]
(batchId=166)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19279/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19279/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19279/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984772 - PreCommit-HIVE-Build
> Make LLAP CacheTags more memory efficient
> -
>
> Key: HIVE-22414
> URL: https://issues.apache.org/jira/browse/HIVE-22414
> Project: Hive
> Issue Type: Improvement
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22414.0.patch, HIVE-22414.1.patch,
> HIVE-22414.2.patch, HIVE-22414.3.patch
>
>
> MultiPartitionCacheTag relies on LinkedLists.
> A LinkedList object that holds e.g. 2 nodes consumes 112 bytes roughly in
> this composition:
> * 16 bytes for LinkedList object header
> * 8 bytes for referring head
> * 8 bytes for referring tail
> * 80 = 2 x (16 bytes for LinkedList$Node header, 3 x 8 bytes for referring
> prev, next, item)
> This is a lot, so I propose to replace LinkedList in MultiPartitionCacheTag
> with a simple String array. (For a similar scenario an array would take 16 +
> 8 + 2 x 8 = 40 bytes, as per header, count, and 2 references for our actual
> objects).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-21114) Create read-only transactions
[
https://issues.apache.org/jira/browse/HIVE-21114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peter Vary updated HIVE-21114:
--
Fix Version/s: 4.0.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Pushed to master.
Thanks for the patch [~dkuzmenko] and [~ashutosh.bapat] for the review.
[~dkuzmenko]: Please file a follow-up jira for adding a configuration to turn
on/off this feature by defaulting to off.
> Create read-only transactions
> -
>
> Key: HIVE-21114
> URL: https://issues.apache.org/jira/browse/HIVE-21114
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 4.0.0
>Reporter: Eugene Koifman
>Assignee: Denys Kuzmenko
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-21114.1.patch, HIVE-21114.10.patch,
> HIVE-21114.11.patch, HIVE-21114.2.patch, HIVE-21114.3.patch,
> HIVE-21114.4.patch, HIVE-21114.5.patch, HIVE-21114.6.patch,
> HIVE-21114.7.patch, HIVE-21114.8.patch, HIVE-21114.9.patch
>
>
> With HIVE-21036 we have a way to indicate that a txn is read only.
> We should (at least in auto-commit mode) determine if the single stmt is a
> read and mark the txn accordingly.
> Then we can optimize {{TxnHandler.commitTxn()}} so that it doesn't do any
> checks in write_set etc.
> {{TxnHandler.commitTxn()}} already starts with {{lockTransactionRecord(stmt,
> txnid, TXN_OPEN)}} so it can read the txn type in the same SQL stmt.
> HiveOperation only has QUERY, which includes Insert and Select, so this
> requires figuring out how to determine if a query is a SELECT. By the time
> {{Driver.openTransaction();}} is called, we have already parsed the query so
> there should be a way to know if the statement only reads.
> For multi-stmt txns (once these are supported) we should allow user to
> indicate that a txn is read-only and then not allow any statements that can
> make modifications in this txn. This should be a different jira.
> cc [~ikryvenko]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22433) Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and TIMESTAMP DataType From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966592#comment-16966592
]
Syed Shameerur Rahman commented on HIVE-22433:
--
[~jcamachorodriguez] Updated PR addressing your comments
> Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and
> TIMESTAMP DataType From JDBC Data Source
> --
>
> Key: HIVE-22433
> URL: https://issues.apache.org/jira/browse/HIVE-22433
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22433.01.patch, HIVE-22433.02.patch,
> HIVE-22433.03.patch, HIVE-22433.04.patch, HIVE-22433.05.patch
>
>
> Steps to Reproduce:
> {code:java}
> //Derby table:
> create table testtbl(a BOOLEAN, b TIMESTAMP);
> // Insert to table via mysql connector
> // data in db
> true 2019-11-11 12:00:00
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a BOOLEAN, b TIMESTAMP
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='DERBY',
> 'hive.sql.dbcp.password'='',
> 'hive.sql.dbcp.username'='',
> 'hive.sql.jdbc.driver'='',
> 'hive.sql.jdbc.url'='',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> // result from select query
> false 2019-11-11 20:00:00
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22451) Secure LLAP configurations are still deemed unsecure in Tez AM processes
[
https://issues.apache.org/jira/browse/HIVE-22451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-22451:
--
Assignee: Ádám Szita
Status: Patch Available (was: Open)
> Secure LLAP configurations are still deemed unsecure in Tez AM processes
>
>
> Key: HIVE-22451
> URL: https://issues.apache.org/jira/browse/HIVE-22451
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Ádám Szita
>Assignee: Ádám Szita
>Priority: Major
> Attachments: HIVE-22451.0.patch
>
>
> Due to the change in HIVE-22354 and HIVE-22195 Zookeeper discovery of LLAP
> workers is not working when invoked from within a Tez AM process: a Tez AM
> process does not log on using Kerberos even in secure environments, hence
> {code:java}
> UserGroupInformation.getLoginUser().hasKerberosCredentials() {code}
> will return false for security-enabled clusters too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22451) Secure LLAP configurations are still deemed unsecure in Tez AM processes
[
https://issues.apache.org/jira/browse/HIVE-22451?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ádám Szita updated HIVE-22451:
--
Attachment: HIVE-22451.0.patch
> Secure LLAP configurations are still deemed unsecure in Tez AM processes
>
>
> Key: HIVE-22451
> URL: https://issues.apache.org/jira/browse/HIVE-22451
> Project: Hive
> Issue Type: Bug
> Components: llap
>Reporter: Ádám Szita
>Priority: Major
> Attachments: HIVE-22451.0.patch
>
>
> Due to the change in HIVE-22354 and HIVE-22195 Zookeeper discovery of LLAP
> workers is not working when invoked from within a Tez AM process: a Tez AM
> process does not log on using Kerberos even in secure environments, hence
> {code:java}
> UserGroupInformation.getLoginUser().hasKerberosCredentials() {code}
> will return false for security-enabled clusters too.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22292:
--
Status: Patch Available (was: Open)
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22292:
--
Attachment: HIVE-22292.7.patch
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Krisztian Kasa updated HIVE-22292:
--
Status: Open (was: Patch Available)
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch,
> HIVE-22292.6.patch, HIVE-22292.7.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966552#comment-16966552
]
Hive QA commented on HIVE-22292:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984769/HIVE-22292.6.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 17546 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[udaf_percentile_disc]
(batchId=23)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19278/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19278/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19278/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984769 - PreCommit-HIVE-Build
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch, HIVE-22292.6.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22292) Implement Hypothetical-Set Aggregate Functions
[
https://issues.apache.org/jira/browse/HIVE-22292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966524#comment-16966524
]
Hive QA commented on HIVE-22292:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
12s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} ql: The patch generated 0 new + 509 unchanged - 4
fixed = 509 total (was 513) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 53s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19278/dev-support/hive-personality.sh
|
| git revision | master / 2a7bbba |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19278/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Implement Hypothetical-Set Aggregate Functions
> --
>
> Key: HIVE-22292
> URL: https://issues.apache.org/jira/browse/HIVE-22292
> Project: Hive
> Issue Type: Improvement
> Components: UDF
>Reporter: Krisztian Kasa
>Assignee: Krisztian Kasa
>Priority: Major
> Fix For: 4.0.0
>
> Attachments: HIVE-22292.1.patch, HIVE-22292.2.patch,
> HIVE-22292.3.patch, HIVE-22292.4.patch, HIVE-22292.5.patch, HIVE-22292.6.patch
>
>
> {code}
> ::=
>
>
>
> ::=
> RANK
> | DENSE_RANK
> | PERCENT_RANK
> | CUME_DIST
> {code}
> Example:
> {code}
> CREATE TABLE table1 (column1 int);
> INSERT INTO table1 VALUES (NULL), (3), (8), (13), (7), (6), (20), (NULL),
> (NULL), (10), (7), (15), (16), (8), (7), (8), (NULL);
> {code}
> {code}
> SELECT rank(6) WITHIN GROUP (ORDER BY column1) FROM table1;
> {code}
> {code}
> 2
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22420) DbTxnManager.stopHeartbeat() should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966523#comment-16966523
]
Peter Vary commented on HIVE-22420:
---
[~hamvas.aron]: LGTM +1, as discussed please throw an exception in
startHeartBeat, since that is still not reentrant.
> DbTxnManager.stopHeartbeat() should be thread-safe
> --
>
> Key: HIVE-22420
> URL: https://issues.apache.org/jira/browse/HIVE-22420
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
> Attachments: HIVE-22420.1.patch, HIVE-22420.2.patch
>
>
> When a transactional query is being executed and interrupted via HS2 close
> operation request, both the background pool thread executing the query and
> the HttpHandler thread running the close operation logic will eventually call
> the below method:
> {noformat}
> Driver.releaseLocksAndCommitOrRollback(commit boolean)
> {noformat}
> Since this method is invoked several times in both threads, it can happen
> that the two threads invoke it at the same time, and due to a race condition,
> the txnId field of the DbTxnManager used by both threads could be set to 0
> without actually successfully aborting the transaction.
> The root cause is stopHeartbeat() method in DbTxnManager not being thread
> safe:
> When Thread-1 and Thread-2 enter stopHeartbeat() with very little time
> difference, Thread-1 might successfully cancel the heartbeat task and set the
> heartbeatTask field to null, while Thread-2 is trying to observe its state.
> Thread-1 will return to the calling rollbackTxn() method and continue
> execution there, while Thread-2 wis thrown back to the same method with a
> NullPointerException. Thread-2 will then set txnId to 0, and Thread-1 is
> sending this 0 value to HMS. So, the txn will not be aborted, and the locks
> cannot be released later on either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22420) DbTxnManager.stopHeartbeat() should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aron Hamvas updated HIVE-22420:
---
Attachment: HIVE-22420.2.patch
Status: Patch Available (was: In Progress)
Do not allow starting multiple heartbeaters for same transaction.
> DbTxnManager.stopHeartbeat() should be thread-safe
> --
>
> Key: HIVE-22420
> URL: https://issues.apache.org/jira/browse/HIVE-22420
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
> Attachments: HIVE-22420.1.patch, HIVE-22420.2.patch
>
>
> When a transactional query is being executed and interrupted via HS2 close
> operation request, both the background pool thread executing the query and
> the HttpHandler thread running the close operation logic will eventually call
> the below method:
> {noformat}
> Driver.releaseLocksAndCommitOrRollback(commit boolean)
> {noformat}
> Since this method is invoked several times in both threads, it can happen
> that the two threads invoke it at the same time, and due to a race condition,
> the txnId field of the DbTxnManager used by both threads could be set to 0
> without actually successfully aborting the transaction.
> The root cause is stopHeartbeat() method in DbTxnManager not being thread
> safe:
> When Thread-1 and Thread-2 enter stopHeartbeat() with very little time
> difference, Thread-1 might successfully cancel the heartbeat task and set the
> heartbeatTask field to null, while Thread-2 is trying to observe its state.
> Thread-1 will return to the calling rollbackTxn() method and continue
> execution there, while Thread-2 wis thrown back to the same method with a
> NullPointerException. Thread-2 will then set txnId to 0, and Thread-1 is
> sending this 0 value to HMS. So, the txn will not be aborted, and the locks
> cannot be released later on either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-22420) DbTxnManager.stopHeartbeat() should be thread-safe
[
https://issues.apache.org/jira/browse/HIVE-22420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aron Hamvas updated HIVE-22420:
---
Status: In Progress (was: Patch Available)
> DbTxnManager.stopHeartbeat() should be thread-safe
> --
>
> Key: HIVE-22420
> URL: https://issues.apache.org/jira/browse/HIVE-22420
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.1.0
>Reporter: Aron Hamvas
>Assignee: Aron Hamvas
>Priority: Major
> Attachments: HIVE-22420.1.patch
>
>
> When a transactional query is being executed and interrupted via HS2 close
> operation request, both the background pool thread executing the query and
> the HttpHandler thread running the close operation logic will eventually call
> the below method:
> {noformat}
> Driver.releaseLocksAndCommitOrRollback(commit boolean)
> {noformat}
> Since this method is invoked several times in both threads, it can happen
> that the two threads invoke it at the same time, and due to a race condition,
> the txnId field of the DbTxnManager used by both threads could be set to 0
> without actually successfully aborting the transaction.
> The root cause is stopHeartbeat() method in DbTxnManager not being thread
> safe:
> When Thread-1 and Thread-2 enter stopHeartbeat() with very little time
> difference, Thread-1 might successfully cancel the heartbeat task and set the
> heartbeatTask field to null, while Thread-2 is trying to observe its state.
> Thread-1 will return to the calling rollbackTxn() method and continue
> execution there, while Thread-2 wis thrown back to the same method with a
> NullPointerException. Thread-2 will then set txnId to 0, and Thread-1 is
> sending this 0 value to HMS. So, the txn will not be aborted, and the locks
> cannot be released later on either.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HIVE-22433) Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and TIMESTAMP DataType From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966498#comment-16966498
]
Hive QA commented on HIVE-22433:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12984766/HIVE-22433.05.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 17546 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/19277/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/19277/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-19277/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12984766 - PreCommit-HIVE-Build
> Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and
> TIMESTAMP DataType From JDBC Data Source
> --
>
> Key: HIVE-22433
> URL: https://issues.apache.org/jira/browse/HIVE-22433
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22433.01.patch, HIVE-22433.02.patch,
> HIVE-22433.03.patch, HIVE-22433.04.patch, HIVE-22433.05.patch
>
>
> Steps to Reproduce:
> {code:java}
> //Derby table:
> create table testtbl(a BOOLEAN, b TIMESTAMP);
> // Insert to table via mysql connector
> // data in db
> true 2019-11-11 12:00:00
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a BOOLEAN, b TIMESTAMP
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
> 'hive.sql.database.type'='DERBY',
> 'hive.sql.dbcp.password'='',
> 'hive.sql.dbcp.username'='',
> 'hive.sql.jdbc.driver'='',
> 'hive.sql.jdbc.url'='',
> 'hive.sql.table'='testtbl');
> //Hive query:
> select * from hive_table;
> // result from select query
> false 2019-11-11 20:00:00
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (HIVE-9452) Use HBase to store Hive metadata
[ https://issues.apache.org/jira/browse/HIVE-9452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Haihua updated HIVE-9452: -- Description: qThis is an umbrella JIRA for a project to explore using HBase to store the Hive data catalog (ie the metastore). This project has several goals: # The current metastore implementation is slow when tables have thousands or more partitions. With Tez and Spark engines we are pushing Hive to a point where queries only take a few seconds to run. But planning the query can take as long as running it. Much of this time is spent in metadata operations. # Due to scale limitations we have never allowed tasks to communicate directly with the metastore. However, with the development of LLAP this requirement will have to be relaxed. If we can relax this there are other use cases that could benefit from this. # Eating our own dogfood. Rather than using external systems to store our metadata there are benefits to using other components in the Hadoop system. The proposal is to create a new branch and work on the prototype there. was: This is an umbrella JIRA for a project to explore using HBase to store the Hive data catalog (ie the metastore). This project has several goals: # The current metastore implementation is slow when tables have thousands or more partitions. With Tez and Spark engines we are pushing Hive to a point where queries only take a few seconds to run. But planning the query can take as long as running it. Much of this time is spent in metadata operations. # Due to scale limitations we have never allowed tasks to communicate directly with the metastore. However, with the development of LLAP this requirement will have to be relaxed. If we can relax this there are other use cases that could benefit from this. # Eating our own dogfood. Rather than using external systems to store our metadata there are benefits to using other components in the Hadoop system. The proposal is to create a new branch and work on the prototype there. > Use HBase to store Hive metadata > > > Key: HIVE-9452 > URL: https://issues.apache.org/jira/browse/HIVE-9452 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: hbase-metastore-branch >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Major > Attachments: HBaseMetastoreApproach.pdf > > > qThis is an umbrella JIRA for a project to explore using HBase to store the > Hive data catalog (ie the metastore). This project has several goals: > # The current metastore implementation is slow when tables have thousands or > more partitions. With Tez and Spark engines we are pushing Hive to a point > where queries only take a few seconds to run. But planning the query can > take as long as running it. Much of this time is spent in metadata > operations. > # Due to scale limitations we have never allowed tasks to communicate > directly with the metastore. However, with the development of LLAP this > requirement will have to be relaxed. If we can relax this there are other > use cases that could benefit from this. > # Eating our own dogfood. Rather than using external systems to store our > metadata there are benefits to using other components in the Hadoop system. > The proposal is to create a new branch and work on the prototype there. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Attachment: (was: HIVE-22411.1.patch) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Status: Patch Available (was: Open) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Status: Open (was: Patch Available) > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22411) Performance degradation on single row inserts
[ https://issues.apache.org/jira/browse/HIVE-22411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar updated HIVE-22411: - Attachment: HIVE-22411.1.patch > Performance degradation on single row inserts > - > > Key: HIVE-22411 > URL: https://issues.apache.org/jira/browse/HIVE-22411 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Attila Magyar >Assignee: Attila Magyar >Priority: Major > Fix For: 4.0.0 > > Attachments: HIVE-22411.1.patch, Screen Shot 2019-10-17 at 8.40.50 > PM.png > > > Executing single insert statements on a transactional table effects write > performance on a s3 file system. Each insert creates a new delta directory. > After each insert hive calculates statistics like number of file in the table > and total size of the table. In order to calculate these, it traverses the > directory recursively. During the recursion for each path a separate > listStatus call is executed. In the end the more delta directory you have the > more time it takes to calculate the statistics. > Therefore insertion time goes up linearly: > !Screen Shot 2019-10-17 at 8.40.50 PM.png|width=601,height=436! > The fix is to use fs.listFiles(path, /**recursive**/ true) instead the > handcrafter recursive method/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HIVE-22433) Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and TIMESTAMP DataType From JDBC Data Source
[
https://issues.apache.org/jira/browse/HIVE-22433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16966468#comment-16966468
]
Hive QA commented on HIVE-22433:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
53s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
30s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 4m
5s{color} | {color:blue} ql in master has 1550 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} jdbc-handler in master has 11 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.43-2+deb8u5 (2017-09-19) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-19277/dev-support/hive-personality.sh
|
| git revision | master / 2a7bbba |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.1 |
| modules | C: ql jdbc-handler U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-19277/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Hive JDBC Storage Handler: Incorrect results fetched from BOOLEAN and
> TIMESTAMP DataType From JDBC Data Source
> --
>
> Key: HIVE-22433
> URL: https://issues.apache.org/jira/browse/HIVE-22433
> Project: Hive
> Issue Type: Bug
> Components: StorageHandler
>Reporter: Syed Shameerur Rahman
>Assignee: Syed Shameerur Rahman
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-22433.01.patch, HIVE-22433.02.patch,
> HIVE-22433.03.patch, HIVE-22433.04.patch, HIVE-22433.05.patch
>
>
> Steps to Reproduce:
> {code:java}
> //Derby table:
> create table testtbl(a BOOLEAN, b TIMESTAMP);
> // Insert to table via mysql connector
> // data in db
> true 2019-11-11 12:00:00
> //Hive table:
> CREATE EXTERNAL TABLE `hive_table`(
> a BOOLEAN, b TIMESTAMP
> )
> STORED BY
> 'org.apache.hive.storage.jdbc.JdbcStorageHandler'
> TBLPROPERTIES (
>
[jira] [Updated] (HIVE-22414) Make LLAP CacheTags more memory efficient
[ https://issues.apache.org/jira/browse/HIVE-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22414: -- Status: Patch Available (was: In Progress) > Make LLAP CacheTags more memory efficient > - > > Key: HIVE-22414 > URL: https://issues.apache.org/jira/browse/HIVE-22414 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22414.0.patch, HIVE-22414.1.patch, > HIVE-22414.2.patch, HIVE-22414.3.patch > > > MultiPartitionCacheTag relies on LinkedLists. > A LinkedList object that holds e.g. 2 nodes consumes 112 bytes roughly in > this composition: > * 16 bytes for LinkedList object header > * 8 bytes for referring head > * 8 bytes for referring tail > * 80 = 2 x (16 bytes for LinkedList$Node header, 3 x 8 bytes for referring > prev, next, item) > This is a lot, so I propose to replace LinkedList in MultiPartitionCacheTag > with a simple String array. (For a similar scenario an array would take 16 + > 8 + 2 x 8 = 40 bytes, as per header, count, and 2 references for our actual > objects). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22414) Make LLAP CacheTags more memory efficient
[ https://issues.apache.org/jira/browse/HIVE-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22414: -- Attachment: HIVE-22414.3.patch > Make LLAP CacheTags more memory efficient > - > > Key: HIVE-22414 > URL: https://issues.apache.org/jira/browse/HIVE-22414 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22414.0.patch, HIVE-22414.1.patch, > HIVE-22414.2.patch, HIVE-22414.3.patch > > > MultiPartitionCacheTag relies on LinkedLists. > A LinkedList object that holds e.g. 2 nodes consumes 112 bytes roughly in > this composition: > * 16 bytes for LinkedList object header > * 8 bytes for referring head > * 8 bytes for referring tail > * 80 = 2 x (16 bytes for LinkedList$Node header, 3 x 8 bytes for referring > prev, next, item) > This is a lot, so I propose to replace LinkedList in MultiPartitionCacheTag > with a simple String array. (For a similar scenario an array would take 16 + > 8 + 2 x 8 = 40 bytes, as per header, count, and 2 references for our actual > objects). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22414) Make LLAP CacheTags more memory efficient
[ https://issues.apache.org/jira/browse/HIVE-22414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ádám Szita updated HIVE-22414: -- Status: In Progress (was: Patch Available) > Make LLAP CacheTags more memory efficient > - > > Key: HIVE-22414 > URL: https://issues.apache.org/jira/browse/HIVE-22414 > Project: Hive > Issue Type: Improvement > Components: llap >Reporter: Ádám Szita >Assignee: Ádám Szita >Priority: Major > Attachments: HIVE-22414.0.patch, HIVE-22414.1.patch, > HIVE-22414.2.patch, HIVE-22414.3.patch > > > MultiPartitionCacheTag relies on LinkedLists. > A LinkedList object that holds e.g. 2 nodes consumes 112 bytes roughly in > this composition: > * 16 bytes for LinkedList object header > * 8 bytes for referring head > * 8 bytes for referring tail > * 80 = 2 x (16 bytes for LinkedList$Node header, 3 x 8 bytes for referring > prev, next, item) > This is a lot, so I propose to replace LinkedList in MultiPartitionCacheTag > with a simple String array. (For a similar scenario an array would take 16 + > 8 + 2 x 8 = 40 bytes, as per header, count, and 2 references for our actual > objects). -- This message was sent by Atlassian Jira (v8.3.4#803005)
