[jira] [Commented] (HIVE-17396) Support DPP with map joins where the source and target belong in the same stage
[
https://issues.apache.org/jira/browse/HIVE-17396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348148#comment-16348148
]
Hive QA commented on HIVE-17396:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
22s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
3s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 3s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 419593e |
| Default Java | 1.8.0_111 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8963/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support DPP with map joins where the source and target belong in the same
> stage
> ---
>
> Key: HIVE-17396
> URL: https://issues.apache.org/jira/browse/HIVE-17396
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-17396.1.patch, HIVE-17396.2.patch,
> HIVE-17396.3.patch, HIVE-17396.4.patch, HIVE-17396.5.patch,
> HIVE-17396.6.patch, HIVE-17396.7.patch, HIVE-17396.8.patch, HIVE-17396.9.patch
>
>
> When the target of a partition pruning sink operator is in not the same as
> the target of hash table sink operator, both source and target gets scheduled
> within the same spark job, and that can result in File Not Found Exception.
> HIVE-17225 has a fix to disable DPP in that scenario. This JIRA is to
> support DPP for such cases.
> Test Case:
> SET hive.spark.dynamic.partition.pruning=true;
> SET hive.auto.convert.join=true;
> SET hive.strict.checks.cartesian.product=false;
> CREATE TABLE part_table1 (col int) PARTITIONED BY (part1_col int);
> CREATE TABLE part_table2 (col int) PARTITIONED BY (part2_col int);
> CREATE TABLE reg_table (col int);
> ALTER TABLE part_table1 ADD PARTITION (part1_col = 1);
> ALTER TABLE part_table2 ADD PARTITION (part2_col = 1);
> ALTER TABLE part_table2 ADD PARTITION (part2_col = 2);
> INSERT INTO TABLE part_table1 PARTITION (part1_col = 1) VALUES (1);
> INSERT INTO TABLE part_table2
[jira] [Commented] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[
https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348136#comment-16348136
]
Hive QA commented on HIVE-18516:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908708/HIVE-18516.9.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8962/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8962/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8962/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-02-01 07:27:45.167
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-8962/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-02-01 07:27:45.170
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 419593e HIVE-18478: Data files deleted from temp table should
not be recycled to CM path (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 419593e HIVE-18478: Data files deleted from temp table should
not be recycled to CM path (Mahesh Kumar Behera, reviewed by Sankar Hariappan)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-02-01 07:27:47.550
+ rm -rf ../yetus
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8962/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/itests/src/test/resources/testconfiguration.properties: does not exist
in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/LoadSemanticAnalyzer.java:
does not exist in index
error: a/ql/src/test/org/apache/hadoop/hive/ql/TestTxnLoadData.java: does not
exist in index
error: a/ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHiveCopyFiles.java:
does not exist in index
error: a/ql/src/test/queries/clientnegative/load_data_into_acid.q: does not
exist in index
error: a/ql/src/test/queries/clientpositive/smb_mapjoin_7.q: does not exist in
index
error: a/ql/src/test/results/clientnegative/load_data_into_acid.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/beeline/smb_mapjoin_7.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/smb_mapjoin_7.q.out: does not exist
in index
error: a/ql/src/test/results/clientpositive/spark/smb_mapjoin_7.q.out: does not
exist in index
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:1881
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
with conflicts.
Going to apply patch with: git apply -p1
error: patch failed:
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:1881
Falling back to three-way merge...
Applied patch to 'ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java'
with conflicts.
U ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908708 - PreCommit-HIVE-Build
> load data should rename files consistent with insert statements for ACID
> Tables
> ---
>
> Key: HIVE-18516
> URL: https://issues.apache.org/jira/browse/HIVE-18516
> Project: Hive
> Issue Type: Bug
>
[jira] [Commented] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348135#comment-16348135
]
Hive QA commented on HIVE-18552:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908625/HIVE-18552.4.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 12966 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8961/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8961/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8961/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908625 - PreCommit-HIVE-Build
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348087#comment-16348087
]
Hive QA commented on HIVE-18552:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
23s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 16m 9s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 419593e |
| Default Java | 1.8.0_111 |
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8961/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18543) Add print sessionid in console
[ https://issues.apache.org/jira/browse/HIVE-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] tartarus updated HIVE-18543: Resolution: Done Status: Resolved (was: Patch Available) This has been merged by HIVE-15631 > Add print sessionid in console > -- > > Key: HIVE-18543 > URL: https://issues.apache.org/jira/browse/HIVE-18543 > Project: Hive > Issue Type: Improvement > Components: CLI, Clients >Affects Versions: 2.3.2 > Environment: CentOS6.5 > Hive-1.2.1 > Hive-2.3.2 >Reporter: tartarus >Assignee: tartarus >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE_18543.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Hive client log file already contains sessionid information, but the console > does not have sessionid information, the user can not be associated with the > log well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18543) Add print sessionid in console
[ https://issues.apache.org/jira/browse/HIVE-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348063#comment-16348063 ] tartarus commented on HIVE-18543: - This has been merged by HIVE-15631 > Add print sessionid in console > -- > > Key: HIVE-18543 > URL: https://issues.apache.org/jira/browse/HIVE-18543 > Project: Hive > Issue Type: Improvement > Components: CLI, Clients >Affects Versions: 2.3.2 > Environment: CentOS6.5 > Hive-1.2.1 > Hive-2.3.2 >Reporter: tartarus >Assignee: tartarus >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE_18543.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > Hive client log file already contains sessionid information, but the console > does not have sessionid information, the user can not be associated with the > log well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18586) Upgrade Derby to 10.14.1.0
[
https://issues.apache.org/jira/browse/HIVE-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348061#comment-16348061
]
Hive QA commented on HIVE-18586:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908624/HIVE-18586.2.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12965 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_sortmerge_join_2]
(batchId=49)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=301)
org.apache.hadoop.hive.metastore.TestAcidTableSetup.testTransactionalValidation
(batchId=223)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8960/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8960/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8960/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908624 - PreCommit-HIVE-Build
> Upgrade Derby to 10.14.1.0
> --
>
> Key: HIVE-18586
> URL: https://issues.apache.org/jira/browse/HIVE-18586
> Project: Hive
> Issue Type: Improvement
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-18586.1.patch, HIVE-18586.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18598) Disallow constraints with EXTERNAL table
[ https://issues.apache.org/jira/browse/HIVE-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348054#comment-16348054 ] Vineet Garg commented on HIVE-18598: First patch only patches CREATE TABLE statement. ALTER TABLE is yet to be patched up. > Disallow constraints with EXTERNAL table > > > Key: HIVE-18598 > URL: https://issues.apache.org/jira/browse/HIVE-18598 > Project: Hive > Issue Type: Task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18598.1.patch > > > Since Hive do not manage the data for external tables and can not enforce > constraints it doesn't make sense to allow constraints like PRIMARY KEY, NOT > NULL etc for such tables. > User can still specify RELY to signal optimizer for constraint related > optimizations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18598) Disallow constraints with EXTERNAL table
[ https://issues.apache.org/jira/browse/HIVE-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18598: --- Status: Patch Available (was: Open) > Disallow constraints with EXTERNAL table > > > Key: HIVE-18598 > URL: https://issues.apache.org/jira/browse/HIVE-18598 > Project: Hive > Issue Type: Task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18598.1.patch > > > Since Hive do not manage the data for external tables and can not enforce > constraints it doesn't make sense to allow constraints like PRIMARY KEY, NOT > NULL etc for such tables. > User can still specify RELY to signal optimizer for constraint related > optimizations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18598) Disallow constraints with EXTERNAL table
[ https://issues.apache.org/jira/browse/HIVE-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-18598: --- Attachment: HIVE-18598.1.patch > Disallow constraints with EXTERNAL table > > > Key: HIVE-18598 > URL: https://issues.apache.org/jira/browse/HIVE-18598 > Project: Hive > Issue Type: Task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18598.1.patch > > > Since Hive do not manage the data for external tables and can not enforce > constraints it doesn't make sense to allow constraints like PRIMARY KEY, NOT > NULL etc for such tables. > User can still specify RELY to signal optimizer for constraint related > optimizations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18598) Disallow constraints with EXTERNAL table
[ https://issues.apache.org/jira/browse/HIVE-18598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg reassigned HIVE-18598: -- > Disallow constraints with EXTERNAL table > > > Key: HIVE-18598 > URL: https://issues.apache.org/jira/browse/HIVE-18598 > Project: Hive > Issue Type: Task > Components: SQL >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Attachments: HIVE-18598.1.patch > > > Since Hive do not manage the data for external tables and can not enforce > constraints it doesn't make sense to allow constraints like PRIMARY KEY, NOT > NULL etc for such tables. > User can still specify RELY to signal optimizer for constraint related > optimizations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18586) Upgrade Derby to 10.14.1.0
[
https://issues.apache.org/jira/browse/HIVE-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348045#comment-16348045
]
Hive QA commented on HIVE-18586:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
39s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} standalone-metastore: The patch generated 0 new + 21
unchanged - 2 fixed = 21 total (was 23) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
12s{color} | {color:red} hcatalog/core: The patch generated 3 new + 17
unchanged - 0 fixed = 20 total (was 17) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch java-client passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
44s{color} | {color:red} root: The patch generated 3 new + 146 unchanged - 2
fixed = 149 total (was 148) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
47s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 45m 14s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 419593e |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8960/yetus/diff-checkstyle-hcatalog_core.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8960/yetus/diff-checkstyle-root.txt
|
| modules | C: standalone-metastore hcatalog/core hcatalog/webhcat/java-client
. U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8960/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade Derby to 10.14.1.0
> --
>
> Key: HIVE-18586
> URL: https://issues.apache.org/jira/browse/HIVE-18586
> Project: Hive
> Issue Type: Improvement
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-18586.1.patch, HIVE-18586.2.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15631) Optimize for hive client logs , you can filter the log for each session itself.
[ https://issues.apache.org/jira/browse/HIVE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348026#comment-16348026 ] tartarus commented on HIVE-15631: - [~prasanth_j] It's my pleasure to participate in the project's contribution. > Optimize for hive client logs , you can filter the log for each session > itself. > --- > > Key: HIVE-15631 > URL: https://issues.apache.org/jira/browse/HIVE-15631 > Project: Hive > Issue Type: Improvement > Components: CLI, Clients, Hive >Reporter: tartarus >Assignee: tartarus >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-15631.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > We have several hadoop cluster, about 15 thousand nodes. Every day we use > hive to submit above 100 thousand jobs. > So we have a large file of hive logs on every client host every day, but i > don not know the logs of my session submitted was which line. > So i hope to print the hive.session.id on every line of logs, and then i > could use grep to find the logs of my session submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[
https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16348007#comment-16348007
]
Hive QA commented on HIVE-18573:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908598/HIVE-18573.3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 191 failed/errored test(s), 12964 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_predicate_pushdown]
(batchId=240)
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[allcolref_in_udf]
(batchId=53)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_7]
(batchId=64)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ba_table_udfs]
(batchId=24)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_1]
(batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_5]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[bucketsortoptimize_insert_8]
(batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[char_udf1] (batchId=91)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[concat_op] (batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[constant_prop_3]
(batchId=44)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ctas_colname]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[decimal_udf] (batchId=9)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map_multi_distinct]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_map_skew]
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_noskew]
(batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby2_noskew_multi_distinct]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_map_ppr]
(batchId=8)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_map_ppr_multi_distinct]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr] (batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby_ppr_multi_distinct]
(batchId=58)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_bucket_sort]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insert0] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[insertoverwrite_bucket]
(batchId=69)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[interval_2] (batchId=56)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[interval_alt] (batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[macro_1] (batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[nonmr_fetch_threshold]
(batchId=83)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[orc_dictionary_threshold]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_ctas]
(batchId=67)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_join]
(batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_map_type_vectorization]
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=16)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_10]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_11]
(batchId=38)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_12]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_15]
(batchId=87)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_16]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_2]
(batchId=3)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_5]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_6]
(batchId=41)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_7]
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_9]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_not]
(batchId=78)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd2] (batchId=26)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppr_allchildsarenull]
(batchId=70)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[rand_partitionpruner3]
(batchId=80)
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[
https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347972#comment-16347972
]
Hive QA commented on HIVE-18573:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
23s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
49s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 60 unchanged - 0 fixed
= 61 total (was 60) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 77d7d09 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8959/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8959/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Use proper Calcite operator instead of UDFs
> ---
>
> Key: HIVE-18573
> URL: https://issues.apache.org/jira/browse/HIVE-18573
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Reporter: slim bouguerra
>Priority: Major
> Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch, HIVE-18573.patch
>
>
> Currently, Hive is mostly using user-defined black box sql operators during
> Query planning. It will be more beneficial to use proper calcite operators.
> Also, Use a single name for Extract operator instead of a different name for
> every Unit,
> Same for Floor function. This will allow unifying the treatment per operator.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18478) Data files deleted from temp table should not be recycled to CM path
[ https://issues.apache.org/jira/browse/HIVE-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347971#comment-16347971 ] Sankar Hariappan commented on HIVE-18478: - Patch committed to master. Thanks [~maheshk114] for the contribution! > Data files deleted from temp table should not be recycled to CM path > > > Key: HIVE-18478 > URL: https://issues.apache.org/jira/browse/HIVE-18478 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Minor > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18478.01.patch, HIVE-18478.02.patch, > HIVE-18478.03.patch > > > Drop TEMP table operation invokes deleteDir which moves the file to $CMROOT > which is not needed as temp tables need not be replicated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18478) Data files deleted from temp table should not be recycled to CM path
[ https://issues.apache.org/jira/browse/HIVE-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sankar Hariappan updated HIVE-18478: Resolution: Fixed Status: Resolved (was: Patch Available) > Data files deleted from temp table should not be recycled to CM path > > > Key: HIVE-18478 > URL: https://issues.apache.org/jira/browse/HIVE-18478 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Minor > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18478.01.patch, HIVE-18478.02.patch, > HIVE-18478.03.patch > > > Drop TEMP table operation invokes deleteDir which moves the file to $CMROOT > which is not needed as temp tables need not be replicated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18478) Data files deleted from temp table should not be recycled to CM path
[ https://issues.apache.org/jira/browse/HIVE-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347969#comment-16347969 ] Sankar Hariappan commented on HIVE-18478: - +1, Patch looks good to me. Will commit it to master in a short while. > Data files deleted from temp table should not be recycled to CM path > > > Key: HIVE-18478 > URL: https://issues.apache.org/jira/browse/HIVE-18478 > Project: Hive > Issue Type: Sub-task > Components: Hive, HiveServer2 >Reporter: mahesh kumar behera >Assignee: mahesh kumar behera >Priority: Minor > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18478.01.patch, HIVE-18478.02.patch, > HIVE-18478.03.patch > > > Drop TEMP table operation invokes deleteDir which moves the file to $CMROOT > which is not needed as temp tables need not be replicated -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[
https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347958#comment-16347958
]
Hive QA commented on HIVE-18467:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908564/HIVE-18467.1.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12966 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet
(batchId=215)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNoSuchTableInThisDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8958/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8958/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8958/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908564 - PreCommit-HIVE-Build
> support whole warehouse dump / load + create/drop database events
> -
>
> Key: HIVE-18467
> URL: https://issues.apache.org/jira/browse/HIVE-18467
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch
>
>
> A complete hive warehouse might be required to replicate to a DR site for
> certain use cases and rather than allowing only a database name in the REPL
> DUMP commands, we should allow dumping of all databases using the "*" option
> as in
> _REPL DUMP *_
> On the repl load side there will not be an option to specify the database
> name when loading from a location used to dump multiple databases, hence only
> _REPL LOAD FROM [location]_ would be supported when dumping via _REPL DUMP *_
> Additionally, incremental dumps will go through all events across databases
> in a warehouse and hence CREATE / DROP Database events have to be serialized
> correctly to allow repl load to create them correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: HIVE-18350.8.patch > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18350.1.patch, HIVE-18350.2.patch, > HIVE-18350.3.patch, HIVE-18350.4.patch, HIVE-18350.5.patch, > HIVE-18350.6.patch, HIVE-18350.7.patch, HIVE-18350.8.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the > bucket in non-strict mode. Hive assumes that the data belongs to same bucket > in a file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. > For existing tables in customer database, it is recommended to reload > bucketed tables otherwise if customer tries to run SMB join and there is a > bucket for which there is no split, then there is a possibility of getting > incorrect results. However, this is not a regression as it would happen even > without the patch. > With this patch however, and reloading data, the results should be correct. > For non-bucketed tables and external tables, there is no difference in > behavior and reloading data is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-15631) Optimize for hive client logs , you can filter the log for each session itself.
[ https://issues.apache.org/jira/browse/HIVE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-15631: - Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Test failures are unrelated. Committed patch to master! Thanks [~tartarus] for the contribution! > Optimize for hive client logs , you can filter the log for each session > itself. > --- > > Key: HIVE-15631 > URL: https://issues.apache.org/jira/browse/HIVE-15631 > Project: Hive > Issue Type: Improvement > Components: CLI, Clients, Hive >Reporter: tartarus >Assignee: tartarus >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-15631.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > We have several hadoop cluster, about 15 thousand nodes. Every day we use > hive to submit above 100 thousand jobs. > So we have a large file of hive logs on every client host every day, but i > don not know the logs of my session submitted was which line. > So i hope to print the hive.session.id on every line of logs, and then i > could use grep to find the logs of my session submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Attachment: HIVE-18516.9.patch > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch, HIVE-18516.2.patch, > HIVE-18516.3.patch, HIVE-18516.4.patch, HIVE-18516.5.patch, > HIVE-18516.6.patch, HIVE-18516.7.patch, HIVE-18516.8.patch, HIVE-18516.9.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[
https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347933#comment-16347933
]
Hive QA commented on HIVE-18467:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
6s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} standalone-metastore: The patch generated 5 new + 13
unchanged - 4 fixed = 18 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
40s{color} | {color:red} ql: The patch generated 4 new + 439 unchanged - 2
fixed = 443 total (was 441) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} itests/hive-unit: The patch generated 9 new + 12
unchanged - 3 fixed = 21 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 16s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 62ae111 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8958/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8958/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8958/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8958/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> support whole warehouse dump / load + create/drop database events
> -
>
> Key: HIVE-18467
> URL: https://issues.apache.org/jira/browse/HIVE-18467
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch
>
>
> A complete hive warehouse might be required to replicate to a DR site for
> certain use cases and rather than allowing only a database name in the REPL
> DUMP commands, we should allow dumping of all databases using the "*"
[jira] [Updated] (HIVE-18350) load data should rename files consistent with insert statements
[ https://issues.apache.org/jira/browse/HIVE-18350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18350: -- Attachment: HIVE-18350.7.patch > load data should rename files consistent with insert statements > --- > > Key: HIVE-18350 > URL: https://issues.apache.org/jira/browse/HIVE-18350 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18350.1.patch, HIVE-18350.2.patch, > HIVE-18350.3.patch, HIVE-18350.4.patch, HIVE-18350.5.patch, > HIVE-18350.6.patch, HIVE-18350.7.patch > > > Insert statements create files of format ending with _0, 0001_0 etc. > However, the load data uses the input file name. That results in inconsistent > naming convention which makes SMB joins difficult in some scenarios and may > cause trouble for other types of queries in future. > We need consistent naming convention. > For non-bucketed table, hive renames all the files regardless of how they > were named by the user. > For bucketed table, hive relies on user to name the files matching the > bucket in non-strict mode. Hive assumes that the data belongs to same bucket > in a file. In strict mode, loading bucketed table is disabled. > This will likely affect most of the tests which load data which is pretty > significant due to which it is further divided into two subtasks for smoother > merge. > For existing tables in customer database, it is recommended to reload > bucketed tables otherwise if customer tries to run SMB join and there is a > bucket for which there is no split, then there is a possibility of getting > incorrect results. However, this is not a regression as it would happen even > without the patch. > With this patch however, and reloading data, the results should be correct. > For non-bucketed tables and external tables, there is no difference in > behavior and reloading data is not needed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18546) Remove unnecessary code introduced in HIVE-14498
[
https://issues.apache.org/jira/browse/HIVE-18546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347911#comment-16347911
]
Hive QA commented on HIVE-18546:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908561/HIVE-18546.04.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8957/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8957/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8957/
Messages:
{noformat}
This message was trimmed, see log for full details
+ mkdir ../yetus
+ git gc
+ cp -R . ../yetus
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-8957/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/metastore/scripts/upgrade/derby/048-HIVE-14498.derby.sql: does not
exist in index
error: a/metastore/scripts/upgrade/derby/hive-schema-3.0.0.derby.sql: does not
exist in index
error: a/metastore/scripts/upgrade/derby/hive-txn-schema-3.0.0.derby.sql: does
not exist in index
error: a/metastore/scripts/upgrade/hive/hive-schema-3.0.0.hive.sql: does not
exist in index
error: a/metastore/scripts/upgrade/mssql/033-HIVE-14498.mssql.sql: does not
exist in index
error: a/metastore/scripts/upgrade/mssql/hive-schema-3.0.0.mssql.sql: does not
exist in index
error: a/metastore/scripts/upgrade/mysql/048-HIVE-14498.mysql.sql: does not
exist in index
error: a/metastore/scripts/upgrade/mysql/hive-schema-3.0.0.mysql.sql: does not
exist in index
error: a/metastore/scripts/upgrade/mysql/hive-txn-schema-3.0.0.mysql.sql: does
not exist in index
error: a/metastore/scripts/upgrade/oracle/048-HIVE-14498.oracle.sql: does not
exist in index
error: a/metastore/scripts/upgrade/oracle/hive-schema-3.0.0.oracle.sql: does
not exist in index
error: a/metastore/scripts/upgrade/oracle/hive-txn-schema-3.0.0.oracle.sql:
does not exist in index
error: a/metastore/scripts/upgrade/postgres/047-HIVE-14498.postgres.sql: does
not exist in index
error: a/metastore/scripts/upgrade/postgres/hive-schema-3.0.0.postgres.sql:
does not exist in index
error: a/metastore/scripts/upgrade/postgres/hive-txn-schema-3.0.0.postgres.sql:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/exec/DDLTask.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/metadata/Table.java: does not
exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/DDLSemanticAnalyzer.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/parse/SemanticAnalyzer.java:
does not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/CreateViewDesc.java: does
not exist in index
error: a/ql/src/java/org/apache/hadoop/hive/ql/plan/ImportTableDesc.java: does
not exist in index
error: a/ql/src/test/org/apache/hadoop/hive/ql/metadata/TestHive.java: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/sysdb.q.out: does not exist in
index
error: a/standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.cpp:
does not exist in index
error: a/standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore.h:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-cpp/ThriftHiveMetastore_server.skeleton.cpp:
does not exist in index
error: a/standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.cpp:
does not exist in index
error: a/standalone-metastore/src/gen/thrift/gen-cpp/hive_metastore_types.h:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AbortTxnsRequest.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddDynamicPartitions.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddForeignKeyRequest.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddNotNullConstraintRequest.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsRequest.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPartitionsResult.java:
does not exist in index
error:
a/standalone-metastore/src/gen/thrift/gen-javabean/org/apache/hadoop/hive/metastore/api/AddPrimaryKeyRequest.java:
does not exist in index
error:
[jira] [Commented] (HIVE-15631) Optimize for hive client logs , you can filter the log for each session itself.
[
https://issues.apache.org/jira/browse/HIVE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347905#comment-16347905
]
Hive QA commented on HIVE-15631:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908559/HIVE-15631.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 12964 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=301)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByValuesNoTblName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8956/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8956/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8956/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908559 - PreCommit-HIVE-Build
> Optimize for hive client logs , you can filter the log for each session
> itself.
> ---
>
> Key: HIVE-15631
> URL: https://issues.apache.org/jira/browse/HIVE-15631
> Project: Hive
> Issue Type: Improvement
> Components: CLI, Clients, Hive
>Reporter: tartarus
>Assignee: tartarus
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-15631.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> We have several hadoop cluster, about 15 thousand nodes. Every day we use
> hive to submit above 100 thousand jobs.
> So we have a large file of hive logs on every client host every day, but i
> don not know the logs of my session submitted was which line.
> So i hope to print the hive.session.id on every line of logs, and then i
> could use grep to find the logs of my session submitted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-15631) Optimize for hive client logs , you can filter the log for each session itself.
[
https://issues.apache.org/jira/browse/HIVE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347852#comment-16347852
]
Hive QA commented on HIVE-15631:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 62ae111 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8956/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Optimize for hive client logs , you can filter the log for each session
> itself.
> ---
>
> Key: HIVE-15631
> URL: https://issues.apache.org/jira/browse/HIVE-15631
> Project: Hive
> Issue Type: Improvement
> Components: CLI, Clients, Hive
>Reporter: tartarus
>Assignee: tartarus
>Priority: Major
> Labels: pull-request-available
> Attachments: HIVE-15631.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> We have several hadoop cluster, about 15 thousand nodes. Every day we use
> hive to submit above 100 thousand jobs.
> So we have a large file of hive logs on every client host every day, but i
> don not know the logs of my session submitted was which line.
> So i hope to print the hive.session.id on every line of logs, and then i
> could use grep to find the logs of my session submitted.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18543) Add print sessionid in console
[
https://issues.apache.org/jira/browse/HIVE-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347839#comment-16347839
]
Hive QA commented on HIVE-18543:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908554/HIVE_18543.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 22 failed/errored test(s), 12942 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=84)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestGetListIndexes.testGetIndexEmptyTableName[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8955/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8955/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8955/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 22 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908554 - PreCommit-HIVE-Build
> Add print sessionid in console
> --
>
> Key: HIVE-18543
> URL: https://issues.apache.org/jira/browse/HIVE-18543
> Project: Hive
> Issue Type: Improvement
> Components: CLI, Clients
>Affects Versions: 2.3.2
> Environment: CentOS6.5
> Hive-1.2.1
> Hive-2.3.2
>Reporter: tartarus
>Assignee: tartarus
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE_18543.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Hive client log file already contains sessionid information, but the console
> does not have sessionid information, the user can not be associated with the
> log well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18528) Stats: In the bitvector codepath, when extrapolating column stats for String type columnStringColumnStatsAggregator uses the min value instead of max
[ https://issues.apache.org/jira/browse/HIVE-18528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18528: Attachment: HIVE-18528.1.patch > Stats: In the bitvector codepath, when extrapolating column stats for String > type columnStringColumnStatsAggregator uses the min value instead of max > - > > Key: HIVE-18528 > URL: https://issues.apache.org/jira/browse/HIVE-18528 > Project: Hive > Issue Type: Bug > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18528.1.patch, HIVE-18528.1.patch > > > This line: > [https://github.com/apache/hive/blob/456a65180dcb84f69f26b4c9b9265165ad16dfe4/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/columnstats/aggr/StringColumnStatsAggregator.java#L181] > Should be: > aggregateData.setAvgColLen(Math.max(aggregateData.getAvgColLen(), > newData.getAvgColLen())); -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18543) Add print sessionid in console
[
https://issues.apache.org/jira/browse/HIVE-18543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347807#comment-16347807
]
Hive QA commented on HIVE-18543:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
5s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 20s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 62ae111 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8955/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add print sessionid in console
> --
>
> Key: HIVE-18543
> URL: https://issues.apache.org/jira/browse/HIVE-18543
> Project: Hive
> Issue Type: Improvement
> Components: CLI, Clients
>Affects Versions: 2.3.2
> Environment: CentOS6.5
> Hive-1.2.1
> Hive-2.3.2
>Reporter: tartarus
>Assignee: tartarus
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE_18543.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> Hive client log file already contains sessionid information, but the console
> does not have sessionid information, the user can not be associated with the
> log well.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18581) Replication events should use lower case db object names
[
https://issues.apache.org/jira/browse/HIVE-18581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347791#comment-16347791
]
Hive QA commented on HIVE-18581:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908539/HIVE-18581.0.patch
{color:green}SUCCESS:{color} +1 due to 6 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12961 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.TestMarkPartition.testMarkingPartitionSet
(batchId=215)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestTriggersMoveWorkloadManager.testTriggerMoveConflictKill
(batchId=235)
org.apache.hive.service.cli.thrift.TestThriftCLIServiceWithBinary.org.apache.hive.service.cli.thrift.TestThriftCLIServiceWithBinary
(batchId=232)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8954/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8954/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8954/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908539 - PreCommit-HIVE-Build
> Replication events should use lower case db object names
>
>
> Key: HIVE-18581
> URL: https://issues.apache.org/jira/browse/HIVE-18581
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: anishek
>Assignee: anishek
>Priority: Minor
> Attachments: HIVE-18581.0.patch
>
>
> events generated by replication should include the database / tables /
> partitions / function names in lower case. this will prevent other
> applications to explicitly do case insensitive match of objects using names.
> in hive all db object names as specified above are explicitly converted to
> lower case when comparing between objects of same types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18475) Vectorization of CASE with NULL makes unexpected NULL values
[ https://issues.apache.org/jira/browse/HIVE-18475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347765#comment-16347765 ] Eric Wohlstadter commented on HIVE-18475: - [~teddy.choi] Is this ready to merge? Can you explain how the problem was solved? > Vectorization of CASE with NULL makes unexpected NULL values > > > Key: HIVE-18475 > URL: https://issues.apache.org/jira/browse/HIVE-18475 > Project: Hive > Issue Type: Bug >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Critical > Attachments: HIVE-18475.patch > > > Vectorization of CASE with NULL (HIVE-16731) makes unexpected NULL values -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18590) Assertion error on transitive join inference in the presence of NOT NULL constraint
[ https://issues.apache.org/jira/browse/HIVE-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347750#comment-16347750 ] Ashutosh Chauhan commented on HIVE-18590: - +1 > Assertion error on transitive join inference in the presence of NOT NULL > constraint > --- > > Key: HIVE-18590 > URL: https://issues.apache.org/jira/browse/HIVE-18590 > Project: Hive > Issue Type: Bug > Components: Logical Optimizer >Affects Versions: 3.0.0 >Reporter: Jesus Camacho Rodriguez >Assignee: Jesus Camacho Rodriguez >Priority: Major > Attachments: HIVE-18590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18589: -- Attachment: HIVE-18589.02.patch > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-18589.01.patch, HIVE-18589.02.patch > > > txnid:17 starts reading T2/P1 > txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 > still running > txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. > before Insert overwrite was supported, only the compactor could produce > base_20 by running major compaction. Major compaction erases history and so > a reader with txnid:17 open, can't use base_20. > Normally, the Cleaner is smart enough to not clean pre-compaction files if > it's possible that there is a reader that requires them. There is a safety > check that creates "Not enough history.." error if it finds that the current > reader can't properly execute based on the files available. > > with the introduction of IOW on acid tables, there is another way to produce > a base. The difference is that here, the base has no history by definition > and so the same check is not needed but is triggered in the scenario above. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18581) Replication events should use lower case db object names
[
https://issues.apache.org/jira/browse/HIVE-18581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347722#comment-16347722
]
Hive QA commented on HIVE-18581:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
12s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
34s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 14 new + 6 unchanged - 0 fixed
= 20 total (was 6) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} itests/hive-unit: The patch generated 0 new + 643
unchanged - 2 fixed = 643 total (was 645) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8954/yetus/diff-checkstyle-ql.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8954/yetus/patch-asflicense-problems.txt
|
| modules | C: ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8954/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Replication events should use lower case db object names
>
>
> Key: HIVE-18581
> URL: https://issues.apache.org/jira/browse/HIVE-18581
> Project: Hive
> Issue Type: Bug
> Components: HiveServer2
>Reporter: anishek
>Assignee: anishek
>Priority: Minor
> Attachments: HIVE-18581.0.patch
>
>
> events generated by replication should include the database / tables /
> partitions / function names in lower case. this will prevent other
> applications to explicitly do case insensitive match of objects using names.
> in hive all db object names as specified above are explicitly converted to
> lower case when comparing between objects of same types.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18587) insert DML event may attempt to calculate a checksum on directories
[ https://issues.apache.org/jira/browse/HIVE-18587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18587: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Committed to master. Thanks for the review! > insert DML event may attempt to calculate a checksum on directories > --- > > Key: HIVE-18587 > URL: https://issues.apache.org/jira/browse/HIVE-18587 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-18587.patch > > > Looks like in union case, some code path may pass directories in newFiles. > Probably legacy copyData/moveData; both seem to assume that these paths are > files, but do not actually enforce it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18447) JDBC: Provide a way for JDBC users to pass cookie info via connection string
[ https://issues.apache.org/jira/browse/HIVE-18447?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-18447: Attachment: HIVE-18447.2.patch > JDBC: Provide a way for JDBC users to pass cookie info via connection string > > > Key: HIVE-18447 > URL: https://issues.apache.org/jira/browse/HIVE-18447 > Project: Hive > Issue Type: Bug > Components: JDBC >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-18447.1.patch, HIVE-18447.2.patch > > > Some authentication mechanisms like Single Sign On, need the ability to pass > a cookie to some intermediate authentication service like Knox via the JDBC > driver. We need to add the mechanism in Hive's JDBC driver (when used in HTTP > transport mode). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18589) java.io.IOException: Not enough history available
[ https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18589: -- Description: txnid:17 starts reading T2/P1 txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 still running txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. before Insert overwrite was supported, only the compactor could produce base_20 by running major compaction. Major compaction erases history and so a reader with txnid:17 open, can't use base_20. Normally, the Cleaner is smart enough to not clean pre-compaction files if it's possible that there is a reader that requires them. There is a safety check that creates "Not enough history.." error if it finds that the current reader can't properly execute based on the files available. with the introduction of IOW on acid tables, there is another way to produce a base. The difference is that here, the base has no history by definition and so the same check is not needed but is triggered in the scenario above. > java.io.IOException: Not enough history available > - > > Key: HIVE-18589 > URL: https://issues.apache.org/jira/browse/HIVE-18589 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > Attachments: HIVE-18589.01.patch > > > txnid:17 starts reading T2/P1 > txnid:20 does insert overwrite T1/P1, creates base_20, commits. txnid:17 > still running > txnid:21 stars reading T1/P1. It's ValidTxnList will txnid:17 as open. > before Insert overwrite was supported, only the compactor could produce > base_20 by running major compaction. Major compaction erases history and so > a reader with txnid:17 open, can't use base_20. > Normally, the Cleaner is smart enough to not clean pre-compaction files if > it's possible that there is a reader that requires them. There is a safety > check that creates "Not enough history.." error if it finds that the current > reader can't properly execute based on the files available. > > with the introduction of IOW on acid tables, there is another way to produce > a base. The difference is that here, the base has no history by definition > and so the same check is not needed but is triggered in the scenario above. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18587) insert DML event may attempt to calculate a checksum on directories
[ https://issues.apache.org/jira/browse/HIVE-18587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347705#comment-16347705 ] Sergey Shelukhin commented on HIVE-18587: - [~ashutoshc] the thing is, existing code that returns directories never needs to traverse these files, and I suspect propagating it from FSOP will be a major pain. E.g. MM tables have to write a separate file for that on FS, cause there's no good channel otherwise. So it makes sense to do it only for the events. > insert DML event may attempt to calculate a checksum on directories > --- > > Key: HIVE-18587 > URL: https://issues.apache.org/jira/browse/HIVE-18587 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18587.patch > > > Looks like in union case, some code path may pass directories in newFiles. > Probably legacy copyData/moveData; both seem to assume that these paths are > files, but do not actually enforce it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyunzhang updated HIVE-18301:
--
Attachment: HIVE-18301.2.patch
> Investigate to enable MapInput cache in Hive on Spark
> -
>
> Key: HIVE-18301
> URL: https://issues.apache.org/jira/browse/HIVE-18301
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>Priority: Major
> Attachments: HIVE-18301.1.patch, HIVE-18301.2.patch, HIVE-18301.patch
>
>
> Before IOContext problem is found in MapTran when spark rdd cache is enabled
> in HIVE-8920.
> so we disabled rdd cache in MapTran at
> [SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
> The problem is IOContext seems not initialized correctly in the spark yarn
> client/cluster mode and caused the exception like
> {code}
> Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most
> recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48):
> java.lang.RuntimeException: Error processing row:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
> at org.apache.spark.scheduler.Task.run(Task.scala:85)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
> ... 12 more
> Driver stacktrace:
> {code}
> in yarn client/cluster mode, sometimes
> [ExecMapperContext#currentInputPath|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java#L109]
> is null when rdd cach is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347696#comment-16347696
]
Hive QA commented on HIVE-16605:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908525/HIVE-16605.3.patch
{color:green}SUCCESS:{color} +1 due to 5 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 21 failed/errored test(s), 12967 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.TestHiveMetaTool.testListFSRoot (batchId=225)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionWithAuthInfoNoDbName[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8953/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8953/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8953/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 21 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908525 - PreCommit-HIVE-Build
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch,
> HIVE-16605.3.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18587) insert DML event may attempt to calculate a checksum on directories
[ https://issues.apache.org/jira/browse/HIVE-18587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347687#comment-16347687 ] Ashutosh Chauhan commented on HIVE-18587: - +1 Patch fixes issue at hand. However, traversing FS in move task will be expensive. List of new files created should be communicated back from job to avoid this traversal. That should be done in a follow-up. > insert DML event may attempt to calculate a checksum on directories > --- > > Key: HIVE-18587 > URL: https://issues.apache.org/jira/browse/HIVE-18587 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18587.patch > > > Looks like in union case, some code path may pass directories in newFiles. > Probably legacy copyData/moveData; both seem to assume that these paths are > files, but do not actually enforce it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Attachment: HIVE-18516.8.patch > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch, HIVE-18516.2.patch, > HIVE-18516.3.patch, HIVE-18516.4.patch, HIVE-18516.5.patch, > HIVE-18516.6.patch, HIVE-18516.7.patch, HIVE-18516.8.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18596) Synchronize value of hive.spark.client.connect.timeout across unit tests
[
https://issues.apache.org/jira/browse/HIVE-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18596:
Attachment: HIVE-18596.1.patch
> Synchronize value of hive.spark.client.connect.timeout across unit tests
>
>
> Key: HIVE-18596
> URL: https://issues.apache.org/jira/browse/HIVE-18596
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18596.1.patch
>
>
> {{hive.spark.client.connect.timeout}} is set to 30 seconds for
> {{TestMiniSparkOnYarnCliDriver}} but it left at the default value for all
> other tests. We should use the same value (30 seconds) for all other tests.
> We have seen flaky tests due to failure to establish the remote within the
> allotted timeout. This could be due to the fact that we run our tests in the
> cloud so maybe occasional network delays are more common.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18596) Synchronize value of hive.spark.client.connect.timeout across unit tests
[
https://issues.apache.org/jira/browse/HIVE-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18596:
Status: Patch Available (was: Open)
> Synchronize value of hive.spark.client.connect.timeout across unit tests
>
>
> Key: HIVE-18596
> URL: https://issues.apache.org/jira/browse/HIVE-18596
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18596.1.patch
>
>
> {{hive.spark.client.connect.timeout}} is set to 30 seconds for
> {{TestMiniSparkOnYarnCliDriver}} but it left at the default value for all
> other tests. We should use the same value (30 seconds) for all other tests.
> We have seen flaky tests due to failure to establish the remote within the
> allotted timeout. This could be due to the fact that we run our tests in the
> cloud so maybe occasional network delays are more common.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347653#comment-16347653
]
Vihang Karajgaonkar commented on HIVE-18552:
patch looks good to me. +1 Since this removes an existing config, we should
only merge it in Hive 3.0.0
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-16605) Enforce NOT NULL constraints
[
https://issues.apache.org/jira/browse/HIVE-16605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347645#comment-16347645
]
Hive QA commented on HIVE-16605:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
23s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
27s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
13s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
46s{color} | {color:red} ql: The patch generated 16 new + 1244 unchanged - 0
fixed = 1260 total (was 1244) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 15m 48s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8953/yetus/diff-checkstyle-ql.txt
|
| modules | C: common ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8953/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Enforce NOT NULL constraints
>
>
> Key: HIVE-16605
> URL: https://issues.apache.org/jira/browse/HIVE-16605
> Project: Hive
> Issue Type: New Feature
>Affects Versions: 3.0.0
>Reporter: Carter Shanklin
>Assignee: Vineet Garg
>Priority: Major
> Attachments: HIVE-16605.1.patch, HIVE-16605.2.patch,
> HIVE-16605.3.patch
>
>
> Since NOT NULL is so common it would be great to have tables start to enforce
> that.
> [~ekoifman] described a possible approach in HIVE-16575:
> {quote}
> One way to enforce not null constraint is to have the optimizer add
> enforce_not_null UDF which throws if it sees a NULL, otherwise it's pass
> through.
> So if 'b' has not null constraint,
> Insert into T select a,b,c... would become
> Insert into T select a, enforce_not_null(b), c.
> This would work for any table type.
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18552:
Release Note:
This change splits {{hive.strict.checks.large.query}} into two new configs:
{{hive.strict.checks.no.partition.filter}} and
{{hive.strict.checks.orderby.no.limit}}
Previously, {{hive.strict.checks.large.query}} controlled whether or not (1)
order bys were allowed without limits, and (2) scans on partitioned tables were
allowed without filters.
After this change, {{hive.strict.checks.no.partition.filter}} controls the
partitioned table check and {{hive.strict.checks.orderby.no.limit}} controls
the order by check.
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18552:
Hadoop Flags: Incompatible change
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (HIVE-18596) Synchronize value of hive.spark.client.connect.timeout across unit tests
[
https://issues.apache.org/jira/browse/HIVE-18596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar reassigned HIVE-18596:
---
> Synchronize value of hive.spark.client.connect.timeout across unit tests
>
>
> Key: HIVE-18596
> URL: https://issues.apache.org/jira/browse/HIVE-18596
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
>
> {{hive.spark.client.connect.timeout}} is set to 30 seconds for
> {{TestMiniSparkOnYarnCliDriver}} but it left at the default value for all
> other tests. We should use the same value (30 seconds) for all other tests.
> We have seen flaky tests due to failure to establish the remote within the
> allotted timeout. This could be due to the fact that we run our tests in the
> cloud so maybe occasional network delays are more common.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347618#comment-16347618 ] Eugene Koifman commented on HIVE-18536: --- +1 > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.02.patch, > HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18582) MSCK REPAIR TABLE Throw MetaException
[
https://issues.apache.org/jira/browse/HIVE-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347611#comment-16347611
]
Hive QA commented on HIVE-18582:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908522/HIVE-18582.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12964 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8952/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8952/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8952/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908522 - PreCommit-HIVE-Build
> MSCK REPAIR TABLE Throw MetaException
> --
>
> Key: HIVE-18582
> URL: https://issues.apache.org/jira/browse/HIVE-18582
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.1
>Reporter: liubangchen
>Assignee: liubangchen
>Priority: Major
> Attachments: HIVE-18582.patch
>
>
> while executing query MSCK REPAIR TABLE tablename I got Exception:
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Expected 1 components, got 2
> (log_date=2015121309/vgameid=lyjt))
> at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1847)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:402)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> --
> Caused by: MetaException(message:Expected 1 components, got 2
> (log_date=2015121309/vgameid=lyjt))
> at
> org.apache.hadoop.hive.metastore.Warehouse.makeValsFromName(Warehouse.java:385)
> at
[jira] [Commented] (HIVE-17370) Some tests are failing with java.io.FileNotFoundException: File file:/tmp/hadoop/mapred/...
[
https://issues.apache.org/jira/browse/HIVE-17370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347583#comment-16347583
]
Peter Vary commented on HIVE-17370:
---
On internal testing it was fixed with something like this:
{code:java}
mapreduce.jobtracker.staging.root.dir
${test.tmp.dir}/hadoop-tmp
A base for other temporary directories.
{code}
But sadly, I did not have time to look deeper into this, but seems like the
same problem
> Some tests are failing with java.io.FileNotFoundException: File
> file:/tmp/hadoop/mapred/...
> ---
>
> Key: HIVE-17370
> URL: https://issues.apache.org/jira/browse/HIVE-17370
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
>
> Mostly independently of the given tests, I found this flakiness in several
> tests:
> {code}
> 2017-08-21T06:34:26,916 WARN [Thread-7815] mapred.LocalJobRunner:
> job_local245886038_0060
> java.io.FileNotFoundException: File
> file:/tmp/hadoop/mapred/staging/hiveptest245886038/.staging/job_local245886038_0060/job.splitmetainfo
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:635)
> ~[hadoop-common-2.8.0.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:861)
> ~[hadoop-common-2.8.0.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:625)
> ~[hadoop-common-2.8.0.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:435)
> ~[hadoop-common-2.8.0.jar:?]
> at
> org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
> ~[hadoop-mapreduce-client-core-2.8.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:534)
> [hadoop-mapreduce-client-common-2.8.0.jar:?]
> {code}
> One example is:
> https://builds.apache.org/job/PreCommit-HIVE-Build/6469/testReport/org.apache.hadoop.hive.cli/TestAccumuloCliDriver/testCliDriver_accumulo_queries_/
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (HIVE-15353) Metastore throws NPE if StorageDescriptor.cols is null
[ https://issues.apache.org/jira/browse/HIVE-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carl Steinbach updated HIVE-15353: -- Comment: was deleted (was: +1) > Metastore throws NPE if StorageDescriptor.cols is null > -- > > Key: HIVE-15353 > URL: https://issues.apache.org/jira/browse/HIVE-15353 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15353.1.patch, HIVE-15353.2.patch, > HIVE-15353.3.patch, HIVE-15353.4.patch > > > When using the HiveMetaStoreClient API directly to talk to the metastore, you > get NullPointerExceptions when StorageDescriptor.cols is null in the > Table/Partition object in the following calls: > * create_table > * alter_table > * alter_partition > Calling add_partition with StorageDescriptor.cols set to null causes null to > be stored in the metastore database and subsequent calls to alter_partition > for that partition to fail with an NPE. > Null checks should be added to eliminate the NPEs in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15353) Metastore throws NPE if StorageDescriptor.cols is null
[ https://issues.apache.org/jira/browse/HIVE-15353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347582#comment-16347582 ] Carl Steinbach commented on HIVE-15353: --- +1 > Metastore throws NPE if StorageDescriptor.cols is null > -- > > Key: HIVE-15353 > URL: https://issues.apache.org/jira/browse/HIVE-15353 > Project: Hive > Issue Type: Bug >Affects Versions: 1.1.0, 2.2.0 >Reporter: Anthony Hsu >Assignee: Anthony Hsu >Priority: Major > Attachments: HIVE-15353.1.patch, HIVE-15353.2.patch, > HIVE-15353.3.patch, HIVE-15353.4.patch > > > When using the HiveMetaStoreClient API directly to talk to the metastore, you > get NullPointerExceptions when StorageDescriptor.cols is null in the > Table/Partition object in the following calls: > * create_table > * alter_table > * alter_partition > Calling add_partition with StorageDescriptor.cols set to null causes null to > be stored in the metastore database and subsequent calls to alter_partition > for that partition to fail with an NPE. > Null checks should be added to eliminate the NPEs in the metastore. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347579#comment-16347579 ] Thejas M Nair commented on HIVE-18536: -- [~sershe] IOW = insert over write and DP = dynamic partitioning ? It took me a minute to guess what these might mean, but not sure if i got it right! It would be easier for wider set of folks to understand what these jiras means if we don't keep inventing new TLAs! :) > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.02.patch, > HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347577#comment-16347577 ] Eugene Koifman edited comment on HIVE-18536 at 1/31/18 8:39 PM: I was asking about "Boolean baseDir" param on RB. why is it needed? I don't understand your answer was (Author: ekoifman): I was asking about "Boolean baseDir" param on RB. why is it needed? > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.02.patch, > HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347577#comment-16347577 ] Eugene Koifman commented on HIVE-18536: --- I was asking about "Boolean baseDir" param on RB. why is it needed? > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.02.patch, > HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17370) Some tests are failing with java.io.FileNotFoundException: File file:/tmp/hadoop/mapred/...
[
https://issues.apache.org/jira/browse/HIVE-17370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347556#comment-16347556
]
Sahil Takiar commented on HIVE-17370:
-
hmmm I've actually seen this happen a lot in some internal testing. Very
similar errors. Always a FNF exception that is thrown by {{LocalJobRunner}}.
Usually a missing {{job.splitmetainfo}} or {{job.split}} file. Maybe there is a
race condition inside the {{LocalJobRunner}}?
> Some tests are failing with java.io.FileNotFoundException: File
> file:/tmp/hadoop/mapred/...
> ---
>
> Key: HIVE-17370
> URL: https://issues.apache.org/jira/browse/HIVE-17370
> Project: Hive
> Issue Type: Sub-task
>Affects Versions: 3.0.0
>Reporter: Peter Vary
>Assignee: Peter Vary
>Priority: Major
>
> Mostly independently of the given tests, I found this flakiness in several
> tests:
> {code}
> 2017-08-21T06:34:26,916 WARN [Thread-7815] mapred.LocalJobRunner:
> job_local245886038_0060
> java.io.FileNotFoundException: File
> file:/tmp/hadoop/mapred/staging/hiveptest245886038/.staging/job_local245886038_0060/job.splitmetainfo
> does not exist
> at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:635)
> ~[hadoop-common-2.8.0.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:861)
> ~[hadoop-common-2.8.0.jar:?]
> at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:625)
> ~[hadoop-common-2.8.0.jar:?]
> at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:435)
> ~[hadoop-common-2.8.0.jar:?]
> at
> org.apache.hadoop.mapreduce.split.SplitMetaInfoReader.readSplitMetaInfo(SplitMetaInfoReader.java:51)
> ~[hadoop-mapreduce-client-core-2.8.0.jar:?]
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:534)
> [hadoop-mapreduce-client-common-2.8.0.jar:?]
> {code}
> One example is:
> https://builds.apache.org/job/PreCommit-HIVE-Build/6469/testReport/org.apache.hadoop.hive.cli/TestAccumuloCliDriver/testCliDriver_accumulo_queries_/
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18582) MSCK REPAIR TABLE Throw MetaException
[
https://issues.apache.org/jira/browse/HIVE-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347536#comment-16347536
]
Hive QA commented on HIVE-18582:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
7s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
1s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
34s{color} | {color:red} ql: The patch generated 8 new + 3 unchanged - 0 fixed
= 11 total (was 3) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 47s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8952/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8952/yetus/whitespace-eol.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8952/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> MSCK REPAIR TABLE Throw MetaException
> --
>
> Key: HIVE-18582
> URL: https://issues.apache.org/jira/browse/HIVE-18582
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.1
>Reporter: liubangchen
>Assignee: liubangchen
>Priority: Major
> Attachments: HIVE-18582.patch
>
>
> while executing query MSCK REPAIR TABLE tablename I got Exception:
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Expected 1 components, got 2
> (log_date=2015121309/vgameid=lyjt))
> at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1847)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:402)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> --
> Caused by: MetaException(message:Expected 1 components, got 2
> (log_date=2015121309/vgameid=lyjt))
> at
> org.apache.hadoop.hive.metastore.Warehouse.makeValsFromName(Warehouse.java:385)
> at
[jira] [Commented] (HIVE-18478) Data files deleted from temp table should not be recycled to CM path
[
https://issues.apache.org/jira/browse/HIVE-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347497#comment-16347497
]
Hive QA commented on HIVE-18478:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908506/HIVE-18478.03.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 19 failed/errored test(s), 12965 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8951/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8951/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8951/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 19 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908506 - PreCommit-HIVE-Build
> Data files deleted from temp table should not be recycled to CM path
>
>
> Key: HIVE-18478
> URL: https://issues.apache.org/jira/browse/HIVE-18478
> Project: Hive
> Issue Type: Sub-task
> Components: Hive, HiveServer2
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18478.01.patch, HIVE-18478.02.patch,
> HIVE-18478.03.patch
>
>
> Drop TEMP table operation invokes deleteDir which moves the file to $CMROOT
> which is not needed as temp tables need not be replicated
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18536) IOW + DP is broken for insert-only ACID
[ https://issues.apache.org/jira/browse/HIVE-18536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347491#comment-16347491 ] Sergey Shelukhin commented on HIVE-18536: - Most errors are the existing ones; some new ones are stuff like " ExitCodeException exitCode=1: chmod: cannot access ‘/tmp/hadoop/mapred/staging/hiveptest1178156071/.staging/job_local1178156071_0036/job.jar’: No such file or directory" [~ekoifman] ping? > IOW + DP is broken for insert-only ACID > --- > > Key: HIVE-18536 > URL: https://issues.apache.org/jira/browse/HIVE-18536 > Project: Hive > Issue Type: Sub-task >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18536.01.patch, HIVE-18536.02.patch, > HIVE-18536.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18582) MSCK REPAIR TABLE Throw MetaException
[
https://issues.apache.org/jira/browse/HIVE-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347487#comment-16347487
]
Sergey Shelukhin commented on HIVE-18582:
-
This code should still respect the mode as specified - i.e. in throw case, it
should throw on bad directories, not skip them.
Partition directories actually do not have to be under a table directory, so
the code that does the sub-directory may be incorrect, unless the caller
ensures they are.
Can you post on review board? There are some minor comments (typo in Valid,
etc) that are easier to leave there.
> MSCK REPAIR TABLE Throw MetaException
> --
>
> Key: HIVE-18582
> URL: https://issues.apache.org/jira/browse/HIVE-18582
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.1
>Reporter: liubangchen
>Assignee: liubangchen
>Priority: Major
> Attachments: HIVE-18582.patch
>
>
> while executing query MSCK REPAIR TABLE tablename I got Exception:
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Expected 1 components, got 2
> (log_date=2015121309/vgameid=lyjt))
> at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1847)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:402)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> --
> Caused by: MetaException(message:Expected 1 components, got 2
> (log_date=2015121309/vgameid=lyjt))
> at
> org.apache.hadoop.hive.metastore.Warehouse.makeValsFromName(Warehouse.java:385)
> at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1845)
> {code}
> table PARTITIONED by (log_date,vgameid)
> The data file on HDFS is:
>
> {code:java}
> /usr/hive/warehouse/a.db/tablename/log_date=2015063023
> drwxr-xr-x - root supergroup 0 2018-01-26 09:41
> /usr/hive/warehouse/a.db/tablename/log_date=2015121309/vgameid=lyjt
> {code}
> The subdir of log_data=2015063023 is empty
> If i set hive.msck.path.validation=ignore Then msck repair table will
> executed ok.
> Then I found code like this:
> {code:java}
> private int msck(Hive db, MsckDesc msckDesc) {
> CheckResult result = new CheckResult();
> List repairOutput = new ArrayList();
> try {
> HiveMetaStoreChecker checker = new HiveMetaStoreChecker(db);
> String[] names = Utilities.getDbTableName(msckDesc.getTableName());
> checker.checkMetastore(names[0], names[1], msckDesc.getPartSpecs(),
> result);
> List partsNotInMs =
> result.getPartitionsNotInMs();
> if (msckDesc.isRepairPartitions() && !partsNotInMs.isEmpty()) {
> //I think bug is here
> AbstractList vals = null;
> String settingStr = HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVE_MSCK_PATH_VALIDATION);
> boolean doValidate = !("ignore".equals(settingStr));
> boolean doSkip = doValidate && "skip".equals(settingStr);
> // The default setting is "throw"; assume doValidate && !doSkip means
> throw.
> if (doValidate) {
> // Validate that we can add partition without escaping. Escaping was
> originally intended
> // to avoid creating invalid HDFS paths; however, if we escape the
> HDFS path (that we
> // deem invalid but HDFS actually supports - it is possible to create
> HDFS paths with
> // unprintable characters like ASCII 7), metastore will create
> another directory instead
> // of the one we are trying to "repair" here.
> Iterator iter = partsNotInMs.iterator();
> while (iter.hasNext()) {
> CheckResult.PartitionResult part = iter.next();
> try {
> vals = Warehouse.makeValsFromName(part.getPartitionName(), vals);
> } catch (MetaException ex) {
> throw new HiveException(ex);
> }
> for (String val : vals) {
> String escapedPath = FileUtils.escapePathName(val);
> assert escapedPath != null;
> if (escapedPath.equals(val)) continue;
> String errorMsg = "Repair: Cannot add partition " +
> msckDesc.getTableName()
> + ':' + part.getPartitionName() + " due to invalid characters
> in the name";
> if (doSkip) {
> repairOutput.add(errorMsg);
> iter.remove();
> } else {
> throw
[jira] [Commented] (HIVE-18587) insert DML event may attempt to calculate a checksum on directories
[ https://issues.apache.org/jira/browse/HIVE-18587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347474#comment-16347474 ] Sergey Shelukhin commented on HIVE-18587: - Test failures are existing unstable tests... [~ashutoshc] can you take a look? > insert DML event may attempt to calculate a checksum on directories > --- > > Key: HIVE-18587 > URL: https://issues.apache.org/jira/browse/HIVE-18587 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-18587.patch > > > Looks like in union case, some code path may pass directories in newFiles. > Probably legacy copyData/moveData; both seem to assume that these paths are > files, but do not actually enforce it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-15631) Optimize for hive client logs , you can filter the log for each session itself.
[ https://issues.apache.org/jira/browse/HIVE-15631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347462#comment-16347462 ] Prasanth Jayachandran commented on HIVE-15631: -- LGTM +1, pending tests. > Optimize for hive client logs , you can filter the log for each session > itself. > --- > > Key: HIVE-15631 > URL: https://issues.apache.org/jira/browse/HIVE-15631 > Project: Hive > Issue Type: Improvement > Components: CLI, Clients, Hive >Reporter: tartarus >Assignee: tartarus >Priority: Major > Labels: pull-request-available > Attachments: HIVE-15631.patch > > Original Estimate: 24h > Remaining Estimate: 24h > > We have several hadoop cluster, about 15 thousand nodes. Every day we use > hive to submit above 100 thousand jobs. > So we have a large file of hive logs on every client host every day, but i > don not know the logs of my session submitted was which line. > So i hope to print the hive.session.id on every line of logs, and then i > could use grep to find the logs of my session submitted. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-18582) MSCK REPAIR TABLE Throw MetaException
[
https://issues.apache.org/jira/browse/HIVE-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-18582:
---
Assignee: liubangchen
> MSCK REPAIR TABLE Throw MetaException
> --
>
> Key: HIVE-18582
> URL: https://issues.apache.org/jira/browse/HIVE-18582
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Affects Versions: 2.1.1
>Reporter: liubangchen
>Assignee: liubangchen
>Priority: Major
> Attachments: HIVE-18582.patch
>
>
> while executing query MSCK REPAIR TABLE tablename I got Exception:
> {code:java}
> org.apache.hadoop.hive.ql.metadata.HiveException:
> MetaException(message:Expected 1 components, got 2
> (log_date=2015121309/vgameid=lyjt))
> at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1847)
> at org.apache.hadoop.hive.ql.exec.DDLTask.execute(DDLTask.java:402)
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:197)
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:100)
> at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:2073)
> at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1744)
> at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1453)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1171)
> at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1161)
> at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:232)
> --
> Caused by: MetaException(message:Expected 1 components, got 2
> (log_date=2015121309/vgameid=lyjt))
> at
> org.apache.hadoop.hive.metastore.Warehouse.makeValsFromName(Warehouse.java:385)
> at org.apache.hadoop.hive.ql.exec.DDLTask.msck(DDLTask.java:1845)
> {code}
> table PARTITIONED by (log_date,vgameid)
> The data file on HDFS is:
>
> {code:java}
> /usr/hive/warehouse/a.db/tablename/log_date=2015063023
> drwxr-xr-x - root supergroup 0 2018-01-26 09:41
> /usr/hive/warehouse/a.db/tablename/log_date=2015121309/vgameid=lyjt
> {code}
> The subdir of log_data=2015063023 is empty
> If i set hive.msck.path.validation=ignore Then msck repair table will
> executed ok.
> Then I found code like this:
> {code:java}
> private int msck(Hive db, MsckDesc msckDesc) {
> CheckResult result = new CheckResult();
> List repairOutput = new ArrayList();
> try {
> HiveMetaStoreChecker checker = new HiveMetaStoreChecker(db);
> String[] names = Utilities.getDbTableName(msckDesc.getTableName());
> checker.checkMetastore(names[0], names[1], msckDesc.getPartSpecs(),
> result);
> List partsNotInMs =
> result.getPartitionsNotInMs();
> if (msckDesc.isRepairPartitions() && !partsNotInMs.isEmpty()) {
> //I think bug is here
> AbstractList vals = null;
> String settingStr = HiveConf.getVar(conf,
> HiveConf.ConfVars.HIVE_MSCK_PATH_VALIDATION);
> boolean doValidate = !("ignore".equals(settingStr));
> boolean doSkip = doValidate && "skip".equals(settingStr);
> // The default setting is "throw"; assume doValidate && !doSkip means
> throw.
> if (doValidate) {
> // Validate that we can add partition without escaping. Escaping was
> originally intended
> // to avoid creating invalid HDFS paths; however, if we escape the
> HDFS path (that we
> // deem invalid but HDFS actually supports - it is possible to create
> HDFS paths with
> // unprintable characters like ASCII 7), metastore will create
> another directory instead
> // of the one we are trying to "repair" here.
> Iterator iter = partsNotInMs.iterator();
> while (iter.hasNext()) {
> CheckResult.PartitionResult part = iter.next();
> try {
> vals = Warehouse.makeValsFromName(part.getPartitionName(), vals);
> } catch (MetaException ex) {
> throw new HiveException(ex);
> }
> for (String val : vals) {
> String escapedPath = FileUtils.escapePathName(val);
> assert escapedPath != null;
> if (escapedPath.equals(val)) continue;
> String errorMsg = "Repair: Cannot add partition " +
> msckDesc.getTableName()
> + ':' + part.getPartitionName() + " due to invalid characters
> in the name";
> if (doSkip) {
> repairOutput.add(errorMsg);
> iter.remove();
> } else {
> throw new HiveException(errorMsg);
> }
> }
> }
> }
> {code}
> I think AbstractList vals = null; must placed after "while
> (iter.hasNext()) {" will work ok.
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[ https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18516: -- Attachment: HIVE-18516.7.patch > load data should rename files consistent with insert statements for ACID > Tables > --- > > Key: HIVE-18516 > URL: https://issues.apache.org/jira/browse/HIVE-18516 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18516.1.patch, HIVE-18516.2.patch, > HIVE-18516.3.patch, HIVE-18516.4.patch, HIVE-18516.5.patch, > HIVE-18516.6.patch, HIVE-18516.7.patch > > > h1. load data should rename files consistent with insert statements for ACID > Tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17396) Support DPP with map joins where the source and target belong in the same stage
[ https://issues.apache.org/jira/browse/HIVE-17396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-17396: --- Attachment: HIVE-17396.9.patch > Support DPP with map joins where the source and target belong in the same > stage > --- > > Key: HIVE-17396 > URL: https://issues.apache.org/jira/browse/HIVE-17396 > Project: Hive > Issue Type: Sub-task > Components: Spark >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-17396.1.patch, HIVE-17396.2.patch, > HIVE-17396.3.patch, HIVE-17396.4.patch, HIVE-17396.5.patch, > HIVE-17396.6.patch, HIVE-17396.7.patch, HIVE-17396.8.patch, HIVE-17396.9.patch > > > When the target of a partition pruning sink operator is in not the same as > the target of hash table sink operator, both source and target gets scheduled > within the same spark job, and that can result in File Not Found Exception. > HIVE-17225 has a fix to disable DPP in that scenario. This JIRA is to > support DPP for such cases. > Test Case: > SET hive.spark.dynamic.partition.pruning=true; > SET hive.auto.convert.join=true; > SET hive.strict.checks.cartesian.product=false; > CREATE TABLE part_table1 (col int) PARTITIONED BY (part1_col int); > CREATE TABLE part_table2 (col int) PARTITIONED BY (part2_col int); > CREATE TABLE reg_table (col int); > ALTER TABLE part_table1 ADD PARTITION (part1_col = 1); > ALTER TABLE part_table2 ADD PARTITION (part2_col = 1); > ALTER TABLE part_table2 ADD PARTITION (part2_col = 2); > INSERT INTO TABLE part_table1 PARTITION (part1_col = 1) VALUES (1); > INSERT INTO TABLE part_table2 PARTITION (part2_col = 1) VALUES (1); > INSERT INTO TABLE part_table2 PARTITION (part2_col = 2) VALUES (2); > INSERT INTO table reg_table VALUES (1), (2), (3), (4), (5), (6); > EXPLAIN SELECT * > FROM part_table1 pt1, >part_table2 pt2, >reg_table rt > WHERE rt.col = pt1.part1_col > ANDpt2.part2_col = pt1.part1_col; > Plan: > STAGE DEPENDENCIES: > Stage-2 is a root stage > Stage-1 depends on stages: Stage-2 > Stage-0 depends on stages: Stage-1 > STAGE PLANS: > Stage: Stage-2 > Spark > A masked pattern was here > Vertices: > Map 1 > Map Operator Tree: > TableScan > alias: pt1 > Statistics: Num rows: 1 Data size: 1 Basic stats: COMPLETE > Column stats: NONE > Select Operator > expressions: col (type: int), part1_col (type: int) > outputColumnNames: _col0, _col1 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col1 (type: int) > 1 _col1 (type: int) > 2 _col0 (type: int) > Select Operator > expressions: _col1 (type: int) > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Group By Operator > keys: _col0 (type: int) > mode: hash > outputColumnNames: _col0 > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > Spark Partition Pruning Sink Operator > Target column: part2_col (int) > partition key expr: part2_col > Statistics: Num rows: 1 Data size: 1 Basic stats: > COMPLETE Column stats: NONE > target work: Map 2 > Local Work: > Map Reduce Local Work > Map 2 > Map Operator Tree: > TableScan > alias: pt2 > Statistics: Num rows: 2 Data size: 2 Basic stats: COMPLETE > Column stats: NONE > Select Operator > expressions: col (type: int), part2_col (type: int) > outputColumnNames: _col0, _col1 > Statistics: Num rows: 2 Data size: 2 Basic stats: > COMPLETE Column stats: NONE > Spark HashTable Sink Operator > keys: > 0 _col1 (type: int) > 1 _col1 (type: int) > 2 _col0 (type: int) > Local Work: > Map Reduce Local Work > Stage: Stage-1 > Spark > A masked pattern was here > Vertices: > Map 3 > Map Operator Tree: > TableScan >
[jira] [Commented] (HIVE-18478) Data files deleted from temp table should not be recycled to CM path
[
https://issues.apache.org/jira/browse/HIVE-18478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347399#comment-16347399
]
Hive QA commented on HIVE-18478:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
34s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
21s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 1 new + 251 unchanged - 0
fixed = 252 total (was 251) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 21m 11s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8951/yetus/diff-checkstyle-ql.txt
|
| modules | C: standalone-metastore ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8951/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Data files deleted from temp table should not be recycled to CM path
>
>
> Key: HIVE-18478
> URL: https://issues.apache.org/jira/browse/HIVE-18478
> Project: Hive
> Issue Type: Sub-task
> Components: Hive, HiveServer2
>Reporter: mahesh kumar behera
>Assignee: mahesh kumar behera
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18478.01.patch, HIVE-18478.02.patch,
> HIVE-18478.03.patch
>
>
> Drop TEMP table operation invokes deleteDir which moves the file to $CMROOT
> which is not needed as temp tables need not be replicated
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18513) Query results caching
[
https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347346#comment-16347346
]
Hive QA commented on HIVE-18513:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908504/HIVE-18513.4.patch
{color:green}SUCCESS:{color} +1 due to 7 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 23 failed/errored test(s), 12971 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=79)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.hcatalog.listener.TestDbNotificationListener.dropDatabase
(batchId=242)
org.apache.hive.hcatalog.pig.TestSequenceFileHCatStorer.testWriteSmallint
(batchId=193)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8950/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8950/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8950/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 23 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908504 - PreCommit-HIVE-Build
> Query results caching
> -
>
> Key: HIVE-18513
> URL: https://issues.apache.org/jira/browse/HIVE-18513
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18513.1.patch, HIVE-18513.2.patch,
> HIVE-18513.3.patch, HIVE-18513.4.patch
>
>
> Add a query results cache that can save the results of an executed Hive query
> for reuse on subsequent queries. This may be useful in cases where the same
> query is issued many times, since Hive can return back the results of a
> cached query rather than having to execute the full query on the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347327#comment-16347327
]
Sahil Takiar commented on HIVE-18552:
-
[~aihuaxu], [~vihangk1] could you take a look?
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18552:
Status: Open (was: Patch Available)
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18552:
Status: Patch Available (was: Open)
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18552) Split hive.strict.checks.large.query into two configs
[
https://issues.apache.org/jira/browse/HIVE-18552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-18552:
Attachment: HIVE-18552.4.patch
> Split hive.strict.checks.large.query into two configs
> -
>
> Key: HIVE-18552
> URL: https://issues.apache.org/jira/browse/HIVE-18552
> Project: Hive
> Issue Type: Improvement
> Components: Hive
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18552.1.patch, HIVE-18552.2.patch,
> HIVE-18552.3.patch, HIVE-18552.4.patch
>
>
> {{hive.strict.checks.large.query}} controls the strict checks for restricting
> order bys with no limits, and scans of a partitioned table without a filter
> on the partition table.
> While both checks prevent "large" queries from being run, they both control
> very different behavior. It would be better if users could control these
> restrictions separately.
> Furthermore, many users make the mistake of abusing partitioned tables and
> often end up in a situation where they are running queries that are doing
> full-table scans of partitioned tables. This can lead to lots of issues for
> Hive - e.g. OOM issues because so many partitions are loaded in memory. So it
> would be good if we enabled this restriction by default.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18513) Query results caching
[
https://issues.apache.org/jira/browse/HIVE-18513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347304#comment-16347304
]
Hive QA commented on HIVE-18513:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
20s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
48s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
9s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
47s{color} | {color:red} ql: The patch generated 19 new + 1328 unchanged - 2
fixed = 1347 total (was 1330) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
55s{color} | {color:red} root: The patch generated 19 new + 1881 unchanged - 2
fixed = 1900 total (was 1883) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
9s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
55s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8950/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8950/yetus/diff-checkstyle-root.txt
|
| modules | C: common ql service . itests itests/hive-blobstore itests/util U:
. |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8950/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Query results caching
> -
>
> Key: HIVE-18513
> URL: https://issues.apache.org/jira/browse/HIVE-18513
> Project: Hive
> Issue Type: Bug
> Components: Query Planning
>Reporter: Jason Dere
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-18513.1.patch, HIVE-18513.2.patch,
> HIVE-18513.3.patch, HIVE-18513.4.patch
>
>
> Add a query results cache that can save the results of an executed Hive query
> for reuse on subsequent queries. This may be useful in cases where the same
> query is issued many times, since Hive can return back the results of a
> cached query rather than having to execute the full query on the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18192) Introduce WriteID per table rather than using global transaction ID
[ https://issues.apache.org/jira/browse/HIVE-18192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347301#comment-16347301 ] Sankar Hariappan commented on HIVE-18192: - Based on discussion with ACID folks, the proposal for pending changes. Must complete before merging the branch to master: # Remove entries from TXNS_TO_WRITE_ID table when drop a table/database. Also, split db and table names into 2 columns. # Scripts to add Metastore tables for write id management. Add for other non-derby databases too. # CompactionTxnHandler cleanup COMPLETED_TXN_COMPONENTS based on highest write id instead of highest txn id. # Non-Acid to Acid conversion through alter table transaction property. Can be done after the merge to master: # New logic for cleaner to remove TXNS_TO_WRITE_ID table entries. Also, need to maintain the LWM for each table write id. # Update classes to use WriteID instead of TxnId as methods and variable names. cc [~thejas], [~ekoifman] > Introduce WriteID per table rather than using global transaction ID > --- > > Key: HIVE-18192 > URL: https://issues.apache.org/jira/browse/HIVE-18192 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Major > Labels: ACID, DR, pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18192.01.patch, HIVE-18192.02.patch, > HIVE-18192.03.patch, HIVE-18192.04.patch, HIVE-18192.05.patch, > HIVE-18192.06.patch > > > To support ACID replication, we will be introducing a per table write Id > which will replace the transaction id in the primary key for each row in a > ACID table. > The current primary key is determined via > > which will move to > > For each table modified by the given transaction will have a table level > write ID allocated and a persisted map of global txn id -> to table -> write > id for that table has to be maintained to allow Snapshot isolation. > Readers should use the combination of ValidTxnList and > ValidWriteIdList(Table) for snapshot isolation. > > [Hive Replication - ACID > Tables.pdf|https://issues.apache.org/jira/secure/attachment/12903157/Hive%20Replication-%20ACID%20Tables.pdf] > has a section "Per Table Sequences (Write-Id)" with more detials -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18586) Upgrade Derby to 10.14.1.0
[ https://issues.apache.org/jira/browse/HIVE-18586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Janaki Lahorani updated HIVE-18586: --- Attachment: HIVE-18586.2.patch > Upgrade Derby to 10.14.1.0 > -- > > Key: HIVE-18586 > URL: https://issues.apache.org/jira/browse/HIVE-18586 > Project: Hive > Issue Type: Improvement >Reporter: Janaki Lahorani >Assignee: Janaki Lahorani >Priority: Major > Attachments: HIVE-18586.1.patch, HIVE-18586.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18301) Investigate to enable MapInput cache in Hive on Spark
[
https://issues.apache.org/jira/browse/HIVE-18301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347253#comment-16347253
]
Xuefu Zhang commented on HIVE-18301:
It seems that IOContext is used in many places and the logics complicated.
Instead of putting the input patch in each row, like what the patch is
proposing, could we send a serialized IOContext object as a special row
whenever the content of the object changes? I'm not sure how feasible it's, so
it's just a rough idea to be explored.
> Investigate to enable MapInput cache in Hive on Spark
> -
>
> Key: HIVE-18301
> URL: https://issues.apache.org/jira/browse/HIVE-18301
> Project: Hive
> Issue Type: Bug
>Reporter: liyunzhang
>Assignee: liyunzhang
>Priority: Major
> Attachments: HIVE-18301.1.patch, HIVE-18301.patch
>
>
> Before IOContext problem is found in MapTran when spark rdd cache is enabled
> in HIVE-8920.
> so we disabled rdd cache in MapTran at
> [SparkPlanGenerator|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java#L202].
> The problem is IOContext seems not initialized correctly in the spark yarn
> client/cluster mode and caused the exception like
> {code}
> Job aborted due to stage failure: Task 93 in stage 0.0 failed 4 times, most
> recent failure: Lost task 93.3 in stage 0.0 (TID 616, bdpe48):
> java.lang.RuntimeException: Error processing row:
> java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:165)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:48)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveMapFunctionResultList.processNextRecord(HiveMapFunctionResultList.java:27)
> at
> org.apache.hadoop.hive.ql.exec.spark.HiveBaseFunctionResultList.hasNext(HiveBaseFunctionResultList.java:85)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)
> at
> org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:125)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:79)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:47)
> at org.apache.spark.scheduler.Task.run(Task.scala:85)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.hive.ql.exec.AbstractMapOperator.getNominalPath(AbstractMapOperator.java:101)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.cleanUpInputFileChangedOp(MapOperator.java:516)
> at
> org.apache.hadoop.hive.ql.exec.Operator.cleanUpInputFileChanged(Operator.java:1187)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkMapRecordHandler.processRow(SparkMapRecordHandler.java:152)
> ... 12 more
> Driver stacktrace:
> {code}
> in yarn client/cluster mode, sometimes
> [ExecMapperContext#currentInputPath|https://github.com/kellyzly/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/exec/mr/ExecMapperContext.java#L109]
> is null when rdd cach is enabled.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18442) HoS: No FileSystem for scheme: nullscan
[
https://issues.apache.org/jira/browse/HIVE-18442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347238#comment-16347238
]
Xuefu Zhang commented on HIVE-18442:
Hi [~lirui], Thanks for the explanation. The patch looks fine. I'm wondering if
there could be other similar issues. Thus, would either of the two options you
mentioned works better?
{quote}
Unless it's added to the driver's extra class path or we enable
{{spark.yarn.user.classpath.first}}.
{quote}
> HoS: No FileSystem for scheme: nullscan
> ---
>
> Key: HIVE-18442
> URL: https://issues.apache.org/jira/browse/HIVE-18442
> Project: Hive
> Issue Type: Bug
> Components: Spark
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Major
> Attachments: HIVE-18442.1.patch
>
>
> Hit the issue when I run following query in yarn-cluster mode:
> {code}
> select * from (select key from src where false) a left outer join (select key
> from srcpart limit 0) b on a.key=b.key;
> {code}
> Stack trace:
> {noformat}
> Job failed with java.io.IOException: No FileSystem for scheme: nullscan
> at
> org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2799)
> at
> org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2810)
> at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:100)
> at
> org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2849)
> at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2831)
> at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:389)
> at org.apache.hadoop.fs.Path.getFileSystem(Path.java:356)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2605)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.isEmptyPath(Utilities.java:2601)
> at
> org.apache.hadoop.hive.ql.exec.Utilities$GetInputPathsCallable.call(Utilities.java:3409)
> at
> org.apache.hadoop.hive.ql.exec.Utilities.getInputPaths(Utilities.java:3347)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.cloneJobConf(SparkPlanGenerator.java:299)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:222)
> at
> org.apache.hadoop.hive.ql.exec.spark.SparkPlanGenerator.generate(SparkPlanGenerator.java:109)
> at
> org.apache.hadoop.hive.ql.exec.spark.RemoteHiveSparkClient$JobStatusJob.call(RemoteHiveSparkClient.java:354)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:358)
> at
> org.apache.hive.spark.client.RemoteDriver$JobWrapper.call(RemoteDriver.java:323)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HIVE-18593) NPE on vectorization group by
[
https://issues.apache.org/jira/browse/HIVE-18593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347231#comment-16347231
]
Amruth S edited comment on HIVE-18593 at 1/31/18 5:29 PM:
--
Sure. Will share a minified version of query by tomorrow.
BTW [~gopalv] I did backport (https://issues.apache.org/jira/browse/HIVE-18258)
and checked. The issue still comes.
was (Author: amrk7):
Sure. Will share a minified version of query by tomorrow.
> NPE on vectorization group by
> -
>
> Key: HIVE-18593
> URL: https://issues.apache.org/jira/browse/HIVE-18593
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2
>Affects Versions: 2.3.2
>Reporter: Amruth S
>Priority: Major
>
> Vectorisation with some queries seem to be failing with null pointer
> exceptions. This happens only with 2.3.2 release and not the older ones.
> In the case, (in BytesColumnVector.java, vector[0] is null, isRepeating is
> true, length[0] is 0, start[0] is 0
> {code:java}
> public void copySelected(
> boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
> // Output has nulls if and only if input has nulls.
> output.noNulls = noNulls;
> output.isRepeating = false;
> // Handle repeating case
> if (isRepeating) {
> output.setVal(0, vector[0], start[0], length[0]);
> output.isNull[0] = isNull[0];
> output.isRepeating = true;
> return;
> }
> {code}
> Exception trace below
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1021)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:137)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:123)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:783)
> ... 18 more
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18593) NPE on vectorization group by
[
https://issues.apache.org/jira/browse/HIVE-18593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347231#comment-16347231
]
Amruth S commented on HIVE-18593:
-
Sure. Will share a minified version of query by tomorrow.
> NPE on vectorization group by
> -
>
> Key: HIVE-18593
> URL: https://issues.apache.org/jira/browse/HIVE-18593
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2
>Affects Versions: 2.3.2
>Reporter: Amruth S
>Priority: Major
>
> Vectorisation with some queries seem to be failing with null pointer
> exceptions. This happens only with 2.3.2 release and not the older ones.
> In the case, (in BytesColumnVector.java, vector[0] is null, isRepeating is
> true, length[0] is 0, start[0] is 0
> {code:java}
> public void copySelected(
> boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
> // Output has nulls if and only if input has nulls.
> output.noNulls = noNulls;
> output.isRepeating = false;
> // Handle repeating case
> if (isRepeating) {
> output.setVal(0, vector[0], start[0], length[0]);
> output.isNull[0] = isNull[0];
> output.isRepeating = true;
> return;
> }
> {code}
> Exception trace below
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1021)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:137)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:123)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:783)
> ... 18 more
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18590) Assertion error on transitive join inference in the presence of NOT NULL constraint
[
https://issues.apache.org/jira/browse/HIVE-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347212#comment-16347212
]
Hive QA commented on HIVE-18590:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908500/HIVE-18590.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 12941 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=85)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=180)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestDropPartitions.testDropPartition[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.hcatalog.pig.TestSequenceFileHCatStorer.testWriteDate2
(batchId=193)
org.apache.hive.hcatalog.pig.TestSequenceFileHCatStorer.testWriteDate3
(batchId=193)
org.apache.hive.hcatalog.pig.TestSequenceFileHCatStorer.testWriteSmallint
(batchId=193)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8949/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8949/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8949/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908500 - PreCommit-HIVE-Build
> Assertion error on transitive join inference in the presence of NOT NULL
> constraint
> ---
>
> Key: HIVE-18590
> URL: https://issues.apache.org/jira/browse/HIVE-18590
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18590.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18593) NPE on vectorization group by
[
https://issues.apache.org/jira/browse/HIVE-18593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347203#comment-16347203
]
Vihang Karajgaonkar commented on HIVE-18593:
Thanks for reporting the issue [~amrk7]. do you have a query which can be used
to reproduce this issue?
> NPE on vectorization group by
> -
>
> Key: HIVE-18593
> URL: https://issues.apache.org/jira/browse/HIVE-18593
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2
>Affects Versions: 2.3.2
>Reporter: Amruth S
>Priority: Major
>
> Vectorisation with some queries seem to be failing with null pointer
> exceptions. This happens only with 2.3.2 release and not the older ones.
> In the case, (in BytesColumnVector.java, vector[0] is null, isRepeating is
> true, length[0] is 0, start[0] is 0
> {code:java}
> public void copySelected(
> boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
> // Output has nulls if and only if input has nulls.
> output.noNulls = noNulls;
> output.isRepeating = false;
> // Handle repeating case
> if (isRepeating) {
> output.setVal(0, vector[0], start[0], length[0]);
> output.isNull[0] = isNull[0];
> output.isRepeating = true;
> return;
> }
> {code}
> Exception trace below
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1021)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:137)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:123)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:783)
> ... 18 more
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18590) Assertion error on transitive join inference in the presence of NOT NULL constraint
[
https://issues.apache.org/jira/browse/HIVE-18590?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347087#comment-16347087
]
Hive QA commented on HIVE-18590:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
54s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 28s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8949/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Assertion error on transitive join inference in the presence of NOT NULL
> constraint
> ---
>
> Key: HIVE-18590
> URL: https://issues.apache.org/jira/browse/HIVE-18590
> Project: Hive
> Issue Type: Bug
> Components: Logical Optimizer
>Affects Versions: 3.0.0
>Reporter: Jesus Camacho Rodriguez
>Assignee: Jesus Camacho Rodriguez
>Priority: Major
> Attachments: HIVE-18590.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[
https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347056#comment-16347056
]
Hive QA commented on HIVE-18516:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908497/HIVE-18516.6.patch
{color:green}SUCCESS:{color} +1 due to 4 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 24 failed/errored test(s), 12940 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.org.apache.hadoop.hive.cli.TestCliDriver
(batchId=86)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestFunctions.testGetFunctionNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnCommandsForMmTable.testInsertOverwriteForMmTable
(batchId=259)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversion
(batchId=259)
org.apache.hadoop.hive.ql.TestTxnLoadData.loadDataNonAcid2AcidConversionVectorized
(batchId=259)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8948/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8948/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8948/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 24 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908497 - PreCommit-HIVE-Build
> load data should rename files consistent with insert statements for ACID
> Tables
> ---
>
> Key: HIVE-18516
> URL: https://issues.apache.org/jira/browse/HIVE-18516
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18516.1.patch, HIVE-18516.2.patch,
> HIVE-18516.3.patch, HIVE-18516.4.patch, HIVE-18516.5.patch, HIVE-18516.6.patch
>
>
> h1. load data should rename files consistent with insert statements for ACID
> Tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18592) DP insert on insert only table causes StatTask to fail
[
https://issues.apache.org/jira/browse/HIVE-18592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eugene Koifman updated HIVE-18592:
--
Component/s: Transactions
> DP insert on insert only table causes StatTask to fail
> --
>
> Key: HIVE-18592
> URL: https://issues.apache.org/jira/browse/HIVE-18592
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Zoltan Haindrich
>Priority: Major
>
> can be reproduced with:
> {code}
> set hive.mapred.mode=nonstrict;
> set
> hive.security.authorization.manager=org.apache.hadoop.hive.ql.security.authorization.plugin.sqlstd.SQLStdHiveAuthorizerFactoryForTest;
> set hive.support.concurrency=true;
> set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;
> set hive.create.as.insert.only=true;
> set metastore.create.as.acid=true;
> drop table if exists student;
> create table student(
> name string,
> age int,
> gpa double);
> insert into student values
> ('asd',1,2),
> ('asdx',2,3),
> ('asdx',2,3),
> ('asdx',3,3),
> ('asdx',3,3),
> ('asdx',3,3);
> create table p1 (name STRING, GPA DOUBLE) PARTITIONED BY (age INT);
> SET hive.exec.dynamic.partition.mode=nonstrict;
> INSERT OVERWRITE TABLE p1 PARTITION (age) SELECT name, gpa, age FROM student;
> {code}
> causes exception
> {code}
> 2018-01-31T02:16:24,135 ERROR [22bd4065-6e2f-4f4c-8f29-8d6aad8edda8 main]
> exec.StatsTask: Failed to run stats task
> org.apache.hadoop.hive.ql.metadata.HiveException:
> NoSuchObjectException(message:Partition for which stats is gathered doesn't
> exist.)
> at
> org.apache.hadoop.hive.ql.metadata.Hive.setPartitionColumnStatistics(Hive.java:4295)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.persistColumnStats(ColStatsProcessor.java:180)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.stats.ColStatsProcessor.process(ColStatsProcessor.java:84)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.StatsTask.execute(StatsTask.java:108)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:205)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:97)
> [hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> ...
> Caused by: org.apache.hadoop.hive.metastore.api.NoSuchObjectException:
> Partition for which stats is gathered doesn't exist.
> at
> org.apache.hadoop.hive.metastore.ObjectStore.updatePartitionColumnStatistics(ObjectStore.java:7757)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_151]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_151]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_151]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
> at
> org.apache.hadoop.hive.metastore.RawStoreProxy.invoke(RawStoreProxy.java:97)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy38.updatePartitionColumnStatistics(Unknown
> Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.updatePartitonColStats(HiveMetaStore.java:5394)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.set_aggr_stats_for(HiveMetaStore.java:6907)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:1.8.0_151]
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_151]
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_151]
> at java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
> ~[hive-standalone-metastore-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at com.sun.proxy.$Proxy40.set_aggr_stats_for(Unknown Source) ~[?:?]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.setPartitionColumnStatistics(HiveMetaStoreClient.java:1736)
>
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16347040#comment-16347040 ] slim bouguerra commented on HIVE-18573: --- in patch 3 i have reverted some changes on the naming of extracts and floor. > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators. > Also, Use a single name for Extract operator instead of a different name for > every Unit, > Same for Floor function. This will allow unifying the treatment per operator. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18573: -- Attachment: HIVE-18573.3.patch > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.3.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators. > Also, Use a single name for Extract operator instead of a different name for > every Unit, > Same for Floor function. This will allow unifying the treatment per operator. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18593) NPE on vectorization group by
[
https://issues.apache.org/jira/browse/HIVE-18593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346979#comment-16346979
]
Gopal V commented on HIVE-18593:
This might be fixed by HIVE-18258 - the query is relevant (& explain
vectorization detail output).
> NPE on vectorization group by
> -
>
> Key: HIVE-18593
> URL: https://issues.apache.org/jira/browse/HIVE-18593
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2
>Affects Versions: 2.3.2
>Reporter: Amruth S
>Priority: Major
>
> Vectorisation with some queries seem to be failing with null pointer
> exceptions. This happens only with 2.3.2 release and not the older ones.
> In the case, (in BytesColumnVector.java, vector[0] is null, isRepeating is
> true, length[0] is 0, start[0] is 0
> {code:java}
> public void copySelected(
> boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
> // Output has nulls if and only if input has nulls.
> output.noNulls = noNulls;
> output.isRepeating = false;
> // Handle repeating case
> if (isRepeating) {
> output.setVal(0, vector[0], start[0], length[0]);
> output.isNull[0] = isNull[0];
> output.isRepeating = true;
> return;
> }
> {code}
> Exception trace below
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1021)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:137)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:123)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:783)
> ... 18 more
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18516) load data should rename files consistent with insert statements for ACID Tables
[
https://issues.apache.org/jira/browse/HIVE-18516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346978#comment-16346978
]
Hive QA commented on HIVE-18516:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
29s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
35s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
39s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 9 new + 338 unchanged - 14
fixed = 347 total (was 352) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 14m 8s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8948/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8948/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> load data should rename files consistent with insert statements for ACID
> Tables
> ---
>
> Key: HIVE-18516
> URL: https://issues.apache.org/jira/browse/HIVE-18516
> Project: Hive
> Issue Type: Bug
>Reporter: Deepak Jaiswal
>Assignee: Deepak Jaiswal
>Priority: Major
> Attachments: HIVE-18516.1.patch, HIVE-18516.2.patch,
> HIVE-18516.3.patch, HIVE-18516.4.patch, HIVE-18516.5.patch, HIVE-18516.6.patch
>
>
> h1. load data should rename files consistent with insert statements for ACID
> Tables.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346941#comment-16346941 ] slim bouguerra commented on HIVE-18573: --- in HIVE-18573.2 i have removed the Div_integer operator since it was causing NPEs > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators. > Also, Use a single name for Extract operator instead of a different name for > every Unit, > Same for Floor function. This will allow unifying the treatment per operator. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18573) Use proper Calcite operator instead of UDFs
[ https://issues.apache.org/jira/browse/HIVE-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] slim bouguerra updated HIVE-18573: -- Attachment: HIVE-18573.2.patch > Use proper Calcite operator instead of UDFs > --- > > Key: HIVE-18573 > URL: https://issues.apache.org/jira/browse/HIVE-18573 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: slim bouguerra >Priority: Major > Attachments: HIVE-18573.2.patch, HIVE-18573.patch > > > Currently, Hive is mostly using user-defined black box sql operators during > Query planning. It will be more beneficial to use proper calcite operators. > Also, Use a single name for Extract operator instead of a different name for > every Unit, > Same for Floor function. This will allow unifying the treatment per operator. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18589) java.io.IOException: Not enough history available
[
https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346937#comment-16346937
]
Hive QA commented on HIVE-18589:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908494/HIVE-18589.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 50 failed/errored test(s), 12964 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.TestTxnCommands2.testACIDwithSchemaEvolutionAndCompaction
(batchId=280)
org.apache.hadoop.hive.ql.TestTxnCommands2.testInsertOverwrite2 (batchId=280)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNoHistory (batchId=280)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion1
(batchId=280)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion2
(batchId=280)
org.apache.hadoop.hive.ql.TestTxnCommands2.testNonAcidToAcidConversion3
(batchId=280)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testACIDwithSchemaEvolutionAndCompaction
(batchId=290)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testInsertOverwrite2
(batchId=290)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNoHistory
(batchId=290)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion1
(batchId=290)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion2
(batchId=290)
org.apache.hadoop.hive.ql.TestTxnCommands2WithSplitUpdateAndVectorization.testNonAcidToAcidConversion3
(batchId=290)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestAcidUtils.testBestBase (batchId=269)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hadoop.hive.ql.txn.compactor.TestCompactor.testTableProperties
(batchId=226)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorNoBaseLotsOfDeltas
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorPartitionWithBase
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorPartitionWithBaseMissingBuckets
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableNoBase
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorTableWithBase
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorWithAborted
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker.majorWithOpenInMiddle
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorNoBaseLotsOfDeltas
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorPartitionWithBase
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorPartitionWithBaseMissingBuckets
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorTableNoBase
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorTableWithBase
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorWithAborted
(batchId=268)
org.apache.hadoop.hive.ql.txn.compactor.TestWorker2.majorWithOpenInMiddle
(batchId=268)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
[jira] [Commented] (HIVE-18593) NPE on vectorization group by
[
https://issues.apache.org/jira/browse/HIVE-18593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346927#comment-16346927
]
Amruth S commented on HIVE-18593:
-
I made the following change to BytesColumnVector#copySelected method.
I am not sure about the behaviour when vector[0] is null, repeating is true
with a non zero length and offset. If someone can point me the behaviour, I can
patch this.
{code:java}
index f914a2248d..7757910a32 100644
---
a/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java
+++
b/storage-api/src/java/org/apache/hadoop/hive/ql/exec/vector/BytesColumnVector.java
@@ -318,7 +318,17 @@ public void copySelected(
// Handle repeating case
if (isRepeating) {
- output.setVal(0, vector[0], start[0], length[0]);
+ final byte[] srcBuffer;
+ if (vector[0] == null) {
+ if (start[0] == 0 && length[0] == 0) {
+ srcBuffer = new byte[0];
+ } else {
+ throw new RuntimeException("isRepeating is true, vector[0] is null but
start[0] is " + start[0] + " and length[0] is " + length[0]);
+ }
+ } else {
+ srcBuffer = vector[0];
+ }
+ output.setVal(0, srcBuffer, start[0], length[0]);
output.isNull[0] = isNull[0];
output.isRepeating = true;
return;
{code}
> NPE on vectorization group by
> -
>
> Key: HIVE-18593
> URL: https://issues.apache.org/jira/browse/HIVE-18593
> Project: Hive
> Issue Type: Bug
> Components: Hive, HiveServer2
>Affects Versions: 2.3.2
>Reporter: Amruth S
>Priority: Major
>
> Vectorisation with some queries seem to be failing with null pointer
> exceptions. This happens only with 2.3.2 release and not the older ones.
> In the case, (in BytesColumnVector.java, vector[0] is null, isRepeating is
> true, length[0] is 0, start[0] is 0
> {code:java}
> public void copySelected(
> boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
> // Output has nulls if and only if input has nulls.
> output.noNulls = noNulls;
> output.isRepeating = false;
> // Handle repeating case
> if (isRepeating) {
> output.setVal(0, vector[0], start[0], length[0]);
> output.isNull[0] = isNull[0];
> output.isRepeating = true;
> return;
> }
> {code}
> Exception trace below
> {code:java}
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
> at
> org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
> ... 17 more
> Caused by: java.lang.NullPointerException
> at java.lang.System.arraycopy(Native Method)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
> at
> org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
> at
> org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1021)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:137)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:123)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
> at
> org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:783)
> ... 18 more
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18593) NPE on vectorization group by
[
https://issues.apache.org/jira/browse/HIVE-18593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amruth S updated HIVE-18593:
Description:
Vectorisation with some queries seem to be failing with null pointer
exceptions. This happens only with 2.3.2 release and not the older ones.
In the case, (in BytesColumnVector.java, vector[0] is null, isRepeating is
true, length[0] is 0, start[0] is 0
{code:java}
public void copySelected(
boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
// Output has nulls if and only if input has nulls.
output.noNulls = noNulls;
output.isRepeating = false;
// Handle repeating case
if (isRepeating) {
output.setVal(0, vector[0], start[0], length[0]);
output.isNull[0] = isNull[0];
output.isRepeating = true;
return;
}
{code}
Exception trace below
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
... 17 more
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
at
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1021)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:137)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at
org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:123)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:783)
... 18 more
{code}
was:
Vectorisation with some queries seem to be failing with null pointer
exceptions. This happens only with 2.3.2 release and not the older ones.
It seems in this method (BytesColumnVector.java, vector[0] is null, isRepeating
is true, length is 0)
{code:java}
public void copySelected(
boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
// Output has nulls if and only if input has nulls.
output.noNulls = noNulls;
output.isRepeating = false;
// Handle repeating case
if (isRepeating) {
output.setVal(0, vector[0], start[0], length[0]);
output.isNull[0] = isNull[0];
output.isRepeating = true;
return;
}
{code}
Exception trace below
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
... 17 more
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
at
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
at
[jira] [Updated] (HIVE-18593) NPE on vectorization group by
[
https://issues.apache.org/jira/browse/HIVE-18593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Amruth S updated HIVE-18593:
Description:
Vectorisation with some queries seem to be failing with null pointer
exceptions. This happens only with 2.3.2 release and not the older ones.
It seems in this method (BytesColumnVector.java, vector[0] is null, isRepeating
is true, length is 0)
{code:java}
public void copySelected(
boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
// Output has nulls if and only if input has nulls.
output.noNulls = noNulls;
output.isRepeating = false;
// Handle repeating case
if (isRepeating) {
output.setVal(0, vector[0], start[0], length[0]);
output.isNull[0] = isNull[0];
output.isRepeating = true;
return;
}
{code}
Exception trace below
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
... 17 more
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
at
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator.process(VectorGroupByOperator.java:1021)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at
org.apache.hadoop.hive.ql.exec.vector.VectorSelectOperator.process(VectorSelectOperator.java:137)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at
org.apache.hadoop.hive.ql.exec.vector.VectorFilterOperator.process(VectorFilterOperator.java:123)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:130)
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:783)
... 18 more
{code}
was:
Vectorisation with some queries seem to be failing with null pointer
exceptions. This happens only with 2.3.2 release and not the older ones.
It seems in this method (BytesColumnVector.java, vector[0] is null, if the
complete batch is null)
{code:java}
public void copySelected(
boolean selectedInUse, int[] sel, int size, BytesColumnVector output) {
// Output has nulls if and only if input has nulls.
output.noNulls = noNulls;
output.isRepeating = false;
// Handle repeating case
if (isRepeating) {
output.setVal(0, vector[0], start[0], length[0]);
output.isNull[0] = isNull[0];
output.isRepeating = true;
return;
}
{code}
Exception trace below
{code:java}
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row
at
org.apache.hadoop.hive.ql.exec.vector.VectorMapOperator.process(VectorMapOperator.java:883)
at
org.apache.hadoop.hive.ql.exec.tez.MapRecordSource.processRow(MapRecordSource.java:86)
... 17 more
Caused by: java.lang.NullPointerException
at java.lang.System.arraycopy(Native Method)
at
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.setVal(BytesColumnVector.java:173)
at
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.copySelected(BytesColumnVector.java:321)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.IfExprStringGroupColumnStringGroupColumn.evaluate(IfExprStringGroupColumnStringGroupColumn.java:85)
at
org.apache.hadoop.hive.ql.exec.vector.expressions.aggregates.gen.VectorUDAFMaxString.aggregateInputSelection(VectorUDAFMaxString.java:135)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processAggregators(VectorGroupByOperator.java:218)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeHashAggregate.doProcessBatch(VectorGroupByOperator.java:408)
at
org.apache.hadoop.hive.ql.exec.vector.VectorGroupByOperator$ProcessingModeBase.processBatch(VectorGroupByOperator.java:179)
at
[jira] [Commented] (HIVE-18589) java.io.IOException: Not enough history available
[
https://issues.apache.org/jira/browse/HIVE-18589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346855#comment-16346855
]
Hive QA commented on HIVE-18589:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
59s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
52s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
2s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
37s{color} | {color:red} ql: The patch generated 1 new + 138 unchanged - 3
fixed = 139 total (was 141) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 13m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8947/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql U: ql |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8947/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> java.io.IOException: Not enough history available
> -
>
> Key: HIVE-18589
> URL: https://issues.apache.org/jira/browse/HIVE-18589
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Critical
> Attachments: HIVE-18589.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[
https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346820#comment-16346820
]
Hive QA commented on HIVE-18467:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12908491/HIVE-18467.0.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 25 failed/errored test(s), 12966 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestAccumuloCliDriver.testCliDriver[accumulo_queries]
(batchId=240)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=36)
org.apache.hadoop.hive.cli.TestEncryptedHDFSCliDriver.testCliDriver[encryption_move_tbl]
(batchId=175)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucket_map_join_tez1]
(batchId=172)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=171)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=166)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[resourceplan]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=161)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vectorization_input_format_excludes]
(batchId=163)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=122)
org.apache.hadoop.hive.cli.control.TestDanglingQOuts.checkDanglingQOut
(batchId=221)
org.apache.hadoop.hive.metastore.client.TestTablesCreateDropAlterTruncate.testAlterTableNullStorageDescriptorInNew[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesGetExists.testGetAllTablesCaseInsensitive[Embedded]
(batchId=206)
org.apache.hadoop.hive.metastore.client.TestTablesList.testListTableNamesByFilterNullDatabase[Embedded]
(batchId=206)
org.apache.hadoop.hive.ql.exec.TestOperators.testNoConditionalTaskSizeForLlap
(batchId=282)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=256)
org.apache.hadoop.hive.ql.lockmgr.TestDummyTxnManager.testSingleReadTable
(batchId=295)
org.apache.hive.beeline.cli.TestHiveCli.testNoErrorDB (batchId=188)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=234)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=234)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=234)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8946/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8946/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8946/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 25 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12908491 - PreCommit-HIVE-Build
> support whole warehouse dump / load + create/drop database events
> -
>
> Key: HIVE-18467
> URL: https://issues.apache.org/jira/browse/HIVE-18467
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch
>
>
> A complete hive warehouse might be required to replicate to a DR site for
> certain use cases and rather than allowing only a database name in the REPL
> DUMP commands, we should allow dumping of all databases using the "*" option
> as in
> _REPL DUMP *_
> On the repl load side there will not be an option to specify the database
> name when loading from a location used to dump multiple databases, hence only
> _REPL LOAD FROM [location]_ would be supported when dumping via _REPL DUMP *_
> Additionally, incremental dumps will go through all events across databases
> in a warehouse and hence CREATE / DROP Database events have to be serialized
> correctly to allow repl load to create them correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[
https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16346753#comment-16346753
]
Hive QA commented on HIVE-18467:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
21s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
32s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
8s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
20s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
15s{color} | {color:red} standalone-metastore: The patch generated 6 new + 13
unchanged - 4 fixed = 19 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
36s{color} | {color:red} ql: The patch generated 4 new + 34 unchanged - 1 fixed
= 38 total (was 35) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} itests/hive-unit: The patch generated 9 new + 12
unchanged - 3 fixed = 21 total (was 15) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 20m 46s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 79e4740 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8946/yetus/diff-checkstyle-standalone-metastore.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8946/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8946/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: standalone-metastore ql itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8946/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> support whole warehouse dump / load + create/drop database events
> -
>
> Key: HIVE-18467
> URL: https://issues.apache.org/jira/browse/HIVE-18467
> Project: Hive
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: anishek
>Assignee: anishek
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.0.0
>
> Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch
>
>
> A complete hive warehouse might be required to replicate to a DR site for
> certain use cases and rather than allowing only a database name in the REPL
> DUMP commands, we should allow dumping of all databases using the "*" option
[jira] [Updated] (HIVE-18467) support whole warehouse dump / load + create/drop database events
[ https://issues.apache.org/jira/browse/HIVE-18467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] anishek updated HIVE-18467: --- Attachment: HIVE-18467.1.patch > support whole warehouse dump / load + create/drop database events > - > > Key: HIVE-18467 > URL: https://issues.apache.org/jira/browse/HIVE-18467 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: anishek >Priority: Major > Labels: pull-request-available > Fix For: 3.0.0 > > Attachments: HIVE-18467.0.patch, HIVE-18467.1.patch > > > A complete hive warehouse might be required to replicate to a DR site for > certain use cases and rather than allowing only a database name in the REPL > DUMP commands, we should allow dumping of all databases using the "*" option > as in > _REPL DUMP *_ > On the repl load side there will not be an option to specify the database > name when loading from a location used to dump multiple databases, hence only > _REPL LOAD FROM [location]_ would be supported when dumping via _REPL DUMP *_ > Additionally, incremental dumps will go through all events across databases > in a warehouse and hence CREATE / DROP Database events have to be serialized > correctly to allow repl load to create them correctly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
