[jira] [Commented] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[
https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453525#comment-16453525
]
Hive QA commented on HIVE-19135:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920721/HIVE-19135.2.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 42 failed/errored test(s), 14308 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3
(batchId=300)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10493/testReport
Console output:
[jira] [Commented] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[
https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453496#comment-16453496
]
Hive QA commented on HIVE-19135:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
33s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
53s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
58s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
14s{color} | {color:red} beeline: The patch generated 8 new + 65 unchanged - 1
fixed = 73 total (was 66) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 2 new + 0
unchanged - 0 fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
1s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10493/dev-support/hive-personality.sh
|
| git revision | master / f30efbe |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10493/yetus/diff-checkstyle-beeline.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10493/yetus/diff-checkstyle-itests_hive-unit.txt
|
| modules | C: beeline itests/hive-unit standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10493/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Need tool to allow admins to create catalogs and move existing dbs to catalog
> during upgrade
>
>
> Key: HIVE-19135
> URL: https://issues.apache.org/jira/browse/HIVE-19135
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-19135.2.patch, HIVE19135.patch
>
>
> As part of upgrading to Hive 3 admins may wish to create new catalogs and
> move some existing databases into those catalogs. We can do this by adding
> options to schematool. This guarantees that only admins can do these
> operations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19233) Add utility for acid 1.0 to 2.0 migration
[ https://issues.apache.org/jira/browse/HIVE-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19233: Resolution: Fixed Fix Version/s: 3.0.0 Status: Resolved (was: Patch Available) Pushed to master & branch-3. Thanks, Eugene! > Add utility for acid 1.0 to 2.0 migration > - > > Key: HIVE-19233 > URL: https://issues.apache.org/jira/browse/HIVE-19233 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Fix For: 3.0.0 > > Attachments: HIVE-19233.01.patch, HIVE-19233.02.patch, > HIVE-19233.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19311) Partition and bucketing support for “load data” statement
[ https://issues.apache.org/jira/browse/HIVE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-19311: -- Attachment: HIVE-19311.2.patch > Partition and bucketing support for “load data” statement > - > > Key: HIVE-19311 > URL: https://issues.apache.org/jira/browse/HIVE-19311 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-19311.1.patch, HIVE-19311.2.patch > > > Currently, "load data" statement is very limited. It errors out if any of the > information is missing such as partitioning info if table is partitioned or > appropriate names when table is bucketed. > It should be able to launch an insert job to load the data instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19313) TestJdbcWithDBTokenStoreNoDoAs tests are failing
[
https://issues.apache.org/jira/browse/HIVE-19313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ashutosh Chauhan reassigned HIVE-19313:
---
> TestJdbcWithDBTokenStoreNoDoAs tests are failing
>
>
> Key: HIVE-19313
> URL: https://issues.apache.org/jira/browse/HIVE-19313
> Project: Hive
> Issue Type: Sub-task
> Components: Test
>Reporter: Ashutosh Chauhan
>Assignee: Thejas M Nair
>Priority: Major
>
> {code}
> Stacktrace
> java.sql.SQLException: Could not open client transport with JDBC Uri:
> jdbc:hive2://localhost:37606/default;principal=hive/localh...@example.com;:
> java.net.ConnectException: Connection refused
> at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:252)
> at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:107)
> at java.sql.DriverManager.getConnection(DriverManager.java:664)
> at java.sql.DriverManager.getConnection(DriverManager.java:270)
> at
> org.apache.hive.minikdc.TestJdbcWithMiniKdc.testRenewDelegationToken(TestJdbcWithMiniKdc.java:180)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {code}
> Failing repeatedly in Hive QA builds.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19269) Vectorization: Turn On by Default
[ https://issues.apache.org/jira/browse/HIVE-19269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19269: Status: Patch Available (was: In Progress) > Vectorization: Turn On by Default > - > > Key: HIVE-19269 > URL: https://issues.apache.org/jira/browse/HIVE-19269 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19269.01.patch, HIVE-19269.02.patch, > HIVE-19269.04.patch > > > Reflect that our most expected Hive deployment will be using vectorization > and change the default of hive.vectorized.execution.enabled to true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19269) Vectorization: Turn On by Default
[ https://issues.apache.org/jira/browse/HIVE-19269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19269: Status: In Progress (was: Patch Available) > Vectorization: Turn On by Default > - > > Key: HIVE-19269 > URL: https://issues.apache.org/jira/browse/HIVE-19269 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19269.01.patch, HIVE-19269.02.patch, > HIVE-19269.04.patch > > > Reflect that our most expected Hive deployment will be using vectorization > and change the default of hive.vectorized.execution.enabled to true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19269) Vectorization: Turn On by Default
[ https://issues.apache.org/jira/browse/HIVE-19269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19269: Attachment: HIVE-19269.04.patch > Vectorization: Turn On by Default > - > > Key: HIVE-19269 > URL: https://issues.apache.org/jira/browse/HIVE-19269 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19269.01.patch, HIVE-19269.02.patch, > HIVE-19269.04.patch > > > Reflect that our most expected Hive deployment will be using vectorization > and change the default of hive.vectorized.execution.enabled to true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18910) Migrate to Murmur hash for shuffle and bucketing
[ https://issues.apache.org/jira/browse/HIVE-18910?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-18910: -- Attachment: HIVE-18910.43.patch > Migrate to Murmur hash for shuffle and bucketing > > > Key: HIVE-18910 > URL: https://issues.apache.org/jira/browse/HIVE-18910 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-18910.1.patch, HIVE-18910.10.patch, > HIVE-18910.11.patch, HIVE-18910.12.patch, HIVE-18910.13.patch, > HIVE-18910.14.patch, HIVE-18910.15.patch, HIVE-18910.16.patch, > HIVE-18910.17.patch, HIVE-18910.18.patch, HIVE-18910.19.patch, > HIVE-18910.2.patch, HIVE-18910.20.patch, HIVE-18910.21.patch, > HIVE-18910.22.patch, HIVE-18910.23.patch, HIVE-18910.24.patch, > HIVE-18910.25.patch, HIVE-18910.26.patch, HIVE-18910.27.patch, > HIVE-18910.28.patch, HIVE-18910.29.patch, HIVE-18910.3.patch, > HIVE-18910.30.patch, HIVE-18910.31.patch, HIVE-18910.32.patch, > HIVE-18910.33.patch, HIVE-18910.34.patch, HIVE-18910.35.patch, > HIVE-18910.36.patch, HIVE-18910.36.patch, HIVE-18910.37.patch, > HIVE-18910.38.patch, HIVE-18910.39.patch, HIVE-18910.4.patch, > HIVE-18910.40.patch, HIVE-18910.41.patch, HIVE-18910.42.patch, > HIVE-18910.43.patch, HIVE-18910.5.patch, HIVE-18910.6.patch, > HIVE-18910.7.patch, HIVE-18910.8.patch, HIVE-18910.9.patch > > > Hive uses JAVA hash which is not as good as murmur for better distribution > and efficiency in bucketing a table. > Migrate to murmur hash but still keep backward compatibility for existing > users so that they dont have to reload the existing tables. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19269) Vectorization: Turn On by Default
[ https://issues.apache.org/jira/browse/HIVE-19269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matt McCline updated HIVE-19269: Attachment: (was: HIVE-19269.03.patch) > Vectorization: Turn On by Default > - > > Key: HIVE-19269 > URL: https://issues.apache.org/jira/browse/HIVE-19269 > Project: Hive > Issue Type: Bug > Components: Hive >Reporter: Matt McCline >Assignee: Matt McCline >Priority: Critical > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19269.01.patch, HIVE-19269.02.patch > > > Reflect that our most expected Hive deployment will be using vectorization > and change the default of hive.vectorized.execution.enabled to true. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[
https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453473#comment-16453473
]
Hive QA commented on HIVE-19309:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920722/HIVE-19309.1.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10492/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10492/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10492/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-04-26 04:09:25.044
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-10492/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-04-26 04:09:25.046
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at f30efbe HIVE-18986: Table rename will run
java.lang.StackOverflowError in dataNucleus if the table contains large number
of columns (Aihua Xu, reviewed by Yongzhi Chen)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at f30efbe HIVE-18986: Table rename will run
java.lang.StackOverflowError in dataNucleus if the table contains large number
of columns (Aihua Xu, reviewed by Yongzhi Chen)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-04-26 04:09:30.478
+ rm -rf ../yetus_PreCommit-HIVE-Build-10492
+ mkdir ../yetus_PreCommit-HIVE-Build-10492
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-10492
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-10492/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
protoc-jar: executing: [/tmp/protoc4644739548511796752.exe, --version]
libprotoc 2.5.0
protoc-jar: executing: [/tmp/protoc4644739548511796752.exe,
-I/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore,
--java_out=/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources,
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/protobuf/org/apache/hadoop/hive/metastore/metastore.proto]
ANTLR Parser Generator Version 3.5.2
Output file
/data/hiveptest/working/apache-github-source-source/standalone-metastore/target/generated-sources/org/apache/hadoop/hive/metastore/parser/FilterParser.java
does not exist: must build
/data/hiveptest/working/apache-github-source-source/standalone-metastore/src/main/java/org/apache/hadoop/hive/metastore/parser/Filter.g
org/apache/hadoop/hive/metastore/parser/Filter.g
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process
(process-resource-bundles) on project hive-shims-scheduler: Execution
process-resource-bundles of goal
org.apache.maven.plugins:maven-remote-resources-plugin:1.5:process failed.
ConcurrentModificationException -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/PluginExecutionException
[ERROR]
[jira] [Updated] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19309: Status: Patch Available (was: Open) Arrow doesn't package dependencies in their Maven artifacts, so needed to also add: https://github.com/google/flatbuffers https://labs.carrotsearch.com/hppc.html > Add Arrow dependencies to LlapServiceDriver > --- > > Key: HIVE-19309 > URL: https://issues.apache.org/jira/browse/HIVE-19309 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19309.1.patch, HIVE-19309.2.patch > > > Need to make arrow jars available to daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19309: Attachment: HIVE-19309.2.patch > Add Arrow dependencies to LlapServiceDriver > --- > > Key: HIVE-19309 > URL: https://issues.apache.org/jira/browse/HIVE-19309 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19309.1.patch, HIVE-19309.2.patch > > > Need to make arrow jars available to daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19309: Status: Open (was: Patch Available) > Add Arrow dependencies to LlapServiceDriver > --- > > Key: HIVE-19309 > URL: https://issues.apache.org/jira/browse/HIVE-19309 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19309.1.patch > > > Need to make arrow jars available to daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19309: Status: Patch Available (was: In Progress) > Add Arrow dependencies to LlapServiceDriver > --- > > Key: HIVE-19309 > URL: https://issues.apache.org/jira/browse/HIVE-19309 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19309.1.patch > > > Need to make arrow jars available to daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19309 started by Eric Wohlstadter. --- > Add Arrow dependencies to LlapServiceDriver > --- > > Key: HIVE-19309 > URL: https://issues.apache.org/jira/browse/HIVE-19309 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19309.1.patch > > > Need to make arrow jars available to daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19309) Add Arrow dependencies to LlapServiceDriver
[ https://issues.apache.org/jira/browse/HIVE-19309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19309: Attachment: HIVE-19309.1.patch > Add Arrow dependencies to LlapServiceDriver > --- > > Key: HIVE-19309 > URL: https://issues.apache.org/jira/browse/HIVE-19309 > Project: Hive > Issue Type: Task > Components: llap >Reporter: Eric Wohlstadter >Assignee: Eric Wohlstadter >Priority: Major > Attachments: HIVE-19309.1.patch > > > Need to make arrow jars available to daemons. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19233) Add utility for acid 1.0 to 2.0 migration
[
https://issues.apache.org/jira/browse/HIVE-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453460#comment-16453460
]
Hive QA commented on HIVE-19233:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920575/HIVE-19233.04.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 46 failed/errored test(s), 14280 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
TestTxnExIm - did not produce a TEST-*.xml file (likely timed out) (batchId=286)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=183)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.llap.security.TestLlapSignerImpl.testSigning
(batchId=309)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3
(batchId=300)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
[jira] [Commented] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[
https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453458#comment-16453458
]
Alan Gates commented on HIVE-19135:
---
I've rebased and added a --ifNotExists flag for createCatalog and changed the
select to select for update. I don't think I can use rs.getInt("CTLG_ID")
since I'm selecting max(CTLG_ID). I pushed my changes to the github branch as
well for easier review.
> Need tool to allow admins to create catalogs and move existing dbs to catalog
> during upgrade
>
>
> Key: HIVE-19135
> URL: https://issues.apache.org/jira/browse/HIVE-19135
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE-19135.2.patch, HIVE19135.patch
>
>
> As part of upgrading to Hive 3 admins may wish to create new catalogs and
> move some existing databases into those catalogs. We can do this by adding
> options to schematool. This guarantees that only admins can do these
> operations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[ https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alan Gates updated HIVE-19135: -- Attachment: HIVE-19135.2.patch > Need tool to allow admins to create catalogs and move existing dbs to catalog > during upgrade > > > Key: HIVE-19135 > URL: https://issues.apache.org/jira/browse/HIVE-19135 > Project: Hive > Issue Type: Sub-task > Components: Metastore >Affects Versions: 3.0.0 >Reporter: Alan Gates >Assignee: Alan Gates >Priority: Blocker > Fix For: 3.1.0 > > Attachments: HIVE-19135.2.patch, HIVE19135.patch > > > As part of upgrading to Hive 3 admins may wish to create new catalogs and > move some existing databases into those catalogs. We can do this by adding > options to schematool. This guarantees that only admins can do these > operations. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19233) Add utility for acid 1.0 to 2.0 migration
[
https://issues.apache.org/jira/browse/HIVE-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453448#comment-16453448
]
Hive QA commented on HIVE-19233:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
51s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
51s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
55s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
32s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
10s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
51s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
54s{color} | {color:red} root: The patch generated 17 new + 47 unchanged - 1
fixed = 64 total (was 48) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
42s{color} | {color:red} ql: The patch generated 2 new + 1 unchanged - 0 fixed
= 3 total (was 1) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
20s{color} | {color:red} standalone-metastore: The patch generated 15 new + 46
unchanged - 1 fixed = 61 total (was 47) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
34s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 57m 39s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc xml compile findbugs
checkstyle |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10491/dev-support/hive-personality.sh
|
| git revision | master / f30efbe |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10491/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10491/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10491/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: . ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10491/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Add utility for acid 1.0 to 2.0 migration
> -
>
> Key: HIVE-19233
> URL: https://issues.apache.org/jira/browse/HIVE-19233
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Affects Versions: 3.0.0
>Reporter: Eugene Koifman
>Assignee: Eugene Koifman
>Priority: Major
> Attachments: HIVE-19233.01.patch, HIVE-19233.02.patch,
> HIVE-19233.04.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Work started] (HIVE-19110) Vectorization: Enabling vectorization causes TestContribCliDriver udf_example_arraymapstruct.q to produce Wrong Results
[ https://issues.apache.org/jira/browse/HIVE-19110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19110 started by Haifeng Chen. --- > Vectorization: Enabling vectorization causes TestContribCliDriver > udf_example_arraymapstruct.q to produce Wrong Results > --- > > Key: HIVE-19110 > URL: https://issues.apache.org/jira/browse/HIVE-19110 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19110) Vectorization: Enabling vectorization causes TestContribCliDriver udf_example_arraymapstruct.q to produce Wrong Results
[ https://issues.apache.org/jira/browse/HIVE-19110?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen reassigned HIVE-19110: --- Assignee: Haifeng Chen > Vectorization: Enabling vectorization causes TestContribCliDriver > udf_example_arraymapstruct.q to produce Wrong Results > --- > > Key: HIVE-19110 > URL: https://issues.apache.org/jira/browse/HIVE-19110 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19108) Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q causes Wrong Query Results
[ https://issues.apache.org/jira/browse/HIVE-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453387#comment-16453387 ] Haifeng Chen commented on HIVE-19108: - Just add that the failure decimal failures are fixed by updating their reference output (explain results) replacing CastDecimalToDouble to CastDecimalToFloat for all float cast cases. > Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q > causes Wrong Query Results > --- > > Key: HIVE-19108 > URL: https://issues.apache.org/jira/browse/HIVE-19108 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19108.01.patch, HIVE-19108.02.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19108) Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q causes Wrong Query Results
[ https://issues.apache.org/jira/browse/HIVE-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19108: Status: Patch Available (was: Open) Updated the patch with the following fixes: # Fixed the one check style error and two trailing white spaces. # Update the related failures qtests: vector_decimal_1, vector_decimal_expressions, llap/vector_decimal_1, llap/vector_decimal_2, llap/vector_decimal_expressions. It seems that other failures are unrelated. I checked filter_in_or_dup.q. it passed locally with the patch. > Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q > causes Wrong Query Results > --- > > Key: HIVE-19108 > URL: https://issues.apache.org/jira/browse/HIVE-19108 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19108.01.patch, HIVE-19108.02.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19288) Implement protobuf logging hive hook.
[ https://issues.apache.org/jira/browse/HIVE-19288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Harish Jaiprakash updated HIVE-19288: - Attachment: HIVE-19288.02.patch > Implement protobuf logging hive hook. > - > > Key: HIVE-19288 > URL: https://issues.apache.org/jira/browse/HIVE-19288 > Project: Hive > Issue Type: Improvement > Components: Hive >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Attachments: HIVE-19288.01.patch, HIVE-19288.02.patch > > > Implement a protobuf based logger which will log hive hook events into date > partitioned directories. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19108) Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q causes Wrong Query Results
[ https://issues.apache.org/jira/browse/HIVE-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19108: Attachment: HIVE-19108.02.patch > Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q > causes Wrong Query Results > --- > > Key: HIVE-19108 > URL: https://issues.apache.org/jira/browse/HIVE-19108 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19108.01.patch, HIVE-19108.02.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19108) Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q causes Wrong Query Results
[ https://issues.apache.org/jira/browse/HIVE-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19108: Attachment: (was: HIVE-19108.02.patch) > Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q > causes Wrong Query Results > --- > > Key: HIVE-19108 > URL: https://issues.apache.org/jira/browse/HIVE-19108 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19108.01.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19211) New streaming ingest API and support for dynamic partitioning
[ https://issues.apache.org/jira/browse/HIVE-19211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453363#comment-16453363 ] Eugene Koifman commented on HIVE-19211: --- left RB comments - a lot of nits but not all > New streaming ingest API and support for dynamic partitioning > - > > Key: HIVE-19211 > URL: https://issues.apache.org/jira/browse/HIVE-19211 > Project: Hive > Issue Type: Sub-task > Components: Streaming >Affects Versions: 3.0.0, 3.1.0 >Reporter: Prasanth Jayachandran >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19211.1.patch, HIVE-19211.2.patch, > HIVE-19211.3.patch, HIVE-19211.4.patch, HIVE-19211.5.patch, > HIVE-19211.6.patch, HIVE-19211.7.patch > > > - New streaming API under new hive sub-module > - Dynamic partitioning support > - Auto-rollover transactions > - Automatic heartbeating -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18908) FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453362#comment-16453362
]
Hive QA commented on HIVE-18908:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920561/HIVE-18908.096.patch
{color:green}SUCCESS:{color} +1 due to 37 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 109 failed/errored test(s), 14321 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=218)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[mapjoin2]
(batchId=254)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_1]
(batchId=254)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_2]
(batchId=254)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_3]
(batchId=254)
org.apache.hadoop.hive.cli.TestBeeLineDriver.testCliDriver[smb_mapjoin_7]
(batchId=254)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[annotate_stats_join]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18] (batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join18_multi_distinct]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join6] (batchId=88)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join7] (batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[cbo_rp_join1]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[correlationoptimizer8]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[fouter_join_ppr]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[infer_join_preds]
(batchId=25)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join18] (batchId=90)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join18_multi_distinct]
(batchId=29)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join45] (batchId=20)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join46] (batchId=1)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join47] (batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join6] (batchId=77)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join7] (batchId=5)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[join_filters_overlap]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin2] (batchId=10)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin46] (batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin47] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoin] (batchId=61)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mergejoins_mixed]
(batchId=74)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[optional_outer]
(batchId=4)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoin_mapjoin2]
(batchId=27)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[skewjoinopt3]
(batchId=22)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_1]
(batchId=51)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_2]
(batchId=59)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_3]
(batchId=23)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_46]
(batchId=43)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_47]
(batchId=30)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_7]
(batchId=60)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[union_offcbo]
(batchId=47)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_left_outer_join2]
(batchId=66)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters1]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[orc_llap_counters]
(batchId=154)
[jira] [Commented] (HIVE-18908) FULL OUTER JOIN to MapJoin
[
https://issues.apache.org/jira/browse/HIVE-18908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453352#comment-16453352
]
Hive QA commented on HIVE-18908:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
59s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
28s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
50s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
25s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
31s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
12s{color} | {color:green} The patch storage-api passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} The patch common passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
23s{color} | {color:green} The patch serde passed checkstyle {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 2m
43s{color} | {color:red} root: The patch generated 362 new + 4237 unchanged -
192 fixed = 4599 total (was 4429) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} itests/hive-jmh: The patch generated 0 new + 11
unchanged - 6 fixed = 11 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
19s{color} | {color:red} ql: The patch generated 362 new + 3168 unchanged - 186
fixed = 3530 total (was 3354) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
1s{color} | {color:red} The patch has 13 line(s) that end in whitespace. Use
git apply --whitespace=fix <>. Refer
https://git-scm.com/docs/git-apply {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 8m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
14s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 2s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10490/dev-support/hive-personality.sh
|
| git revision | master / f30efbe |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10490/yetus/diff-checkstyle-root.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10490/yetus/diff-checkstyle-ql.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10490/yetus/whitespace-eol.txt
|
| modules | C: storage-api common serde . itests itests/hive-jmh ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10490/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> FULL OUTER JOIN to MapJoin
> --
>
> Key: HIVE-18908
> URL: https://issues.apache.org/jira/browse/HIVE-18908
> Project: Hive
> Issue Type: Improvement
[jira] [Updated] (HIVE-19108) Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q causes Wrong Query Results
[ https://issues.apache.org/jira/browse/HIVE-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19108: Attachment: HIVE-19108.02.patch > Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q > causes Wrong Query Results > --- > > Key: HIVE-19108 > URL: https://issues.apache.org/jira/browse/HIVE-19108 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19108.01.patch, HIVE-19108.02.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19108) Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q causes Wrong Query Results
[ https://issues.apache.org/jira/browse/HIVE-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19108: Status: Open (was: Patch Available) > Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q > causes Wrong Query Results > --- > > Key: HIVE-19108 > URL: https://issues.apache.org/jira/browse/HIVE-19108 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19108.01.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF
[ https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19312: Attachment: (was: HIVE-19312.patch) > MM tables don't work with BucketizedHIF > --- > > Key: HIVE-19312 > URL: https://issues.apache.org/jira/browse/HIVE-19312 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19312.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF
[ https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19312: Attachment: HIVE-19312.patch > MM tables don't work with BucketizedHIF > --- > > Key: HIVE-19312 > URL: https://issues.apache.org/jira/browse/HIVE-19312 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19312.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-6980) Drop table by using direct sql
[ https://issues.apache.org/jira/browse/HIVE-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453326#comment-16453326 ] Sergey Shelukhin commented on HIVE-6980: Hmm... isn't this going to mess with internal caches for datanucleus and potentially make objects invalid? I wonder if this needs some concurrency tests where we open 2 DN sessions, one gets some tables/etc as objects, the other drops them, and we make sure the first one still works and also doesn't produce incorrect results on committing? > Drop table by using direct sql > -- > > Key: HIVE-6980 > URL: https://issues.apache.org/jira/browse/HIVE-6980 > Project: Hive > Issue Type: Improvement > Components: Metastore >Affects Versions: 0.12.0 >Reporter: Selina Zhang >Assignee: Peter Vary >Priority: Major > Attachments: HIVE-6980.patch > > > Dropping table which has lots of partitions is slow. Even after applying the > patch of HIVE-6265, the drop table still takes hours (100K+ partitions). > The fixes come with two parts: > 1. use directSQL to query the partitions protect mode; > the current implementation needs to transfer the Partition object to client > and check the protect mode for each partition. I'd like to move this part of > logic to metastore. The check will be done by direct sql (if direct sql is > disabled, execute the same logic in the ObjectStore); > 2. use directSQL to drop partitions for table; > there maybe two solutions here: > 1. add "DELETE CASCADE" in the schema. In this way we only need to delete > entries from partitions table use direct sql. May need to change > datanucleus.deletionPolicy = DataNucleus. > 2. clean up the dependent tables by issue DELETE statement. This also needs > to turn on datanucleus.query.sql.allowAll > Both of above solutions should be able to fix the problem. The DELETE CASCADE > has to change schemas and prepare upgrade scripts. The second solutions added > maintenance cost if new tables added in the future releases. > Please advice. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453325#comment-16453325
]
Sergey Shelukhin commented on HIVE-19310:
-
It makes sense to make all these init calls in test only.
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Status: Patch Available (was: Open) Fix the last check style and whitespace error in code style. The unit test failures should not related to this patch. (I checked row__id test case, it passed with the patch locally) > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch, > HIVE-19118.03.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Attachment: HIVE-19118.03.patch > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch, > HIVE-19118.03.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haifeng Chen updated HIVE-19118: Status: Open (was: Patch Available) > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch, > HIVE-19118.03.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF
[ https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19312: Status: Patch Available (was: Open) [~ekoifman] [~steveyeom2017] can you take a look? small code change, plus test > MM tables don't work with BucketizedHIF > --- > > Key: HIVE-19312 > URL: https://issues.apache.org/jira/browse/HIVE-19312 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19312.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF
[ https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19312: Attachment: HIVE-19312.patch > MM tables don't work with BucketizedHIF > --- > > Key: HIVE-19312 > URL: https://issues.apache.org/jira/browse/HIVE-19312 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19312.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19306) Arrow batch serializer
[ https://issues.apache.org/jira/browse/HIVE-19306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eric Wohlstadter updated HIVE-19306: Description: Leverage the ThriftJDBCBinarySerDe code path that already exists in SemanticAnalyzer/FileSinkOperator to create a serializer that batches rows into Arrow vector batches. (was: Leverage the ThriftJDBCBinarySerDe code path that already exists in SematicAnalyzer/FileSinkOperator to create a serializer that batches rows into Arrow vector batches.) > Arrow batch serializer > -- > > Key: HIVE-19306 > URL: https://issues.apache.org/jira/browse/HIVE-19306 > Project: Hive > Issue Type: Task > Components: Serializers/Deserializers >Reporter: Eric Wohlstadter >Assignee: Teddy Choi >Priority: Major > Attachments: BUG-100471.11.patch > > > Leverage the ThriftJDBCBinarySerDe code path that already exists in > SemanticAnalyzer/FileSinkOperator to create a serializer that batches rows > into Arrow vector batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF
[ https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19312: Attachment: (was: HIVE-19312.patch) > MM tables don't work with BucketizedHIF > --- > > Key: HIVE-19312 > URL: https://issues.apache.org/jira/browse/HIVE-19312 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19312) MM tables don't work with BucketizedHIF
[ https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19312: --- > MM tables don't work with BucketizedHIF > --- > > Key: HIVE-19312 > URL: https://issues.apache.org/jira/browse/HIVE-19312 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19312.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19312) MM tables don't work with BucketizedHIF
[ https://issues.apache.org/jira/browse/HIVE-19312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19312: Attachment: HIVE-19312.patch > MM tables don't work with BucketizedHIF > --- > > Key: HIVE-19312 > URL: https://issues.apache.org/jira/browse/HIVE-19312 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19312.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19096) query result cache interferes with explain analyze
[
https://issues.apache.org/jira/browse/HIVE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453299#comment-16453299
]
Jason Dere commented on HIVE-19096:
---
results_cache tests have been moved over to MiniLlapLocalCliDriver, updating
patch as a result.
> query result cache interferes with explain analyze
> ---
>
> Key: HIVE-19096
> URL: https://issues.apache.org/jira/browse/HIVE-19096
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-19096.1.patch, HIVE-19096.2.patch
>
>
> if result cache is active; the explain analyze doesn't really return usefull
> informations; even for unseen queries the result is like this:
> {code}
> ++
> |Explain |
> ++
> | Stage-0|
> | Fetch Operator |
> | Cached Query Result:true,limit:-1 |
> ||
> ++
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19306) Arrow batch serializer
[ https://issues.apache.org/jira/browse/HIVE-19306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-19306: -- Status: Patch Available (was: Open) > Arrow batch serializer > -- > > Key: HIVE-19306 > URL: https://issues.apache.org/jira/browse/HIVE-19306 > Project: Hive > Issue Type: Task > Components: Serializers/Deserializers >Reporter: Eric Wohlstadter >Assignee: Teddy Choi >Priority: Major > Attachments: BUG-100471.11.patch > > > Leverage the ThriftJDBCBinarySerDe code path that already exists in > SematicAnalyzer/FileSinkOperator to create a serializer that batches rows > into Arrow vector batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19306) Arrow batch serializer
[ https://issues.apache.org/jira/browse/HIVE-19306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-19306: -- Attachment: BUG-100471.11.patch > Arrow batch serializer > -- > > Key: HIVE-19306 > URL: https://issues.apache.org/jira/browse/HIVE-19306 > Project: Hive > Issue Type: Task > Components: Serializers/Deserializers >Reporter: Eric Wohlstadter >Assignee: Teddy Choi >Priority: Major > Attachments: BUG-100471.11.patch > > > Leverage the ThriftJDBCBinarySerDe code path that already exists in > SematicAnalyzer/FileSinkOperator to create a serializer that batches rows > into Arrow vector batches. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453296#comment-16453296
]
Hive QA commented on HIVE-19124:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920557/HIVE-19124.09.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 44 failed/errored test(s), 14300 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[list_bucket_dml_6]
(batchId=33)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb]
(batchId=176)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3
(batchId=300)
org.apache.hadoop.hive.ql.parse.TestCopyUtils.testPrivilegedDistCpWithSameUserAsCurrentDoesNotTryToImpersonate
(batchId=231)
org.apache.hadoop.hive.ql.parse.TestReplicationOnHDFSEncryptedZones.targetAndSourceHaveDifferentEncryptionZoneKeys
(batchId=231)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
[jira] [Updated] (HIVE-19096) query result cache interferes with explain analyze
[
https://issues.apache.org/jira/browse/HIVE-19096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-19096:
--
Attachment: HIVE-19096.2.patch
> query result cache interferes with explain analyze
> ---
>
> Key: HIVE-19096
> URL: https://issues.apache.org/jira/browse/HIVE-19096
> Project: Hive
> Issue Type: Sub-task
>Reporter: Zoltan Haindrich
>Assignee: Jason Dere
>Priority: Major
> Attachments: HIVE-19096.1.patch, HIVE-19096.2.patch
>
>
> if result cache is active; the explain analyze doesn't really return usefull
> informations; even for unseen queries the result is like this:
> {code}
> ++
> |Explain |
> ++
> | Stage-0|
> | Fetch Operator |
> | Cached Query Result:true,limit:-1 |
> ||
> ++
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19124) implement a basic major compactor for MM tables
[
https://issues.apache.org/jira/browse/HIVE-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453257#comment-16453257
]
Hive QA commented on HIVE-19124:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
47s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
22s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
4s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
53s{color} | {color:red} ql in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
14s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} itests/hive-unit: The patch generated 6 new + 76
unchanged - 0 fixed = 82 total (was 76) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
50s{color} | {color:red} ql: The patch generated 28 new + 696 unchanged - 8
fixed = 724 total (was 704) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
22s{color} | {color:red} standalone-metastore: The patch generated 5 new + 569
unchanged - 0 fixed = 574 total (was 569) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m
0s{color} | {color:red} standalone-metastore generated 1 new + 54 unchanged - 0
fixed = 55 total (was 54) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 29m 37s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10489/dev-support/hive-personality.sh
|
| git revision | master / f30efbe |
| Default Java | 1.8.0_111 |
| mvninstall |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10489/yetus/patch-mvninstall-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10489/yetus/diff-checkstyle-itests_hive-unit.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10489/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10489/yetus/diff-checkstyle-standalone-metastore.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10489/yetus/whitespace-eol.txt
|
| javadoc |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10489/yetus/diff-javadoc-javadoc-standalone-metastore.txt
|
| modules | C: storage-api common itests/hive-unit ql standalone-metastore U: .
|
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10489/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> implement a basic major compactor for MM tables
> ---
>
> Key: HIVE-19124
> URL: https://issues.apache.org/jira/browse/HIVE-19124
> Project: Hive
> Issue Type: Bug
>
[jira] [Commented] (HIVE-19015) Vectorization and Parquet: When vectorized, parquet_map_of_arrays_of_ints.q gets a ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-19015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453249#comment-16453249
]
Matt McCline commented on HIVE-19015:
-
And, HIVE-19016 seems similar.
> Vectorization and Parquet: When vectorized, parquet_map_of_arrays_of_ints.q
> gets a ClassCastException
> -
>
> Key: HIVE-19015
> URL: https://issues.apache.org/jira/browse/HIVE-19015
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Vihang Karajgaonkar
>Priority: Critical
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> parquet_map_of_arrays_of_ints.q triggers this call stack:
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.readBatch(VectorizedListColumnReader.java:67)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedMapColumnReader.readBatch(VectorizedMapColumnReader.java:57)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {noformat}
> FYI: [~vihangk1]
> Adding parquet_map_of_maps.q, too. Stack trace seems related.
> {noformat}
> Caused by: java.lang.ClassCastException: optional group value (MAP) {
> repeated group key_value {
> optional binary key (UTF8);
> required int32 value;
> }
> } is not primitive
> at org.apache.parquet.schema.Type.asPrimitiveType(Type.java:213)
> ~[parquet-hadoop-bundle-1.9.0.jar:1.9.0]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.(BaseVectorizedColumnReader.java:130)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.(VectorizedListColumnReader.java:52)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:568)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19015) Vectorization and Parquet: When vectorized, parquet_map_of_arrays_of_ints.q gets a ClassCastException
[
https://issues.apache.org/jira/browse/HIVE-19015?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453232#comment-16453232
]
Matt McCline commented on HIVE-19015:
-
[~vihangk1] seems like the vectorized Parquet reader does not handle complex
types as LIST elements.
I can add a new HiveConf variable to disallow non-primitive LIST elements for
Parquet input format in the Vectorizer. Sound like a plan?
> Vectorization and Parquet: When vectorized, parquet_map_of_arrays_of_ints.q
> gets a ClassCastException
> -
>
> Key: HIVE-19015
> URL: https://issues.apache.org/jira/browse/HIVE-19015
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Vihang Karajgaonkar
>Priority: Critical
>
> Adding "SET hive.vectorized.execution.enabled=true;" to
> parquet_map_of_arrays_of_ints.q triggers this call stack:
> {noformat}
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hive.serde2.typeinfo.ListTypeInfo cannot be cast to
> org.apache.hadoop.hive.serde2.typeinfo.PrimitiveTypeInfo
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.readBatch(VectorizedListColumnReader.java:67)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedMapColumnReader.readBatch(VectorizedMapColumnReader.java:57)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:410)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {noformat}
> FYI: [~vihangk1]
> Adding parquet_map_of_maps.q, too. Stack trace seems related.
> {noformat}
> Caused by: java.lang.ClassCastException: optional group value (MAP) {
> repeated group key_value {
> optional binary key (UTF8);
> required int32 value;
> }
> } is not primitive
> at org.apache.parquet.schema.Type.asPrimitiveType(Type.java:213)
> ~[parquet-hadoop-bundle-1.9.0.jar:1.9.0]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.BaseVectorizedColumnReader.(BaseVectorizedColumnReader.java:130)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedListColumnReader.(VectorizedListColumnReader.java:52)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.buildVectorizedParquetReader(VectorizedParquetRecordReader.java:568)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.checkEndOfRowGroup(VectorizedParquetRecordReader.java:440)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:401)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:353)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.parquet.vector.VectorizedParquetRecordReader.next(VectorizedParquetRecordReader.java:92)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.doNext(HiveContextAwareRecordReader.java:360)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19282) don't nest delta directories inside LB directories for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-19282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453231#comment-16453231 ] Steve Yeom commented on HIVE-19282: --- +1 on the conditions that 1. p-tests passed without any errors (which will ensure no regression by the refactoring work in the patch) 2. successful testing on the added functions like read/write on the directories/files whose format are changed. > don't nest delta directories inside LB directories for ACID tables > -- > > Key: HIVE-19282 > URL: https://issues.apache.org/jira/browse/HIVE-19282 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19282.01.patch, HIVE-19282.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[
https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-18986:
Release Note: Added HMS configuration {{metastore.rawstore.batch.size}} to
control batch size when executing the query using JDO. That will avoid the
issues like StackOverflow if lots of objects such as column names are involved
in the query.
> Table rename will run java.lang.StackOverflowError in dataNucleus if the
> table contains large number of columns
> ---
>
> Key: HIVE-18986
> URL: https://issues.apache.org/jira/browse/HIVE-18986
> Project: Hive
> Issue Type: Sub-task
> Components: Standalone Metastore
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch,
> HIVE-18986.3.patch, HIVE-18986.4.patch
>
>
> If the table contains a lot of columns e.g, 5k, simple table rename would
> fail with the following stack trace. The issue is datanucleus can't handle
> the query with lots of colName='c1' && colName='c2' && ... .
>
> 2018-03-13 17:19:52,770 INFO
> org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]:
> ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table:
> db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13
> 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler:
> [pool-5-thread-200]: java.lang.StackOverflowError at
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at
> org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339)
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19204) Detailed errors from some tasks are not displayed to the client because the tasks don't set exception when they fail
[
https://issues.apache.org/jira/browse/HIVE-19204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aihua Xu updated HIVE-19204:
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Pushed to master. Thanks [~stakiar] for reviewing.
> Detailed errors from some tasks are not displayed to the client because the
> tasks don't set exception when they fail
>
>
> Key: HIVE-19204
> URL: https://issues.apache.org/jira/browse/HIVE-19204
> Project: Hive
> Issue Type: Improvement
> Components: Query Processor
>Affects Versions: 3.0.0
>Reporter: Aihua Xu
>Assignee: Aihua Xu
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19204.1.patch, HIVE-19204.2.patch
>
>
> In TaskRunner.java, if the tasks have exception set, then the task result
> will have such exception set and Driver.java will get such details and
> display to the client. But some tasks don't set such exceptions so the client
> won't see such details unless you check the HS2 log.
>
> {noformat}
> public void runSequential() {
> int exitVal = -101;
> try {
> exitVal = tsk.executeTask(ss == null ? null : ss.getHiveHistory());
> } catch (Throwable t) {
> if (tsk.getException() == null) {
> tsk.setException(t);
> }
> LOG.error("Error in executeTask", t);
> }
> result.setExitVal(exitVal);
> if (tsk.getException() != null) {
> result.setTaskError(tsk.getException());
> }
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18986) Table rename will run java.lang.StackOverflowError in dataNucleus if the table contains large number of columns
[ https://issues.apache.org/jira/browse/HIVE-18986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aihua Xu updated HIVE-18986: Resolution: Fixed Status: Resolved (was: Patch Available) Pushed to master. Thanks [~ychena] for reviewing. > Table rename will run java.lang.StackOverflowError in dataNucleus if the > table contains large number of columns > --- > > Key: HIVE-18986 > URL: https://issues.apache.org/jira/browse/HIVE-18986 > Project: Hive > Issue Type: Sub-task > Components: Standalone Metastore >Reporter: Aihua Xu >Assignee: Aihua Xu >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18986.1.patch, HIVE-18986.2.patch, > HIVE-18986.3.patch, HIVE-18986.4.patch > > > If the table contains a lot of columns e.g, 5k, simple table rename would > fail with the following stack trace. The issue is datanucleus can't handle > the query with lots of colName='c1' && colName='c2' && ... . > > 2018-03-13 17:19:52,770 INFO > org.apache.hadoop.hive.metastore.HiveMetaStore.audit: [pool-5-thread-200]: > ugi=anonymous ip=10.17.100.135 cmd=source:10.17.100.135 alter_table: > db=default tbl=fgv_full_var_pivoted02 newtbl=fgv_full_var_pivoted 2018-03-13 > 17:20:00,495 ERROR org.apache.hadoop.hive.metastore.RetryingHMSHandler: > [pool-5-thread-200]: java.lang.StackOverflowError at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:330) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) at > org.datanucleus.store.rdbms.sql.SQLText.toSQL(SQLText.java:339) > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[
https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453219#comment-16453219
]
Hive QA commented on HIVE-19118:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920568/HIVE-19118.02.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 40 failed/errored test(s), 14297 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3
(batchId=300)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10488/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/10488/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-10488/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing
[jira] [Updated] (HIVE-19311) Partition and bucketing support for “load data” statement
[ https://issues.apache.org/jira/browse/HIVE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-19311: -- Status: Patch Available (was: In Progress) > Partition and bucketing support for “load data” statement > - > > Key: HIVE-19311 > URL: https://issues.apache.org/jira/browse/HIVE-19311 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-19311.1.patch > > > Currently, "load data" statement is very limited. It errors out if any of the > information is missing such as partitioning info if table is partitioned or > appropriate names when table is bucketed. > It should be able to launch an insert job to load the data instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19311) Partition and bucketing support for “load data” statement
[ https://issues.apache.org/jira/browse/HIVE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-19311: -- Attachment: HIVE-19311.1.patch > Partition and bucketing support for “load data” statement > - > > Key: HIVE-19311 > URL: https://issues.apache.org/jira/browse/HIVE-19311 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > Attachments: HIVE-19311.1.patch > > > Currently, "load data" statement is very limited. It errors out if any of the > information is missing such as partitioning info if table is partitioned or > appropriate names when table is bucketed. > It should be able to launch an insert job to load the data instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19311) Partition and bucketing support for “load data” statement
[ https://issues.apache.org/jira/browse/HIVE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal reassigned HIVE-19311: - > Partition and bucketing support for “load data” statement > - > > Key: HIVE-19311 > URL: https://issues.apache.org/jira/browse/HIVE-19311 > Project: Hive > Issue Type: Bug >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Currently, "load data" statement is very limited. It errors out if any of the > information is missing such as partitioning info if table is partitioned or > appropriate names when table is bucketed. > It should be able to launch an insert job to load the data instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19311) Partition and bucketing support for “load data” statement
[ https://issues.apache.org/jira/browse/HIVE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Deepak Jaiswal updated HIVE-19311: -- Issue Type: Task (was: Bug) > Partition and bucketing support for “load data” statement > - > > Key: HIVE-19311 > URL: https://issues.apache.org/jira/browse/HIVE-19311 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Currently, "load data" statement is very limited. It errors out if any of the > information is missing such as partitioning info if table is partitioned or > appropriate names when table is bucketed. > It should be able to launch an insert job to load the data instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Work started] (HIVE-19311) Partition and bucketing support for “load data” statement
[ https://issues.apache.org/jira/browse/HIVE-19311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HIVE-19311 started by Deepak Jaiswal. - > Partition and bucketing support for “load data” statement > - > > Key: HIVE-19311 > URL: https://issues.apache.org/jira/browse/HIVE-19311 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Deepak Jaiswal >Priority: Major > > Currently, "load data" statement is very limited. It errors out if any of the > information is missing such as partitioning info if table is partitioned or > appropriate names when table is bucketed. > It should be able to launch an insert job to load the data instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[
https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453172#comment-16453172
]
Hive QA commented on HIVE-19118:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
49s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
19s{color} | {color:red} serde: The patch generated 1 new + 401 unchanged - 0
fixed = 402 total (was 401) {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 11m 45s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10488/dev-support/hive-personality.sh
|
| git revision | master / 29a8690 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10488/yetus/diff-checkstyle-serde.txt
|
| whitespace |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10488/yetus/whitespace-eol.txt
|
| modules | C: serde U: serde |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10488/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization: Turning on vectorization in escape_crlf produces wrong results
> -
>
> Key: HIVE-19118
> URL: https://issues.apache.org/jira/browse/HIVE-19118
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Haifeng Chen
>Priority: Critical
> Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch
>
>
> Found in vectorization enable by default experiment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18827) useless dynamic value exceptions strike back
[
https://issues.apache.org/jira/browse/HIVE-18827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Dere updated HIVE-18827:
--
Resolution: Fixed
Fix Version/s: 3.1.0
Status: Resolved (was: Patch Available)
Committed to master
> useless dynamic value exceptions strike back
>
>
> Key: HIVE-18827
> URL: https://issues.apache.org/jira/browse/HIVE-18827
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Jason Dere
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-18827.1.patch, HIVE-18827.2.patch
>
>
> Looking at ~master, I can see tons of exceptions like this in LLAP log:
> {noformat}
> 2018-02-27T14:07:51,989 WARN [IO-Elevator-Thread-12
> (1515669035295_0909_1_08_000117_0)] impl.RecordReaderImpl:
> NoDynamicValuesException when evaluating predicate. Skipping ORC PPD. Stats:
> numberOfValues: 9750
> intStatistics {
> minimum: 11335
> maximum: 560
> sum: 27648854404
> }
> hasNull: true
> Predicate: (BETWEEN ss_addr_sk
> DynamicValue(RS_27_customer_address_ca_address_sk_min)
> DynamicValue(RS_27_customer_address_ca_address_sk_max))
> org.apache.hadoop.hive.ql.plan.DynamicValue$NoDynamicValuesException: Value
> does not exist in registry: RS_27_customer_address_ca_address_sk_min
> at
> org.apache.hadoop.hive.ql.exec.tez.DynamicValueRegistryTez.getValue(DynamicValueRegistryTez.java:77)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getValue(DynamicValue.java:137)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getJavaValue(DynamicValue.java:97)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.plan.DynamicValue.getLiteral(DynamicValue.java:93)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.io.sarg.SearchArgumentImpl$PredicateLeafImpl.getLiteralList(SearchArgumentImpl.java:120)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateMinMax(RecordReaderImpl.java:553)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateRange(RecordReaderImpl.java:463)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl.evaluatePredicateProto(RecordReaderImpl.java:423)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.orc.impl.RecordReaderImpl$SargApplier.pickRowGroups(RecordReaderImpl.java:848)
> ~[hive-exec-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.determineRgsToRead(OrcEncodedDataReader.java:835)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.performDataRead(OrcEncodedDataReader.java:335)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:276)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader$4.run(OrcEncodedDataReader.java:273)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_112]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_112]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1965)
> ~[hadoop-common-3.0.0.3.0.0.0-776.jar:?]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:273)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.io.encoded.OrcEncodedDataReader.callInternal(OrcEncodedDataReader.java:110)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at org.apache.tez.common.CallableWithNdc.call(CallableWithNdc.java:36)
> ~[tez-common-0.9.2-SNAPSHOT.jar:0.9.2-SNAPSHOT]
> at
> org.apache.hadoop.hive.llap.daemon.impl.StatsRecordingThreadPool$WrappedCallable.call(StatsRecordingThreadPool.java:110)
> ~[hive-llap-server-3.0.0-SNAPSHOT.jar:3.0.0-SNAPSHOT]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[?:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[?:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19135) Need tool to allow admins to create catalogs and move existing dbs to catalog during upgrade
[
https://issues.apache.org/jira/browse/HIVE-19135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453155#comment-16453155
]
Thejas M Nair commented on HIVE-19135:
--
[~alangates]
It would be good to have a 'create if not exists' semantics for the create
catalog. That way ambari can just run it in an idempotent fashion.
Also, should we do a "select .. for update" to get a lock on the table, so that
two concurrently running commands don't end up failing because they use same
CTLG_ID.
rs.getInt("CTLG_ID") would be more readable than rs.getInt(1)
getConnectionToMetastore(false) - pass true instead for useful debug info ?
> Need tool to allow admins to create catalogs and move existing dbs to catalog
> during upgrade
>
>
> Key: HIVE-19135
> URL: https://issues.apache.org/jira/browse/HIVE-19135
> Project: Hive
> Issue Type: Sub-task
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 3.1.0
>
> Attachments: HIVE19135.patch
>
>
> As part of upgrading to Hive 3 admins may wish to create new catalogs and
> move some existing databases into those catalogs. We can do this by adding
> options to schematool. This guarantees that only admins can do these
> operations.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18958) Fix Spark config warnings
[
https://issues.apache.org/jira/browse/HIVE-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453145#comment-16453145
]
Bharathkrishna Guruvayoor Murali commented on HIVE-18958:
-
Fixing checkstyle issues.
[~stakiar] can you please review the code.
Also, will changing the config names of spark.yarn.driver.memoryOverhead and
spark.yarn.executor.memoryOverhead cause any backward incompatibility?
> Fix Spark config warnings
> -
>
> Key: HIVE-18958
> URL: https://issues.apache.org/jira/browse/HIVE-18958
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18958.01.patch, HIVE-18958.02.patch
>
>
> Getting a few configuration warnings in the logs that we should fix:
> {code}
> 2018-03-14T10:06:19,164 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.driver.memoryOverhead' has
> been deprecated as of Spark 2.3 and may be removed in the future. Please use
> the new key 'spark.driver.memoryOverhead' instead.
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key spark.akka.logLifecycleEvents is not
> supported any more because Spark doesn't use Akka since 2.0
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead'
> has been deprecated as of Spark 2.3 and may be removed in the future. Please
> use the new key 'spark.executor.memoryOverhead' instead.
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.server.connect.timeout=9
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.threads=8
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.connect.timeout=3
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.secret.bits=256
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.max.size=52428800
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18958) Fix Spark config warnings
[
https://issues.apache.org/jira/browse/HIVE-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-18958:
Attachment: HIVE-18958.02.patch
> Fix Spark config warnings
> -
>
> Key: HIVE-18958
> URL: https://issues.apache.org/jira/browse/HIVE-18958
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18958.01.patch, HIVE-18958.02.patch
>
>
> Getting a few configuration warnings in the logs that we should fix:
> {code}
> 2018-03-14T10:06:19,164 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.driver.memoryOverhead' has
> been deprecated as of Spark 2.3 and may be removed in the future. Please use
> the new key 'spark.driver.memoryOverhead' instead.
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key spark.akka.logLifecycleEvents is not
> supported any more because Spark doesn't use Akka since 2.0
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead'
> has been deprecated as of Spark 2.3 and may be removed in the future. Please
> use the new key 'spark.executor.memoryOverhead' instead.
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.server.connect.timeout=9
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.threads=8
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.connect.timeout=3
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.secret.bits=256
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.max.size=52428800
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18958) Fix Spark config warnings
[
https://issues.apache.org/jira/browse/HIVE-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-18958:
Attachment: (was: HIVE-18958.02.patch)
> Fix Spark config warnings
> -
>
> Key: HIVE-18958
> URL: https://issues.apache.org/jira/browse/HIVE-18958
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18958.01.patch
>
>
> Getting a few configuration warnings in the logs that we should fix:
> {code}
> 2018-03-14T10:06:19,164 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.driver.memoryOverhead' has
> been deprecated as of Spark 2.3 and may be removed in the future. Please use
> the new key 'spark.driver.memoryOverhead' instead.
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key spark.akka.logLifecycleEvents is not
> supported any more because Spark doesn't use Akka since 2.0
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead'
> has been deprecated as of Spark 2.3 and may be removed in the future. Please
> use the new key 'spark.executor.memoryOverhead' instead.
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.server.connect.timeout=9
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.threads=8
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.connect.timeout=3
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.secret.bits=256
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.max.size=52428800
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18958) Fix Spark config warnings
[
https://issues.apache.org/jira/browse/HIVE-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharathkrishna Guruvayoor Murali updated HIVE-18958:
Attachment: HIVE-18958.02.patch
> Fix Spark config warnings
> -
>
> Key: HIVE-18958
> URL: https://issues.apache.org/jira/browse/HIVE-18958
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18958.01.patch, HIVE-18958.02.patch
>
>
> Getting a few configuration warnings in the logs that we should fix:
> {code}
> 2018-03-14T10:06:19,164 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.driver.memoryOverhead' has
> been deprecated as of Spark 2.3 and may be removed in the future. Please use
> the new key 'spark.driver.memoryOverhead' instead.
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key spark.akka.logLifecycleEvents is not
> supported any more because Spark doesn't use Akka since 2.0
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead'
> has been deprecated as of Spark 2.3 and may be removed in the future. Please
> use the new key 'spark.executor.memoryOverhead' instead.
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.server.connect.timeout=9
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.threads=8
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.connect.timeout=3
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.secret.bits=256
> 2018-03-14T10:06:20,351 INFO
> [RemoteDriver-stderr-redir-d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> client.SparkClientImpl: Warning: Ignoring non-spark config property:
> hive.spark.client.rpc.max.size=52428800
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18958) Fix Spark config warnings
[
https://issues.apache.org/jira/browse/HIVE-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453125#comment-16453125
]
Hive QA commented on HIVE-18958:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920550/HIVE-18958.01.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 43 failed/errored test(s), 14296 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=54)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[partition_type_in_plan]
(batchId=73)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3
(batchId=300)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.jdbc.TestActivePassiveHA.testActivePassiveHA (batchId=242)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testTokenAuth (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testCancelRenewTokenFlow
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testConnection
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValid (batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testIsValidNeg
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testNegativeTokenAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testProxyAuth
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testRenewDelegationToken
(batchId=254)
org.apache.hive.minikdc.TestJdbcWithDBTokenStoreNoDoAs.testTokenAuth
(batchId=254)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/10487/testReport
Console output:
[jira] [Commented] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453091#comment-16453091
]
Thejas M Nair commented on HIVE-19310:
--
The whole initialization done via ObjectStore.setConf also needs to be
re-examined. But that can be a follow up jira.
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453088#comment-16453088
]
Thejas M Nair commented on HIVE-19310:
--
I think we we are skipping the init, we should skip all the initQueries, as I
see all of them serving the same purpose.
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18786) NPE in Hive windowing functions
[ https://issues.apache.org/jira/browse/HIVE-18786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Jadhav updated HIVE-18786: --- Description: When I run a Hive query with windowing functions, if there's enough data I get an NPE. For example something like this query might break: select id, created_date, max(created_date) over (partition by id) latest_created_any from ... The only workaround I've found is to remove the windowing functions entirely. The stacktrace looks suspiciously similar to +HIVE-15278+, but I'm in hive-2.3.2 which appears to have the bugfix applied. Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:297) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:365) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:287) ... 16 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.persistence.PTFRowContainer.first(PTFRowContainer.java:115) at org.apache.hadoop.hive.ql.exec.PTFPartition.iterator(PTFPartition.java:114) at org.apache.hadoop.hive.ql.udf.ptf.BasePartitionEvaluator.getPartitionAgg(BasePartitionEvaluator.java:200) at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.evaluateFunctionOnPartition(WindowingTableFunction.java:155) at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.iterator(WindowingTableFunction.java:538) at org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:349) at org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:123) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:356) was: When I run a Hive query with windowing functions, if there's enough data I get an NPE. For example something like this query might break: select id, created_date, max(created_date) over (partition by id) latest_created_any from ... The only workaround I've found is to remove the windowing functions entirely. The stacktrace looks suspiciously similar to +HIVE-15278+, but I'm in hive-2.3.2 which appears to have the bugfix applied. Caused by: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:297) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordProcessor.run(ReduceRecordProcessor.java:317) at org.apache.hadoop.hive.ql.exec.tez.TezProcessor.initializeAndRunProcessor(TezProcessor.java:185) ... 14 more Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing row (tag=0) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:365) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource.pushRecord(ReduceRecordSource.java:287) ... 16 more Caused by: java.lang.NullPointerException at org.apache.hadoop.hive.ql.exec.persistence.PTFRowContainer.first(PTFRowContainer.java:115) at org.apache.hadoop.hive.ql.exec.PTFPartition.iterator(PTFPartition.java:114) at org.apache.hadoop.hive.ql.udf.ptf.BasePartitionEvaluator.getPartitionAgg(BasePartitionEvaluator.java:200) at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.evaluateFunctionOnPartition(WindowingTableFunction.java:155) at org.apache.hadoop.hive.ql.udf.ptf.WindowingTableFunction.iterator(WindowingTableFunction.java:538) at org.apache.hadoop.hive.ql.exec.PTFOperator$PTFInvocation.finishPartition(PTFOperator.java:349) at org.apache.hadoop.hive.ql.exec.PTFOperator.process(PTFOperator.java:123) at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:897) at org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:95) at org.apache.hadoop.hive.ql.exec.tez.ReduceRecordSource$GroupIterator.next(ReduceRecordSource.java:356) > NPE in Hive windowing functions >
[jira] [Commented] (HIVE-19212) Fix findbugs yetus pre-commit checks
[
https://issues.apache.org/jira/browse/HIVE-19212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453071#comment-16453071
]
Sahil Takiar commented on HIVE-19212:
-
[~szita] attached an updated patch that downloads the FindBugs distribution to
the scratch dir. Looks like the scratch dir is automatically cleaned up after
every build already.
I tested this manually on the hiveptest-server and FindBugs works, here is the
output of running {{bash yetus-exec.sh}}
{code}
tomcat@hiveptest-server-upstream:/data/hiveptest/working/scratch/PreCommit-HIVE-Build-10177$
bash yetus-exec.sh
+ date '+%Y-%m-%d %T.%3N'
2018-04-25 19:25:35.780
+ FINDBUGS_VERSION=3.0.1
+ FINDBUGS_BASEURL=http://prdownloads.sourceforge.net/findbugs/
+ FINDBUGS_TARBALL=findbugs-3.0.1.tar
+ pushd /data/hiveptest/working/scratch/PreCommit-HIVE-Build-10177
/data/hiveptest/working/scratch/PreCommit-HIVE-Build-10177
/data/hiveptest/working/scratch/PreCommit-HIVE-Build-10177
+ curl -f -s -L -O
http://prdownloads.sourceforge.net/findbugs//findbugs-3.0.1.tar.gz
+ [[ 0 != 0 ]]
+ gunzip -c findbugs-3.0.1.tar.gz
+ tar xpf -
+ [[ 0 != 0 ]]
+ popd
/data/hiveptest/working/scratch/PreCommit-HIVE-Build-10177
+ export
FINDBUGS_HOME=/data/hiveptest/working/scratch/PreCommit-HIVE-Build-10177/findbugs-3.0.1
+
FINDBUGS_HOME=/data/hiveptest/working/scratch/PreCommit-HIVE-Build-10177/findbugs-3.0.1
+ pushd /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10177
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10177
/data/hiveptest/working/scratch/PreCommit-HIVE-Build-10177
+ export JIRA_ISSUE=HIVE-19108
+ JIRA_ISSUE=HIVE-19108
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ ./dev-support/test-patch.sh
/data/hiveptest/logs/PreCommit-HIVE-Build-10177/patches/PreCommit-HIVE-Build-10177.patch
--jenkins --jira-base-url=https://issues.apache.org/jira --jira-user=hiveqa
'--jira-password=c0mmit!'
--patch-dir=/data/hiveptest/logs/PreCommit-HIVE-Build-10177/yetus
--build-url=http://104.198.109.242/logs//PreCommit-HIVE-Build-10177/
--build-url-console=yetus.txt --build-url-artifacts=yetus
Modes: Jenkins Robot ResetRepo UnitTests
Processing:
/data/hiveptest/logs/PreCommit-HIVE-Build-10177/patches/PreCommit-HIVE-Build-10177.patch
Confirming git environment
HEAD is now at 0b6967e HIVE-19215 : JavaUtils.AnyIdDirFilter ignores base_n
directories (Sergey Shelukhin, reviewed by Prasanth Jayachandran)
Removing classification/target/
Removing patchprocess/
Removing shims/common/target/
Removing storage-api/target/
Removing target/
Already on 'master'
Your branch is up-to-date with 'origin/master'.
Current branch master is up to date.
Already on 'master'
Your branch is up-to-date with 'origin/master'.
HEAD is now at 0b6967e HIVE-19215 : JavaUtils.AnyIdDirFilter ignores base_n
directories (Sergey Shelukhin, reviewed by Prasanth Jayachandran)
Testing
/data/hiveptest/logs/PreCommit-HIVE-Build-10177/patches/PreCommit-HIVE-Build-10177.patch
patch on master.
Determining needed tests
(Depending upon input size and number of plug-ins, this may take a while)
Checking for @author tags: patch
There appear to be 0 @author tags in the patch.
maven install: master
cd /data/hiveptest/working/yetus_PreCommit-HIVE-Build-10177
mvn -DskipTests -Pitests -fae clean install -DskipTests=true
-Dmaven.javadoc.skip=true
[jira] [Updated] (HIVE-19212) Fix findbugs yetus pre-commit checks
[ https://issues.apache.org/jira/browse/HIVE-19212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HIVE-19212: Attachment: HIVE-19212.2.patch > Fix findbugs yetus pre-commit checks > > > Key: HIVE-19212 > URL: https://issues.apache.org/jira/browse/HIVE-19212 > Project: Hive > Issue Type: Sub-task > Components: Testing Infrastructure >Reporter: Sahil Takiar >Assignee: Sahil Takiar >Priority: Major > Attachments: HIVE-19212.1.patch, HIVE-19212.2.patch > > > Follow up from HIVE-18883, the committed patch isn't working and Findbugs is > still not working. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18958) Fix Spark config warnings
[
https://issues.apache.org/jira/browse/HIVE-18958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453068#comment-16453068
]
Hive QA commented on HIVE-18958:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
50s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
29s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
28s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 1 new + 19 unchanged - 3 fixed
= 20 total (was 22) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} spark-client: The patch generated 0 new + 21
unchanged - 5 fixed = 21 total (was 26) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
16s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 18m 56s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10487/dev-support/hive-personality.sh
|
| git revision | master / 29a8690 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10487/yetus/diff-checkstyle-ql.txt
|
| modules | C: ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10487/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Fix Spark config warnings
> -
>
> Key: HIVE-18958
> URL: https://issues.apache.org/jira/browse/HIVE-18958
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-18958.01.patch
>
>
> Getting a few configuration warnings in the logs that we should fix:
> {code}
> 2018-03-14T10:06:19,164 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.driver.memoryOverhead' has
> been deprecated as of Spark 2.3 and may be removed in the future. Please use
> the new key 'spark.driver.memoryOverhead' instead.
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key spark.akka.logLifecycleEvents is not
> supported any more because Spark doesn't use Akka since 2.0
> 2018-03-14T10:06:19,165 WARN [d5ade9e4-9354-40f1-8f74-631f373709b3 main]
> spark.SparkConf: The configuration key 'spark.yarn.executor.memoryOverhead'
> has been deprecated as of Spark 2.3 and may be removed in the
[jira] [Commented] (HIVE-18862) qfiles: prepare .q files for using datasets
[ https://issues.apache.org/jira/browse/HIVE-18862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453043#comment-16453043 ] Laszlo Bodor commented on HIVE-18862: - Thanks [~kgyrtkirk] > qfiles: prepare .q files for using datasets > --- > > Key: HIVE-18862 > URL: https://issues.apache.org/jira/browse/HIVE-18862 > Project: Hive > Issue Type: Sub-task >Reporter: Laszlo Bodor >Assignee: Laszlo Bodor >Priority: Major > Fix For: 3.1.0 > > Attachments: HIVE-18862.01.patch, HIVE-18862.02.patch, > HIVE-18862.03.patch, HIVE-18862.04.patch, HIVE-18862.05.patch, > HIVE-18862.06.patch, HIVE-18862.07.patch, HIVE-18862.08.patch, > HIVE-18862.09-branch-3.patch, HIVE-18862.09.patch > > > # Parse .q files for source table usage > # Add needed dataset annotations > # Remove create table statements from "q_test_init.sql" like files > # Handle oncoming issues related to dataset introduction -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19108) Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q causes Wrong Query Results
[
https://issues.apache.org/jira/browse/HIVE-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16453007#comment-16453007
]
Hive QA commented on HIVE-19108:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12920569/HIVE-19108.01.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 60 failed/errored test(s), 14298 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=93)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q,bucket_num_reducers_acid2.q]
TestNonCatCallsWithCatalog - did not produce a TEST-*.xml file (likely timed
out) (batchId=217)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_nullscan]
(batchId=68)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[acid_table_stats]
(batchId=55)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[autoColumnStats_4]
(batchId=13)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[filter_in_or_dup]
(batchId=36)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_vectorization_0]
(batchId=17)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[row__id] (batchId=80)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_1]
(batchId=82)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=54)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[acid_bucket_pruning]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[acid_vectorization_original]
(batchId=173)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[check_constraint]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[default_constraint]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[dynpart_sort_optimization_acid]
(batchId=165)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[enforce_constraint_notnull]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=169)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_4]
(batchId=157)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[materialized_view_create_rewrite_5]
(batchId=154)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_stats]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_1]
(batchId=174)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_2]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_decimal_expressions]
(batchId=167)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[acid_vectorization_original_tez]
(batchId=106)
org.apache.hadoop.hive.cli.TestMiniTezCliDriver.testCliDriver[explainanalyze_5]
(batchId=105)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[cluster_tasklog_retrieval]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[mapreduce_stack_trace_turnoff]
(batchId=98)
org.apache.hadoop.hive.cli.TestNegativeMinimrCliDriver.testCliDriver[minimr_broken_pipe]
(batchId=98)
org.apache.hadoop.hive.ql.TestAcidOnTez.testAcidInsertWithRemoveUnion
(batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testCtasTezUnion (batchId=228)
org.apache.hadoop.hive.ql.TestAcidOnTez.testNonStandardConversion01
(batchId=228)
org.apache.hadoop.hive.ql.TestMTQueries.testMTQueries1 (batchId=232)
org.apache.hadoop.hive.ql.lockmgr.TestDbTxnManager2.testWriteSetTracking3
(batchId=300)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgress (batchId=235)
org.apache.hive.beeline.TestBeeLineWithArgs.testQueryProgressParallel
(batchId=235)
org.apache.hive.jdbc.TestActivePassiveHA.testClientConnectionsOnFailover
(batchId=242)
org.apache.hive.jdbc.TestSSL.testSSLFetchHttp (batchId=239)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testMultipleTriggers2
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitions
(batchId=242)
org.apache.hive.jdbc.TestTriggersWorkloadManager.testTriggerCustomCreatedDynamicPartitionsMultiInsert
(batchId=242)
[jira] [Assigned] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta reassigned HIVE-19310:
---
Assignee: Vaibhav Gumashta
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Vaibhav Gumashta
>Assignee: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19310) Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might need to be run only in test env
[
https://issues.apache.org/jira/browse/HIVE-19310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vaibhav Gumashta updated HIVE-19310:
Attachment: HIVE-19310.1.patch
> Metastore: MetaStoreDirectSql.ensureDbInit has some slow DN calls which might
> need to be run only in test env
> -
>
> Key: HIVE-19310
> URL: https://issues.apache.org/jira/browse/HIVE-19310
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0, 3.1.0
>Reporter: Vaibhav Gumashta
>Priority: Major
> Attachments: HIVE-19310.1.patch
>
>
> MetaStoreDirectSql.ensureDbInit has the following 2 calls which we have
> observed taking a long time in our testing:
> {code}
> initQueries.add(pm.newQuery(MNotificationLog.class, "dbName == ''"));
> initQueries.add(pm.newQuery(MNotificationNextId.class, "nextEventId < -1"));
> {code}
> In a production environment, these tables should be initialized using
> schematool, however in a test environment, these calls might be needed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452984#comment-16452984 ] Matt McCline commented on HIVE-19118: - (Consider petitioning [~vgarg] to see if this should go into branch-3) > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19118) Vectorization: Turning on vectorization in escape_crlf produces wrong results
[ https://issues.apache.org/jira/browse/HIVE-19118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452983#comment-16452983 ] Matt McCline commented on HIVE-19118: - +1 LGTM tests pending. [~jerrychenhf] I'm you researched this issue and created a fix! Thank you! > Vectorization: Turning on vectorization in escape_crlf produces wrong results > - > > Key: HIVE-19118 > URL: https://issues.apache.org/jira/browse/HIVE-19118 > Project: Hive > Issue Type: Bug > Components: Hive >Affects Versions: 3.0.0 >Reporter: Matt McCline >Assignee: Haifeng Chen >Priority: Critical > Attachments: HIVE-19118.01.patch, HIVE-19118.02.patch > > > Found in vectorization enable by default experiment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18193: -- Fix Version/s: (was: 3.1.0) > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452979#comment-16452979 ] Eugene Koifman commented on HIVE-18193: --- this is blocker for 3.0 > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18193) Migrate existing ACID tables to use write id per table rather than global transaction id
[ https://issues.apache.org/jira/browse/HIVE-18193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-18193: -- Priority: Blocker (was: Major) > Migrate existing ACID tables to use write id per table rather than global > transaction id > > > Key: HIVE-18193 > URL: https://issues.apache.org/jira/browse/HIVE-18193 > Project: Hive > Issue Type: Sub-task > Components: HiveServer2, Transactions >Affects Versions: 3.0.0 >Reporter: anishek >Assignee: Sankar Hariappan >Priority: Blocker > Labels: ACID, Upgrade > > dependent upon HIVE-18192 > For existing ACID Tables we need to update the table level write id > metatables/sequences so any new operations on these tables works seamlessly > without any conflicting data in existing base/delta files. > 1. Need to create metadata tables such as NEXT_WRITE_ID and TXN_TO_WRITE_ID. > 2. Add entries for each ACID/MM tables into NEXT_WRITE_ID where NWI_NEXT is > set to current value of NEXT_TXN_ID.NTXN_NEXT. > 3. All current open/abort transactions to have an entry in TXN_TO_WRITE_ID > such that T2W_TXNID=T2W_WRITEID=Open/AbortedTxnId. > 4. Added new column TC_WRITEID in TXN_COMPONENTS and CTC_WRITEID in > COMPLETED_TXN_COMPONENTS to store the write id which should be set as > respective values of TC_TXNID and CTC_TXNID from the same row. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19260) Streaming Ingest API doesn't normalize db.table names
[ https://issues.apache.org/jira/browse/HIVE-19260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452978#comment-16452978 ] Eugene Koifman commented on HIVE-19260: --- committed to branch-3 > Streaming Ingest API doesn't normalize db.table names > - > > Key: HIVE-19260 > URL: https://issues.apache.org/jira/browse/HIVE-19260 > Project: Hive > Issue Type: Bug > Components: HCatalog, Streaming, Transactions >Affects Versions: 0.13 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HIVE-19260.01-branch-3.patch, HIVE-19260.01.patch > > > Hive is case insensitive wrt db/table names. These gets normalized to lower > case for SQL processing. > When HiveEndPoint is created it uses db.table strings as is, and they end up > propagated this way to transaction metadata tables in the metastore via lock > acquisition. This makes them look like different tables in Cleaner and lock > manager. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
[ https://issues.apache.org/jira/browse/HIVE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vaibhav Gumashta updated HIVE-19252: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to branch-3 and master. Thanks [~daijy] > TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently > > > Key: HIVE-19252 > URL: https://issues.apache.org/jira/browse/HIVE-19252 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Ashutosh Chauhan >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19252.1.patch, HIVE-19252.1.patch > > > For last 8 builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19252) TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently
[ https://issues.apache.org/jira/browse/HIVE-19252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452966#comment-16452966 ] Vaibhav Gumashta commented on HIVE-19252: - Test failures are unrelated. Will commit this > TestJdbcWithMiniKdcCookie.testCookieNegative is failing consistently > > > Key: HIVE-19252 > URL: https://issues.apache.org/jira/browse/HIVE-19252 > Project: Hive > Issue Type: Sub-task > Components: Test >Reporter: Ashutosh Chauhan >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19252.1.patch, HIVE-19252.1.patch > > > For last 8 builds. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19198) Few flaky hcatalog tests
[ https://issues.apache.org/jira/browse/HIVE-19198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452964#comment-16452964 ] Vaibhav Gumashta commented on HIVE-19198: - +1 > Few flaky hcatalog tests > > > Key: HIVE-19198 > URL: https://issues.apache.org/jira/browse/HIVE-19198 > Project: Hive > Issue Type: Sub-task >Reporter: Ashutosh Chauhan >Assignee: Daniel Dai >Priority: Major > Attachments: HIVE-19198.1.patch, HIVE-19198.2.patch > > > TestPermsGrp : Consider removing this since hcat cli is not widely used. > TestHCatPartitionPublish.testPartitionPublish > TestHCatMultiOutputFormat.testOutputFormat -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19282) don't nest delta directories inside LB directories for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-19282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452948#comment-16452948 ] Sergey Shelukhin commented on HIVE-19282: - See links, RB is linked > don't nest delta directories inside LB directories for ACID tables > -- > > Key: HIVE-19282 > URL: https://issues.apache.org/jira/browse/HIVE-19282 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19282.01.patch, HIVE-19282.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-12342) Set default value of hive.optimize.index.filter to true
[ https://issues.apache.org/jira/browse/HIVE-12342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Igor Kryvenko reassigned HIVE-12342: Assignee: Igor Kryvenko (was: Ashutosh Chauhan) > Set default value of hive.optimize.index.filter to true > --- > > Key: HIVE-12342 > URL: https://issues.apache.org/jira/browse/HIVE-12342 > Project: Hive > Issue Type: Task > Components: Configuration >Reporter: Ashutosh Chauhan >Assignee: Igor Kryvenko >Priority: Major > Attachments: HIVE-12342.1.patch, HIVE-12342.2.patch, > HIVE-12342.3.patch, HIVE-12342.patch > > > This configuration governs ppd for storage layer. When applicable, it will > always help. It should be on by default. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19233) Add utility for acid 1.0 to 2.0 migration
[ https://issues.apache.org/jira/browse/HIVE-19233?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452940#comment-16452940 ] Ashutosh Chauhan commented on HIVE-19233: - Couple of high level comments: * Currently it generates 4 scripts. It should accept an argument and generate only ones specified. * Its better to have this standalone in a module of its own so users of older versions of Hive can use it. These can be addressed in a follow-up. +1 for this. > Add utility for acid 1.0 to 2.0 migration > - > > Key: HIVE-19233 > URL: https://issues.apache.org/jira/browse/HIVE-19233 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > Attachments: HIVE-19233.01.patch, HIVE-19233.02.patch, > HIVE-19233.04.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19108) Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q causes Wrong Query Results
[
https://issues.apache.org/jira/browse/HIVE-19108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452935#comment-16452935
]
Hive QA commented on HIVE-19108:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
26s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
18s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
29s{color} | {color:blue} spark-client in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
13s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 9m 36s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-10177/dev-support/hive-personality.sh
|
| git revision | master / 0b6967e |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: spark-client U: spark-client |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-10177/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Vectorization and Parquet: Turning on vectorization in parquet_ppd_decimal.q
> causes Wrong Query Results
> ---
>
> Key: HIVE-19108
> URL: https://issues.apache.org/jira/browse/HIVE-19108
> Project: Hive
> Issue Type: Bug
> Components: Hive
>Affects Versions: 3.0.0
>Reporter: Matt McCline
>Assignee: Haifeng Chen
>Priority: Critical
> Attachments: HIVE-19108.01.patch
>
>
> Found in vectorization enable by default experiment.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19271) TestMiniLlapLocalCliDriver default_constraint and check_constraint failing
[ https://issues.apache.org/jira/browse/HIVE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19271: --- Target Version/s: 3.0.0, 3.1.0 (was: 3.1.0) > TestMiniLlapLocalCliDriver default_constraint and check_constraint failing > -- > > Key: HIVE-19271 > URL: https://issues.apache.org/jira/browse/HIVE-19271 > Project: Hive > Issue Type: Test >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19271.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19271) TestMiniLlapLocalCliDriver default_constraint and check_constraint failing
[ https://issues.apache.org/jira/browse/HIVE-19271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vineet Garg updated HIVE-19271: --- Resolution: Fixed Fix Version/s: 3.1.0 3.0.0 Status: Resolved (was: Patch Available) Pushed to master and branch-3 > TestMiniLlapLocalCliDriver default_constraint and check_constraint failing > -- > > Key: HIVE-19271 > URL: https://issues.apache.org/jira/browse/HIVE-19271 > Project: Hive > Issue Type: Test >Reporter: Vineet Garg >Assignee: Vineet Garg >Priority: Major > Fix For: 3.0.0, 3.1.0 > > Attachments: HIVE-19271.1.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19247) StatsOptimizer: Missing stats fast-path for Date
[
https://issues.apache.org/jira/browse/HIVE-19247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vineet Garg updated HIVE-19247:
---
Fix Version/s: 3.0.0
> StatsOptimizer: Missing stats fast-path for Date
>
>
> Key: HIVE-19247
> URL: https://issues.apache.org/jira/browse/HIVE-19247
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 2.2.0, 3.0.0, 2.3.2
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Fix For: 3.0.0, 3.1.0
>
> Attachments: HIVE-19247.1.patch
>
>
> {code}
> 2018-04-19T18:57:24,268 DEBUG [67259108-c184-4c92-9e18-9e296922
> HiveServer2-Handler-Pool: Thread-73]: optimizer.StatsOptimizer
> (StatsOptimizer.java:process(614)) - Unsupported type: date encountered in
> metadata optimizer for column : jour
> {code}
> {code}
> if (udaf instanceof GenericUDAFMin) {
> ExprNodeColumnDesc colDesc =
> (ExprNodeColumnDesc)exprMap.get(((ExprNodeColumnDesc)aggr.getParameters().get(0)).getColumn());
> String colName = colDesc.getColumn();
> StatType type = getType(colDesc.getTypeString());
> if (!tbl.isPartitioned()) {
> if
> (!StatsSetupConst.areColumnStatsUptoDate(tbl.getParameters(), colName)) {
> Logger.debug("Stats for table : " + tbl.getTableName() + "
> column " + colName
> + " are not up to date.");
> return null;
> }
> ColumnStatisticsData statData =
> hive.getMSC().getTableColumnStatistics(
> tbl.getDbName(), tbl.getTableName(),
> Lists.newArrayList(colName))
> .get(0).getStatsData();
> String name = colDesc.getTypeString().toUpperCase();
> switch (type) {
> case Integeral: {
> LongSubType subType = LongSubType.valueOf(name);
> LongColumnStatsData lstats = statData.getLongStats();
> if (lstats.isSetLowValue()) {
> oneRow.add(subType.cast(lstats.getLowValue()));
> } else {
> oneRow.add(null);
> }
> break;
> }
> case Double: {
> DoubleSubType subType = DoubleSubType.valueOf(name);
> DoubleColumnStatsData dstats = statData.getDoubleStats();
> if (dstats.isSetLowValue()) {
> oneRow.add(subType.cast(dstats.getLowValue()));
> } else {
> oneRow.add(null);
> }
> break;
> }
> default: // unsupported type
> Logger.debug("Unsupported type: " + colDesc.getTypeString()
> + " encountered in " +
> "metadata optimizer for column : " + colName);
> return null;
> }
> }
> {code}
> {code}
> enum StatType{
> Integeral,
> Double,
> String,
> Boolean,
> Binary,
> Unsupported
> }
> enum LongSubType {
> BIGINT { @Override
> Object cast(long longValue) { return longValue; } },
> INT { @Override
> Object cast(long longValue) { return (int)longValue; } },
> SMALLINT { @Override
> Object cast(long longValue) { return (short)longValue; } },
> TINYINT { @Override
> Object cast(long longValue) { return (byte)longValue; } };
> abstract Object cast(long longValue);
> }
> {code}
> Date is stored in stats (& also the typo there).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19247) StatsOptimizer: Missing stats fast-path for Date
[
https://issues.apache.org/jira/browse/HIVE-19247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452926#comment-16452926
]
Vineet Garg commented on HIVE-19247:
Pushed to branch-3
> StatsOptimizer: Missing stats fast-path for Date
>
>
> Key: HIVE-19247
> URL: https://issues.apache.org/jira/browse/HIVE-19247
> Project: Hive
> Issue Type: Bug
> Components: Statistics
>Affects Versions: 2.2.0, 3.0.0, 2.3.2
>Reporter: Gopal V
>Assignee: Gopal V
>Priority: Major
> Fix For: 3.1.0
>
> Attachments: HIVE-19247.1.patch
>
>
> {code}
> 2018-04-19T18:57:24,268 DEBUG [67259108-c184-4c92-9e18-9e296922
> HiveServer2-Handler-Pool: Thread-73]: optimizer.StatsOptimizer
> (StatsOptimizer.java:process(614)) - Unsupported type: date encountered in
> metadata optimizer for column : jour
> {code}
> {code}
> if (udaf instanceof GenericUDAFMin) {
> ExprNodeColumnDesc colDesc =
> (ExprNodeColumnDesc)exprMap.get(((ExprNodeColumnDesc)aggr.getParameters().get(0)).getColumn());
> String colName = colDesc.getColumn();
> StatType type = getType(colDesc.getTypeString());
> if (!tbl.isPartitioned()) {
> if
> (!StatsSetupConst.areColumnStatsUptoDate(tbl.getParameters(), colName)) {
> Logger.debug("Stats for table : " + tbl.getTableName() + "
> column " + colName
> + " are not up to date.");
> return null;
> }
> ColumnStatisticsData statData =
> hive.getMSC().getTableColumnStatistics(
> tbl.getDbName(), tbl.getTableName(),
> Lists.newArrayList(colName))
> .get(0).getStatsData();
> String name = colDesc.getTypeString().toUpperCase();
> switch (type) {
> case Integeral: {
> LongSubType subType = LongSubType.valueOf(name);
> LongColumnStatsData lstats = statData.getLongStats();
> if (lstats.isSetLowValue()) {
> oneRow.add(subType.cast(lstats.getLowValue()));
> } else {
> oneRow.add(null);
> }
> break;
> }
> case Double: {
> DoubleSubType subType = DoubleSubType.valueOf(name);
> DoubleColumnStatsData dstats = statData.getDoubleStats();
> if (dstats.isSetLowValue()) {
> oneRow.add(subType.cast(dstats.getLowValue()));
> } else {
> oneRow.add(null);
> }
> break;
> }
> default: // unsupported type
> Logger.debug("Unsupported type: " + colDesc.getTypeString()
> + " encountered in " +
> "metadata optimizer for column : " + colName);
> return null;
> }
> }
> {code}
> {code}
> enum StatType{
> Integeral,
> Double,
> String,
> Boolean,
> Binary,
> Unsupported
> }
> enum LongSubType {
> BIGINT { @Override
> Object cast(long longValue) { return longValue; } },
> INT { @Override
> Object cast(long longValue) { return (int)longValue; } },
> SMALLINT { @Override
> Object cast(long longValue) { return (short)longValue; } },
> TINYINT { @Override
> Object cast(long longValue) { return (byte)longValue; } };
> abstract Object cast(long longValue);
> }
> {code}
> Date is stored in stats (& also the typo there).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19282) don't nest delta directories inside LB directories for ACID tables
[ https://issues.apache.org/jira/browse/HIVE-19282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452914#comment-16452914 ] Steve Yeom commented on HIVE-19282: --- hi [~sershe] i think RB will be very helpful. Thanks, Steve. > don't nest delta directories inside LB directories for ACID tables > -- > > Key: HIVE-19282 > URL: https://issues.apache.org/jira/browse/HIVE-19282 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19282.01.patch, HIVE-19282.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
