[jira] [Commented] (HIVE-19588) Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader
[ https://issues.apache.org/jira/browse/HIVE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480196#comment-16480196 ] Prasanth Jayachandran commented on HIVE-19588: -- Per [~gopalv] the perf slowness comes from inner loop creation of VectorizedOrcAcidRowBatchReader for LLAP. Removed it in .2 patch. [~ekoifman] can you review this diff [https://reviews.apache.org/r/67197/diff/1-2/] ? > Several invocation of file listing when creating > VectorizedOrcAcidRowBatchReader > > > Key: HIVE-19588 > URL: https://issues.apache.org/jira/browse/HIVE-19588 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19588.1.patch, HIVE-19588.2.patch, Screen Shot > 2018-05-16 at 2.23.25 PM.png > > > Looks like we are doing file listing several times when creating one instance > of VectorizedOrcAcidRowBatchReader > AcidUtils.parseBaseOrDeltaBucketFilename() does full file listing (when > there are files with bucket_* prefix) just to get a single file out of a path > to figure out if it has ACID schema (as part of HIVE-18190). > There is full file listing where we populate > 1) ColumnizedDeleteEventRegistry > 2) SortMergedDeleteEventRegistry > 3) Twice in computeOffsetAndBucket() > > Attaching profiles which [~gopalv] took while debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19588) Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader
[ https://issues.apache.org/jira/browse/HIVE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19588: - Attachment: HIVE-19588.2.patch > Several invocation of file listing when creating > VectorizedOrcAcidRowBatchReader > > > Key: HIVE-19588 > URL: https://issues.apache.org/jira/browse/HIVE-19588 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19588.1.patch, HIVE-19588.2.patch, Screen Shot > 2018-05-16 at 2.23.25 PM.png > > > Looks like we are doing file listing several times when creating one instance > of VectorizedOrcAcidRowBatchReader > AcidUtils.parseBaseOrDeltaBucketFilename() does full file listing (when > there are files with bucket_* prefix) just to get a single file out of a path > to figure out if it has ACID schema (as part of HIVE-18190). > There is full file listing where we populate > 1) ColumnizedDeleteEventRegistry > 2) SortMergedDeleteEventRegistry > 3) Twice in computeOffsetAndBucket() > > Attaching profiles which [~gopalv] took while debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[
https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480193#comment-16480193
]
Hive QA commented on HIVE-18079:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12924000/HIVE-18079.16.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11042/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11042/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11042/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-18 05:38:04.275
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11042/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z master ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-18 05:38:04.278
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 0b6bea8 HIVE-19572: Add option to mask stats and data size in q
files (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran) (addendum)
+ git clean -f -d
+ git checkout master
Already on 'master'
Your branch is up-to-date with 'origin/master'.
+ git reset --hard origin/master
HEAD is now at 0b6bea8 HIVE-19572: Add option to mask stats and data size in q
files (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran) (addendum)
+ git merge --ff-only origin/master
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-18 05:38:05.190
+ rm -rf ../yetus_PreCommit-HIVE-Build-11042
+ mkdir ../yetus_PreCommit-HIVE-Build-11042
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11042
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11042/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
error: a/ql/src/test/queries/clientpositive/bucket_map_join_tez2.q: does not
exist in index
error: a/ql/src/test/results/clientpositive/autoColumnStats_2.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/autoColumnStats_9.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/bitvector.q.out: does not exist in
index
error: a/ql/src/test/results/clientpositive/compute_stats_date.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/confirm_initial_tbl_stats.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/cross_join_merge.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/describe_table.q.out: does not
exist in index
error:
a/ql/src/test/results/clientpositive/encrypted/encryption_move_tbl.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/hll.q.out: does not exist in index
error: a/ql/src/test/results/clientpositive/llap/acid_no_buckets.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/llap/autoColumnStats_2.q.out: does
not exist in index
error: a/ql/src/test/results/clientpositive/llap/auto_join1.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/auto_join21.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/auto_join29.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/auto_join30.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/auto_sortmerge_join_6.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/llap/bucket_groupby.q.out: does not
exist in index
error: a/ql/src/test/results/clientpositive/llap/bucket_map_join_tez1.q.out:
does not exist in index
error: a/ql/src/test/results/clientpositive/llap/bucket_map_join_tez2.q.out:
does not exist in index
error:
[jira] [Commented] (HIVE-19531) TransactionalValidationListener is getting catalog name from conf instead of table object.
[
https://issues.apache.org/jira/browse/HIVE-19531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480192#comment-16480192
]
Hive QA commented on HIVE-19531:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923954/HIVE-19531.1take3.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 5 failed/errored test(s), 14397 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testAutoPurgeUnset (batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testPartitionedNoAutoPurge
(batchId=233)
org.apache.hadoop.hive.ql.TestAutoPurgeTables.testTruncateUnsetAutoPurge
(batchId=233)
org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark.org.apache.hive.jdbc.TestMultiSessionsHS2WithLocalClusterSpark
(batchId=243)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11041/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11041/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11041/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 5 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923954 - PreCommit-HIVE-Build
> TransactionalValidationListener is getting catalog name from conf instead of
> table object.
> --
>
> Key: HIVE-19531
> URL: https://issues.apache.org/jira/browse/HIVE-19531
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 3.0.1
>
> Attachments: HIVE-19531.1again.patch, HIVE-19531.1take3.patch,
> HIVE-19531.patch
>
>
> TransactionalValidationListener.validateTableStructure get the catalog from
> the conf file rather than taking it from the passed in table structure. This
> causes createTable operations to fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19531) TransactionalValidationListener is getting catalog name from conf instead of table object.
[
https://issues.apache.org/jira/browse/HIVE-19531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480148#comment-16480148
]
Hive QA commented on HIVE-19531:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
4s{color} | {color:blue} standalone-metastore in master has 216 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
8s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore: The patch generated 1 new + 19
unchanged - 0 fixed = 20 total (was 19) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
11s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 19m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11041/dev-support/hive-personality.sh
|
| git revision | master / 0b6bea8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11041/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: standalone-metastore U: standalone-metastore |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11041/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> TransactionalValidationListener is getting catalog name from conf instead of
> table object.
> --
>
> Key: HIVE-19531
> URL: https://issues.apache.org/jira/browse/HIVE-19531
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Blocker
> Fix For: 3.0.1
>
> Attachments: HIVE-19531.1again.patch, HIVE-19531.1take3.patch,
> HIVE-19531.patch
>
>
> TransactionalValidationListener.validateTableStructure get the catalog from
> the conf file rather than taking it from the passed in table structure. This
> causes createTable operations to fail.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19602) Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor instance
[ https://issues.apache.org/jira/browse/HIVE-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-19602: Description: We can refactor the HOS inplace progress monitor code (SparkJobMonitor#printStatusInPlace) to use InplaceUpdate#render. We can create an instance of ProgressMonitor and use it to show the progress. This would be similar to : [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] was: We can refactor the HOS inplace progress monitor code (SparkJobMonitor#printStatusInPlace) to use InplaceUpdate#render. This would be similar to : [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] > Refactor inplace progress code in Hive-on-spark progress monitor to use > ProgressMonitor instance > > > Key: HIVE-19602 > URL: https://issues.apache.org/jira/browse/HIVE-19602 > Project: Hive > Issue Type: Bug >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > We can refactor the HOS inplace progress monitor code > (SparkJobMonitor#printStatusInPlace) to use InplaceUpdate#render. > We can create an instance of ProgressMonitor and use it to show the progress. > This would be similar to : > [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19491) Branch-3 Start using storage-api 2.6.1 once available.
[ https://issues.apache.org/jira/browse/HIVE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480122#comment-16480122 ] Chaoran Yu commented on HIVE-19491: --- Thanks! [~vgarg] > Branch-3 Start using storage-api 2.6.1 once available. > -- > > Key: HIVE-19491 > URL: https://issues.apache.org/jira/browse/HIVE-19491 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Vineet Garg >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HIVE-19491.01-branch-3.patch > > > branch-3 needs storage-api 2.6.1 which is in the process of being released. > > cc. [~vgarg] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19576) IHMSHandler.getTable not always fetching the right catalog
[
https://issues.apache.org/jira/browse/HIVE-19576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480120#comment-16480120
]
Hive QA commented on HIVE-19576:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923955/HIVE-19576.1take2.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 1 failed/errored test(s), 14398 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11040/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11040/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11040/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 1 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923955 - PreCommit-HIVE-Build
> IHMSHandler.getTable not always fetching the right catalog
> --
>
> Key: HIVE-19576
> URL: https://issues.apache.org/jira/browse/HIVE-19576
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.1
>
> Attachments: HIVE-19576.1take2.patch, HIVE-19576.patch
>
>
> {{IHMSHandler.get_table_core(String dbName, String tableName)}} fetches the
> catalog name from the conf. This causes issues when doing an operation where
> the catalog is known and does not match the default provided in the
> configuration file (e.g. adding a partition). This method should be removed
> and callers forced to use {{IHMSHandler.get_table_core(String catName, String
> dbName, String tableName)}} instead since callers will know whether they have
> the catalog name or not.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19576) IHMSHandler.getTable not always fetching the right catalog
[
https://issues.apache.org/jira/browse/HIVE-19576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480111#comment-16480111
]
Hive QA commented on HIVE-19576:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
46s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
57s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
54s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 2m
57s{color} | {color:blue} standalone-metastore in master has 216 extant
Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
2s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
18s{color} | {color:red} standalone-metastore: The patch generated 1 new + 2
unchanged - 1 fixed = 3 total (was 3) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 7m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
9s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 34m 33s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11040/dev-support/hive-personality.sh
|
| git revision | master / 0b6bea8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11040/yetus/diff-checkstyle-standalone-metastore.txt
|
| modules | C: ql standalone-metastore U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11040/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> IHMSHandler.getTable not always fetching the right catalog
> --
>
> Key: HIVE-19576
> URL: https://issues.apache.org/jira/browse/HIVE-19576
> Project: Hive
> Issue Type: Bug
> Components: Metastore
>Affects Versions: 3.0.0
>Reporter: Alan Gates
>Assignee: Alan Gates
>Priority: Major
> Fix For: 3.0.1
>
> Attachments: HIVE-19576.1take2.patch, HIVE-19576.patch
>
>
> {{IHMSHandler.get_table_core(String dbName, String tableName)}} fetches the
> catalog name from the conf. This causes issues when doing an operation where
> the catalog is known and does not match the default provided in the
> configuration file (e.g. adding a partition). This method should be removed
> and callers forced to use {{IHMSHandler.get_table_core(String catName, String
> dbName, String tableName)}} instead since callers will know whether they have
> the catalog name or not.
--
[jira] [Commented] (HIVE-19491) Branch-3 Start using storage-api 2.6.1 once available.
[ https://issues.apache.org/jira/browse/HIVE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480106#comment-16480106 ] Vineet Garg commented on HIVE-19491: Thanks for pointing this out [~yuchaoran2011]. storage-api/pom.xml change is not required since hive release will not be building storage-api module. But standalone-metastore/pom.xml does need to update the reference. I'll fix this. > Branch-3 Start using storage-api 2.6.1 once available. > -- > > Key: HIVE-19491 > URL: https://issues.apache.org/jira/browse/HIVE-19491 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Vineet Garg >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HIVE-19491.01-branch-3.patch > > > branch-3 needs storage-api 2.6.1 which is in the process of being released. > > cc. [~vgarg] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19602) Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor instance
[ https://issues.apache.org/jira/browse/HIVE-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-19602: Summary: Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor instance (was: Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor) > Refactor inplace progress code in Hive-on-spark progress monitor to use > ProgressMonitor instance > > > Key: HIVE-19602 > URL: https://issues.apache.org/jira/browse/HIVE-19602 > Project: Hive > Issue Type: Bug >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > We can refactor the HOS inplace progress monitor code > (SparkJobMonitor#printStatusInPlace) to use InplaceUpdate#render. > This would be similar to : > [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19602) Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor
[ https://issues.apache.org/jira/browse/HIVE-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-19602: Description: We can refactor the HOS inplace progress monitor code (SparkJobMonitor#printStatusInPlace) to use InplaceUpdate#render. This would be similar to : [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] was: We can refactor the HOS inplace progress monitor code to use InplaceUpdate#render. This would be similar to : [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] > Refactor inplace progress code in Hive-on-spark progress monitor to use > ProgressMonitor > --- > > Key: HIVE-19602 > URL: https://issues.apache.org/jira/browse/HIVE-19602 > Project: Hive > Issue Type: Bug >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > We can refactor the HOS inplace progress monitor code > (SparkJobMonitor#printStatusInPlace) to use InplaceUpdate#render. > This would be similar to : > [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19602) Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor
[ https://issues.apache.org/jira/browse/HIVE-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480102#comment-16480102 ] Bharathkrishna Guruvayoor Murali commented on HIVE-19602: - Idea is to create a ProgressMonitor instance in SparkJobMonitor#printStatusInPlace and display it using InplaceUpdate#render. This would help in HIVE-19176 as here we need to pass an instance of ProgressMonitor in the response to beeline, so we can maintain uniform format in both cases. [~stakiar] can you validate if this is a reasonable change. > Refactor inplace progress code in Hive-on-spark progress monitor to use > ProgressMonitor > --- > > Key: HIVE-19602 > URL: https://issues.apache.org/jira/browse/HIVE-19602 > Project: Hive > Issue Type: Bug >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > We can refactor the HOS inplace progress monitor code to use > InplaceUpdate#render. > This would be similar to : > [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19602) Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor
[ https://issues.apache.org/jira/browse/HIVE-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-19602: Description: We can refactor the HOS inplace progress monitor code to use InplaceUpdate#render. This would be similar to : [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] was: We can refactor the HOS inplace progress monitor code to use InplaceUpdate. This would be similar to : [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] > Refactor inplace progress code in Hive-on-spark progress monitor to use > ProgressMonitor > --- > > Key: HIVE-19602 > URL: https://issues.apache.org/jira/browse/HIVE-19602 > Project: Hive > Issue Type: Bug >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > We can refactor the HOS inplace progress monitor code to use > InplaceUpdate#render. > This would be similar to : > [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19602) Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor
[ https://issues.apache.org/jira/browse/HIVE-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali updated HIVE-19602: Summary: Refactor inplace progress code in Hive-on-spark progress monitor to use ProgressMonitor (was: Refactor inplace progress code in Hive-on-spark progress monitor to use InplaceUpdate) > Refactor inplace progress code in Hive-on-spark progress monitor to use > ProgressMonitor > --- > > Key: HIVE-19602 > URL: https://issues.apache.org/jira/browse/HIVE-19602 > Project: Hive > Issue Type: Bug >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > We can refactor the HOS inplace progress monitor code to use InplaceUpdate. > This would be similar to : > [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19602) Refactor inplace progress code in Hive-on-spark progress monitor to use InplaceUpdate
[ https://issues.apache.org/jira/browse/HIVE-19602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharathkrishna Guruvayoor Murali reassigned HIVE-19602: --- > Refactor inplace progress code in Hive-on-spark progress monitor to use > InplaceUpdate > - > > Key: HIVE-19602 > URL: https://issues.apache.org/jira/browse/HIVE-19602 > Project: Hive > Issue Type: Bug >Reporter: Bharathkrishna Guruvayoor Murali >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > > We can refactor the HOS inplace progress monitor code to use InplaceUpdate. > This would be similar to : > [https://github.com/apache/hive/blob/0b6bea89f74b607299ad944b37e4b62c711aaa69/ql/src/java/org/apache/hadoop/hive/ql/exec/tez/monitoring/RenderStrategy.java#L181] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18748) Rename table impacts the ACID behavior as table names are not updated in meta-tables.
[
https://issues.apache.org/jira/browse/HIVE-18748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480084#comment-16480084
]
Hive QA commented on HIVE-18748:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923949/HIVE-18748.02-branch-3.patch
{color:red}ERROR:{color} -1 due to build exiting with an error
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11039/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11039/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11039/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Tests exited with: NonZeroExitCodeException
Command 'bash /data/hiveptest/working/scratch/source-prep.sh' failed with exit
status 1 and output '+ date '+%Y-%m-%d %T.%3N'
2018-05-18 03:21:51.772
+ [[ -n /usr/lib/jvm/java-8-openjdk-amd64 ]]
+ export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
+ export
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+
PATH=/usr/lib/jvm/java-8-openjdk-amd64/bin/:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games
+ export 'ANT_OPTS=-Xmx1g -XX:MaxPermSize=256m '
+ ANT_OPTS='-Xmx1g -XX:MaxPermSize=256m '
+ export 'MAVEN_OPTS=-Xmx1g '
+ MAVEN_OPTS='-Xmx1g '
+ cd /data/hiveptest/working/
+ tee /data/hiveptest/logs/PreCommit-HIVE-Build-11039/source-prep.txt
+ [[ false == \t\r\u\e ]]
+ mkdir -p maven ivy
+ [[ git = \s\v\n ]]
+ [[ git = \g\i\t ]]
+ [[ -z branch-3 ]]
+ [[ -d apache-github-source-source ]]
+ [[ ! -d apache-github-source-source/.git ]]
+ [[ ! -d apache-github-source-source ]]
+ date '+%Y-%m-%d %T.%3N'
2018-05-18 03:21:51.775
+ cd apache-github-source-source
+ git fetch origin
+ git reset --hard HEAD
HEAD is now at 0b6bea8 HIVE-19572: Add option to mask stats and data size in q
files (Jesus Camacho Rodriguez, reviewed by Prasanth Jayachandran) (addendum)
+ git clean -f -d
+ git checkout branch-3
Switched to branch 'branch-3'
Your branch is up-to-date with 'origin/branch-3'.
+ git reset --hard origin/branch-3
HEAD is now at 9cd6258 HIVE-19317 : Handle schema evolution from int like types
to decimal (Janaki Lahorani, reviewed by Vihang Karajgaonkar)
+ git merge --ff-only origin/branch-3
Already up-to-date.
+ date '+%Y-%m-%d %T.%3N'
2018-05-18 03:21:53.555
+ rm -rf ../yetus_PreCommit-HIVE-Build-11039
+ mkdir ../yetus_PreCommit-HIVE-Build-11039
+ git gc
+ cp -R . ../yetus_PreCommit-HIVE-Build-11039
+ mkdir /data/hiveptest/logs/PreCommit-HIVE-Build-11039/yetus
+ patchCommandPath=/data/hiveptest/working/scratch/smart-apply-patch.sh
+ patchFilePath=/data/hiveptest/working/scratch/build.patch
+ [[ -f /data/hiveptest/working/scratch/build.patch ]]
+ chmod +x /data/hiveptest/working/scratch/smart-apply-patch.sh
+ /data/hiveptest/working/scratch/smart-apply-patch.sh
/data/hiveptest/working/scratch/build.patch
Going to apply patch with: git apply -p0
+ [[ maven == \m\a\v\e\n ]]
+ rm -rf /data/hiveptest/working/maven/org/apache/hive
+ mvn -B clean install -DskipTests -T 4 -q
-Dmaven.repo.local=/data/hiveptest/working/maven
[ERROR] Failed to execute goal on project hive-standalone-metastore: Could not
resolve dependencies for project
org.apache.hive:hive-standalone-metastore:jar:3.0.0: Could not find artifact
org.apache.hive:hive-storage-api:jar:2.6.0-SNAPSHOT in apache.snapshots
(https://repository.apache.org/snapshots) -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/DependencyResolutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn -rf :hive-standalone-metastore
+ exit 1
'
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923949 - PreCommit-HIVE-Build
> Rename table impacts the ACID behavior as table names are not updated in
> meta-tables.
> -
>
> Key: HIVE-18748
> URL: https://issues.apache.org/jira/browse/HIVE-18748
> Project: Hive
> Issue Type: Sub-task
> Components: HiveServer2, Transactions
>Affects Versions: 3.0.0
>Reporter: Sankar Hariappan
>Assignee: Eugene Koifman
>Priority: Critical
> Labels: ACID, DDL
> Attachments: HIVE-18748.01-branch-3.patch,
> HIVE-18748.02-branch-3.patch, HIVE-18748.02.patch, HIVE-18748.03.patch,
> HIVE-18748.04.patch, HIVE-18748.05.patch
>
>
> ACID
[jira] [Commented] (HIVE-19421) Upgrade version of Jetty to 9.3.20.v20170531
[
https://issues.apache.org/jira/browse/HIVE-19421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480082#comment-16480082
]
Hive QA commented on HIVE-19421:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923937/HIVE-19421.4.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14398 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[multi_insert_move_tasks_share_dependencies]
(batchId=57)
org.apache.hadoop.hive.cli.TestTezPerfCliDriver.testCliDriver[query64]
(batchId=253)
org.apache.hadoop.hive.ql.TestTxnNoBuckets.testToAcidConversion02 (batchId=286)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11038/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11038/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11038/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923937 - PreCommit-HIVE-Build
> Upgrade version of Jetty to 9.3.20.v20170531
>
>
> Key: HIVE-19421
> URL: https://issues.apache.org/jira/browse/HIVE-19421
> Project: Hive
> Issue Type: Improvement
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-19421.1.patch, HIVE-19421.2.patch,
> HIVE-19421.3.patch, HIVE-19421.3.patch, HIVE-19421.3.patch,
> HIVE-19421.4.patch, HIVE-19421.4.patch, HIVE-19421.4.patch
>
>
> Move Jetty up to 9.3.20.v20170531
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19421) Upgrade version of Jetty to 9.3.20.v20170531
[
https://issues.apache.org/jira/browse/HIVE-19421?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480057#comment-16480057
]
Hive QA commented on HIVE-19421:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
0s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
55s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
33s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
55s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 40m 58s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile
xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11038/dev-support/hive-personality.sh
|
| git revision | master / 0b6bea8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| modules | C: . itests/hive-unit U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11038/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Upgrade version of Jetty to 9.3.20.v20170531
>
>
> Key: HIVE-19421
> URL: https://issues.apache.org/jira/browse/HIVE-19421
> Project: Hive
> Issue Type: Improvement
>Reporter: Janaki Lahorani
>Assignee: Janaki Lahorani
>Priority: Major
> Attachments: HIVE-19421.1.patch, HIVE-19421.2.patch,
> HIVE-19421.3.patch, HIVE-19421.3.patch, HIVE-19421.3.patch,
> HIVE-19421.4.patch, HIVE-19421.4.patch, HIVE-19421.4.patch
>
>
> Move Jetty up to 9.3.20.v20170531
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19308) Provide an Arrow stream reader for external LLAP clients
[
https://issues.apache.org/jira/browse/HIVE-19308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480042#comment-16480042
]
Eric Wohlstadter commented on HIVE-19308:
-
[~jdere]
For the current test failure ... interestingly:
1. All the tests pass for both Row and Arrow, except when
{{testNonAsciiStrings}} is run for the second test.
2. It doesn't matter whether Row or Arrow runs first. {{testNonAsciiStrings}}
will fail for the second.
3. There doesn't seem to be anything special about {{testNonAsciiStrings}},
e.g. it isn't the first test case run, it reuses the same create table function
as the other test cases, etc.
I'll keep digging. If you think of anything special about
{{testNonAsciiStrings}} that I might be missing, please let me know.
> Provide an Arrow stream reader for external LLAP clients
> -
>
> Key: HIVE-19308
> URL: https://issues.apache.org/jira/browse/HIVE-19308
> Project: Hive
> Issue Type: Task
> Components: llap
>Reporter: Eric Wohlstadter
>Assignee: Eric Wohlstadter
>Priority: Major
> Attachments: HIVE-19308.1.patch, HIVE-19308.2.patch,
> HIVE-19308.3.patch, HIVE-19308.4.patch, HIVE-19308.5.patch
>
>
> This is a sub-class of LlapBaseRecordReader that wraps the socket inputStream
> and produces Arrow batches for an external client.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[ https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480028#comment-16480028 ] Bharathkrishna Guruvayoor Murali commented on HIVE-14388: - TestHCatLoaderComplexSchema: This test failure did not happen when I tried on my machine but some other tests in this set failed, but it is noted in HIVE-15404. TestRCFileHCatStorer : ran successfully on local machine. org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby8] - Ran successfully on local machine. Hence, failures look unrelated. > Add number of rows inserted message after insert command in Beeline > --- > > Key: HIVE-14388 > URL: https://issues.apache.org/jira/browse/HIVE-14388 > Project: Hive > Issue Type: Improvement > Components: Beeline >Reporter: Vihang Karajgaonkar >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Minor > Attachments: HIVE-14388-WIP.patch, HIVE-14388.02.patch, > HIVE-14388.03.patch, HIVE-14388.05.patch, HIVE-14388.06.patch, > HIVE-14388.07.patch, HIVE-14388.08.patch, HIVE-14388.09.patch, > HIVE-14388.10.patch, HIVE-14388.12.patch > > > Currently, when you run insert command on beeline, it returns a message > saying "No rows affected .." > A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19562) Flaky test: TestMiniSparkOnYarn FileNotFoundException in spark-submit
[
https://issues.apache.org/jira/browse/HIVE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16480014#comment-16480014
]
Hive QA commented on HIVE-19562:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923932/HIVE-19562.4.patch
{color:red}ERROR:{color} -1 due to no test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 3 failed/errored test(s), 14398 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_smb]
(batchId=176)
org.apache.hive.service.server.TestInformationSchemaWithPrivilege.test
(batchId=238)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11037/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11037/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11037/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 3 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923932 - PreCommit-HIVE-Build
> Flaky test: TestMiniSparkOnYarn FileNotFoundException in spark-submit
> -
>
> Key: HIVE-19562
> URL: https://issues.apache.org/jira/browse/HIVE-19562
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19562.1.patch, HIVE-19562.3.patch,
> HIVE-19562.4.patch
>
>
> Seeing sporadic failures during test setup. Specifically, when spark-submit
> runs this error (or a similar error) gets thrown:
> {code}
> 2018-05-15T10:55:02,112 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: Exception in thread "main"
> java.io.FileNotFoundException: File
> file:/tmp/spark-56e217f7-b8a5-4c63-9a6b-d737a64f2820/__spark_libs__7371510645900072447.zip
> does not exist
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:356)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:478)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:565)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
>
[jira] [Updated] (HIVE-19584) Dictionary encoding for string types
[ https://issues.apache.org/jira/browse/HIVE-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Teddy Choi updated HIVE-19584: -- Description: Apache Arrow supports dictionary encoding for some data types. So implement dictionary encoding for string types in Arrow SerDe. > Dictionary encoding for string types > > > Key: HIVE-19584 > URL: https://issues.apache.org/jira/browse/HIVE-19584 > Project: Hive > Issue Type: Sub-task >Reporter: Teddy Choi >Assignee: Teddy Choi >Priority: Major > Attachments: HIVE-19584.1.patch > > > Apache Arrow supports dictionary encoding for some data types. So implement > dictionary encoding for string types in Arrow SerDe. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19370) Issue: ADD Months function on timestamp datatype fields in hive
[ https://issues.apache.org/jira/browse/HIVE-19370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479990#comment-16479990 ] Bharathkrishna Guruvayoor Murali commented on HIVE-19370: - Test localtimezone.q ran in local machine successfully using TestCliDriver. The test failure looks unrelated. > Issue: ADD Months function on timestamp datatype fields in hive > --- > > Key: HIVE-19370 > URL: https://issues.apache.org/jira/browse/HIVE-19370 > Project: Hive > Issue Type: Bug >Reporter: Amit Chauhan >Assignee: Bharathkrishna Guruvayoor Murali >Priority: Major > Attachments: HIVE-19370.01.patch, HIVE-19370.02.patch, > HIVE-19370.03.patch > > > *Issue:* > while using ADD_Months function on a timestamp datatype column the output > omits the time part[HH:MM:SS] part from output. > which should not be the case. > *query:* EMAIL_FAILURE_DTMZ is of datatype timestamp in hive. > hive> select CUSTOMER_ID,EMAIL_FAILURE_DTMZ,ADD_MONTHS (EMAIL_FAILURE_DTMZ , > 1) from TABLE1 where CUSTOMER_ID=125674937; > OK > 125674937 2015-12-09 12:25:53 2016-01-09 > *hiver version :* > hive> !hive --version; > Hive 1.2.1000.2.5.6.0-40 > > can you please help if somehow I can get below as output: > > 125674937 2015-12-09 12:25:53 2016-01-09 12:25:53 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19562) Flaky test: TestMiniSparkOnYarn FileNotFoundException in spark-submit
[
https://issues.apache.org/jira/browse/HIVE-19562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479965#comment-16479965
]
Hive QA commented on HIVE-19562:
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
13s{color} | {color:green} The patch has no ill-formed XML file. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
54s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 1m 54s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense xml |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11037/dev-support/hive-personality.sh
|
| git revision | master / 0b6bea8 |
| modules | C: . U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11037/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Flaky test: TestMiniSparkOnYarn FileNotFoundException in spark-submit
> -
>
> Key: HIVE-19562
> URL: https://issues.apache.org/jira/browse/HIVE-19562
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-19562.1.patch, HIVE-19562.3.patch,
> HIVE-19562.4.patch
>
>
> Seeing sporadic failures during test setup. Specifically, when spark-submit
> runs this error (or a similar error) gets thrown:
> {code}
> 2018-05-15T10:55:02,112 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: Exception in thread "main"
> java.io.FileNotFoundException: File
> file:/tmp/spark-56e217f7-b8a5-4c63-9a6b-d737a64f2820/__spark_libs__7371510645900072447.zip
> does not exist
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:641)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:867)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:631)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:442)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:365)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:316)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.copyFileToRemote(Client.scala:356)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.org$apache$spark$deploy$yarn$Client$$distribute$1(Client.scala:478)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5 main]
> client.SparkSubmitSparkClient: at
> org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:565)
> 2018-05-15T10:55:02,113 INFO
> [RemoteDriver-stderr-redir-27d3dcfb-2a10-4118-9fae-c200d2e095a5
[jira] [Commented] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[
https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479959#comment-16479959
]
Hive QA commented on HIVE-14388:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923931/HIVE-14388.12.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 4 failed/errored test(s), 14398 tests
executed
*Failed tests:*
{noformat}
TestMinimrCliDriver - did not produce a TEST-*.xml file (likely timed out)
(batchId=94)
[infer_bucket_sort_num_buckets.q,infer_bucket_sort_reducers_power_two.q,parallel_orderby.q,bucket_num_reducers_acid.q,scriptfile1.q,infer_bucket_sort_map_operators.q,infer_bucket_sort_merge.q,root_dir_external_table.q,infer_bucket_sort_dyn_part.q,udf_using.q]
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[groupby8] (batchId=77)
org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema.testSyntheticComplexSchema[4]
(batchId=196)
org.apache.hive.hcatalog.pig.TestRCFileHCatStorer.testWriteDecimalXY
(batchId=196)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11036/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11036/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11036/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 4 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923931 - PreCommit-HIVE-Build
> Add number of rows inserted message after insert command in Beeline
> ---
>
> Key: HIVE-14388
> URL: https://issues.apache.org/jira/browse/HIVE-14388
> Project: Hive
> Issue Type: Improvement
> Components: Beeline
>Reporter: Vihang Karajgaonkar
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Minor
> Attachments: HIVE-14388-WIP.patch, HIVE-14388.02.patch,
> HIVE-14388.03.patch, HIVE-14388.05.patch, HIVE-14388.06.patch,
> HIVE-14388.07.patch, HIVE-14388.08.patch, HIVE-14388.09.patch,
> HIVE-14388.10.patch, HIVE-14388.12.patch
>
>
> Currently, when you run insert command on beeline, it returns a message
> saying "No rows affected .."
> A better and more intuitive msg would be "xxx rows inserted (26.068 seconds)"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19491) Branch-3 Start using storage-api 2.6.1 once available.
[ https://issues.apache.org/jira/browse/HIVE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479940#comment-16479940 ] Chaoran Yu commented on HIVE-19491: --- This issue is unresolved yet. There are two places that still refer to 2.6.0-SNAPSHOT: [https://github.com/apache/hive/blob/branch-3/storage-api/pom.xml#L28] [https://github.com/apache/hive/blob/branch-3/standalone-metastore/pom.xml#L86] They are causing the build to fail: [https://travis-ci.org/apache/hive/builds/379450173?utm_source=github_status_medium=notification] > Branch-3 Start using storage-api 2.6.1 once available. > -- > > Key: HIVE-19491 > URL: https://issues.apache.org/jira/browse/HIVE-19491 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Vineet Garg >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HIVE-19491.01-branch-3.patch > > > branch-3 needs storage-api 2.6.1 which is in the process of being released. > > cc. [~vgarg] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (HIVE-19491) Branch-3 Start using storage-api 2.6.1 once available.
[ https://issues.apache.org/jira/browse/HIVE-19491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chaoran Yu reopened HIVE-19491: --- > Branch-3 Start using storage-api 2.6.1 once available. > -- > > Key: HIVE-19491 > URL: https://issues.apache.org/jira/browse/HIVE-19491 > Project: Hive > Issue Type: Task >Reporter: Deepak Jaiswal >Assignee: Vineet Garg >Priority: Blocker > Fix For: 3.0.0 > > Attachments: HIVE-19491.01-branch-3.patch > > > branch-3 needs storage-api 2.6.1 which is in the process of being released. > > cc. [~vgarg] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19586) Optimize Count(distinct X) pushdown based on the storage capabilities
[
https://issues.apache.org/jira/browse/HIVE-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19586:
--
Attachment: HIVE-19586.2.patch
> Optimize Count(distinct X) pushdown based on the storage capabilities
> --
>
> Key: HIVE-19586
> URL: https://issues.apache.org/jira/browse/HIVE-19586
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration, Logical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19586.2.patch, HIVE-19586.patch
>
>
> h1. Goal
> Provide a way to rewrite queries with combination of COUNT(Distinct) and
> Aggregates like SUM as a series of Group By.
> This can be useful to push down to Druid queries like
> {code}
> select count(DISTINCT interval_marker), count (distinct dim), sum(num_l)
> FROM druid_test_table GROUP BY `__time`, `zone` ;
> {code}
> In general this can be useful to be used in cases where storage handlers can
> not perform count (distinct column)
> h1. How to do it.
> Use the Calcite rule {code}
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule{code} that
> breaks down Count distinct to a single Group by with Grouping sets or
> multiple series of Group by that might be linked with Joins if multiple
> counts are present.
> FYI today Hive does have a similar rule {code}
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule{code},
> but it only provides a rewrite to Grouping sets based plan.
> I am planing to use the actual Calcite rule, [~ashutoshc] any concerns or
> caveats to be aware of?
> h2. Concerns/questions
> Need to have a way to switch between Grouping sets or Simple chained group by
> based on the plan cost. For instance for Druid based scan it makes always
> sense (at least today) to push down a series of Group by and stitch result
> sets in Hive later (as oppose to scan everything).
> But this might be not true for other storage handler that can handle Grouping
> sets it is better to push down the Grouping sets as one table scan.
> Am still unsure how i can lean on the cost optimizer to select the best plan,
> [~ashutoshc]/[~jcamachorodriguez] any inputs?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19586) Optimize Count(distinct X) pushdown based on the storage capabilities
[
https://issues.apache.org/jira/browse/HIVE-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479891#comment-16479891
]
slim bouguerra commented on HIVE-19586:
---
After Talking with [~ashutoshc] Offline we decided to tackle the simple case
first.
This patch will allow us to expand an aggregate with single count distinct and
valid Druid aggregates as a series of Group by. Thus this will lead to a plan
where the first GB is executed by Druid while the rest will be done in HIVE.
The next step will be to fix the couple of related issue and enable push down
of Grouping set down to Druid once it is supported.
> Optimize Count(distinct X) pushdown based on the storage capabilities
> --
>
> Key: HIVE-19586
> URL: https://issues.apache.org/jira/browse/HIVE-19586
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration, Logical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19586.patch
>
>
> h1. Goal
> Provide a way to rewrite queries with combination of COUNT(Distinct) and
> Aggregates like SUM as a series of Group By.
> This can be useful to push down to Druid queries like
> {code}
> select count(DISTINCT interval_marker), count (distinct dim), sum(num_l)
> FROM druid_test_table GROUP BY `__time`, `zone` ;
> {code}
> In general this can be useful to be used in cases where storage handlers can
> not perform count (distinct column)
> h1. How to do it.
> Use the Calcite rule {code}
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule{code} that
> breaks down Count distinct to a single Group by with Grouping sets or
> multiple series of Group by that might be linked with Joins if multiple
> counts are present.
> FYI today Hive does have a similar rule {code}
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule{code},
> but it only provides a rewrite to Grouping sets based plan.
> I am planing to use the actual Calcite rule, [~ashutoshc] any concerns or
> caveats to be aware of?
> h2. Concerns/questions
> Need to have a way to switch between Grouping sets or Simple chained group by
> based on the plan cost. For instance for Druid based scan it makes always
> sense (at least today) to push down a series of Group by and stitch result
> sets in Hive later (as oppose to scan everything).
> But this might be not true for other storage handler that can handle Grouping
> sets it is better to push down the Grouping sets as one table scan.
> Am still unsure how i can lean on the cost optimizer to select the best plan,
> [~ashutoshc]/[~jcamachorodriguez] any inputs?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19586) Optimize Count(distinct X) pushdown based on the storage capabilities
[
https://issues.apache.org/jira/browse/HIVE-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19586:
--
Status: Patch Available (was: Open)
> Optimize Count(distinct X) pushdown based on the storage capabilities
> --
>
> Key: HIVE-19586
> URL: https://issues.apache.org/jira/browse/HIVE-19586
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration, Logical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19586.patch
>
>
> h1. Goal
> Provide a way to rewrite queries with combination of COUNT(Distinct) and
> Aggregates like SUM as a series of Group By.
> This can be useful to push down to Druid queries like
> {code}
> select count(DISTINCT interval_marker), count (distinct dim), sum(num_l)
> FROM druid_test_table GROUP BY `__time`, `zone` ;
> {code}
> In general this can be useful to be used in cases where storage handlers can
> not perform count (distinct column)
> h1. How to do it.
> Use the Calcite rule {code}
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule{code} that
> breaks down Count distinct to a single Group by with Grouping sets or
> multiple series of Group by that might be linked with Joins if multiple
> counts are present.
> FYI today Hive does have a similar rule {code}
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule{code},
> but it only provides a rewrite to Grouping sets based plan.
> I am planing to use the actual Calcite rule, [~ashutoshc] any concerns or
> caveats to be aware of?
> h2. Concerns/questions
> Need to have a way to switch between Grouping sets or Simple chained group by
> based on the plan cost. For instance for Druid based scan it makes always
> sense (at least today) to push down a series of Group by and stitch result
> sets in Hive later (as oppose to scan everything).
> But this might be not true for other storage handler that can handle Grouping
> sets it is better to push down the Grouping sets as one table scan.
> Am still unsure how i can lean on the cost optimizer to select the best plan,
> [~ashutoshc]/[~jcamachorodriguez] any inputs?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19586) Optimize Count(distinct X) pushdown based on the storage capabilities
[
https://issues.apache.org/jira/browse/HIVE-19586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19586:
--
Attachment: HIVE-19586.patch
> Optimize Count(distinct X) pushdown based on the storage capabilities
> --
>
> Key: HIVE-19586
> URL: https://issues.apache.org/jira/browse/HIVE-19586
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration, Logical Optimizer
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Fix For: 3.0.0
>
> Attachments: HIVE-19586.patch
>
>
> h1. Goal
> Provide a way to rewrite queries with combination of COUNT(Distinct) and
> Aggregates like SUM as a series of Group By.
> This can be useful to push down to Druid queries like
> {code}
> select count(DISTINCT interval_marker), count (distinct dim), sum(num_l)
> FROM druid_test_table GROUP BY `__time`, `zone` ;
> {code}
> In general this can be useful to be used in cases where storage handlers can
> not perform count (distinct column)
> h1. How to do it.
> Use the Calcite rule {code}
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule{code} that
> breaks down Count distinct to a single Group by with Grouping sets or
> multiple series of Group by that might be linked with Joins if multiple
> counts are present.
> FYI today Hive does have a similar rule {code}
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule{code},
> but it only provides a rewrite to Grouping sets based plan.
> I am planing to use the actual Calcite rule, [~ashutoshc] any concerns or
> caveats to be aware of?
> h2. Concerns/questions
> Need to have a way to switch between Grouping sets or Simple chained group by
> based on the plan cost. For instance for Druid based scan it makes always
> sense (at least today) to push down a series of Group by and stitch result
> sets in Hive later (as oppose to scan everything).
> But this might be not true for other storage handler that can handle Grouping
> sets it is better to push down the Grouping sets as one table scan.
> Am still unsure how i can lean on the cost optimizer to select the best plan,
> [~ashutoshc]/[~jcamachorodriguez] any inputs?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-14388) Add number of rows inserted message after insert command in Beeline
[
https://issues.apache.org/jira/browse/HIVE-14388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479881#comment-16479881
]
Hive QA commented on HIVE-14388:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
36s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
45s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
35s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
30s{color} | {color:blue} beeline in master has 69 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
34s{color} | {color:blue} itests/hive-unit in master has 2 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
25s{color} | {color:blue} jdbc in master has 17 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
58s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
37s{color} | {color:blue} service in master has 50 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
5s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
6s{color} | {color:green} The patch service-rpc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch beeline passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} The patch hive-unit passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
11s{color} | {color:green} The patch jdbc passed checkstyle {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} ql: The patch generated 0 new + 206 unchanged - 13
fixed = 206 total (was 219) {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} The patch service passed checkstyle {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m
0s{color} | {color:red} The patch has 2 line(s) that end in whitespace. Use git
apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply
{color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
6s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
11s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 38m 4s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11036/dev-support/hive-personality.sh
|
| git revision | master / 0b6bea8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| whitespace |
[jira] [Commented] (HIVE-19528) Beeline: When beeline-site.xml is present and the default named url is incorrect, throw an exception instead of relying on resolution via hive-site.xml/beeline-hs2-conn
[ https://issues.apache.org/jira/browse/HIVE-19528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479847#comment-16479847 ] Thejas M Nair commented on HIVE-19528: -- bq. miniHS2.getBaseJdbcURL() + "default;user=hive;password=hive;transportMode=http;httpPath=cliservice"; This could use the constants used to set configs in HS2 startup (HS2_HTTP_ENDPOINT, HS2_HTTP_MODE). Can you add a negative test also, since this test is improving the error handling ? > Beeline: When beeline-site.xml is present and the default named url is > incorrect, throw an exception instead of relying on resolution via > hive-site.xml/beeline-hs2-connection.xml > --- > > Key: HIVE-19528 > URL: https://issues.apache.org/jira/browse/HIVE-19528 > Project: Hive > Issue Type: Bug > Components: Beeline >Affects Versions: 3.0.0, 3.1.0 >Reporter: Vaibhav Gumashta >Assignee: Vaibhav Gumashta >Priority: Major > Attachments: HIVE-19528.1.patch, HIVE-19528.2.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19584) Dictionary encoding for string types
[
https://issues.apache.org/jira/browse/HIVE-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479842#comment-16479842
]
Hive QA commented on HIVE-19584:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923929/HIVE-19584.1.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14409 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11035/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11035/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11035/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923929 - PreCommit-HIVE-Build
> Dictionary encoding for string types
>
>
> Key: HIVE-19584
> URL: https://issues.apache.org/jira/browse/HIVE-19584
> Project: Hive
> Issue Type: Sub-task
>Reporter: Teddy Choi
>Assignee: Teddy Choi
>Priority: Major
> Attachments: HIVE-19584.1.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19441) Add support for float aggregator and use LLAP test Driver
[
https://issues.apache.org/jira/browse/HIVE-19441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19441:
--
Attachment: HIVE-19441.3.patch
> Add support for float aggregator and use LLAP test Driver
> -
>
> Key: HIVE-19441
> URL: https://issues.apache.org/jira/browse/HIVE-19441
> Project: Hive
> Issue Type: Improvement
> Components: Druid integration
>Reporter: slim bouguerra
>Assignee: slim bouguerra
>Priority: Major
> Attachments: HIVE-19441.2.patch, HIVE-19441.3.patch, HIVE-19441.patch
>
>
> Adding support to the float kind aggregator.
> Use LLAP as test Driver to reduce execution time of tests from about 2 hours
> to 15 min:
> Although this patches unveiling an issue with timezone, maybe it is fixed by
> [~jcamachorodriguez] upcoming set of patches.
>
> Before
> {code}
> [INFO] Executed tasks
> [INFO]
> [INFO] --- maven-compiler-plugin:3.6.1:testCompile (default-testCompile) @
> hive-it-qfile ---
> [INFO] Compiling 21 source files to
> /Users/sbouguerra/Hdev/hive/itests/qtest/target/test-classes
> [INFO]
> [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ hive-it-qfile
> ---
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 6,654.117 s - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 01:51 h
> [INFO] Finished at: 2018-05-04T12:43:19-07:00
> [INFO]
>
> {code}
> After
> {code}
> INFO] Executed tasks
> [INFO]
> [INFO] --- maven-compiler-plugin:3.6.1:testCompile (default-testCompile) @
> hive-it-qfile ---
> [INFO] Compiling 22 source files to
> /Users/sbouguerra/Hdev/hive/itests/qtest/target/test-classes
> [INFO]
> [INFO] --- maven-surefire-plugin:2.21.0:test (default-test) @ hive-it-qfile
> ---
> [INFO]
> [INFO] ---
> [INFO] T E S T S
> [INFO] ---
> [INFO] Running org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed:
> 907.167 s - in org.apache.hadoop.hive.cli.TestMiniDruidCliDriver
> [INFO]
> [INFO] Results:
> [INFO]
> [INFO] Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
> [INFO]
> [INFO]
>
> [INFO] BUILD SUCCESS
> [INFO]
>
> [INFO] Total time: 15:31 min
> [INFO] Finished at: 2018-05-04T13:15:11-07:00
> [INFO]
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19600) Hive and Calcite have different semantics for Grouping sets
[
https://issues.apache.org/jira/browse/HIVE-19600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479830#comment-16479830
]
slim bouguerra commented on HIVE-19600:
---
DDLs
{code}
create table test_base_table(`timecolumn` timestamp, `interval_marker` string,
`num_l` double, `zone` string, `dim` string);
-- failed with 2018-05-17T08:42:01,365 ERROR
[459c864e-5fab-40dd-b848-00f5f0432697 main] ql.Driver: FAILED:
SemanticException Line 0:-1 Argument type mismatch 'timecolumn': The first
argument to grouping() must be an int/long. Got: TIMESTAMP
-- org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Argument type
mismatch 'timecolumn': The first argument to grouping() must be an int/long.
Got: TIMESTAMP
EXPLAIN select count(DISTINCT interval_marker), count(DISTINCT dim) FROM
test_base_table GROUP BY `timecolumn` ;
{code}
> Hive and Calcite have different semantics for Grouping sets
> ---
>
> Key: HIVE-19600
> URL: https://issues.apache.org/jira/browse/HIVE-19600
> Project: Hive
> Issue Type: Sub-task
>Reporter: slim bouguerra
>Priority: Major
> Fix For: 3.1.0
>
>
> h1. Issue:
> Tried to use the calcite rule {code}
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule#AggregateExpandDistinctAggregatesRule(java.lang.Class extends org.apache.calcite.rel.core.Aggregate>, boolean,
> org.apache.calcite.tools.RelBuilderFactory) {code} to replace current rule
> used by Hive {code}
> org.apache.hadoop.hive.ql.optimizer.calcite.rules.HiveExpandDistinctAggregatesRule#HiveExpandDistinctAggregatesRule
> {code}
> But i got an exception when generating the Operator tree out of calcite plan.
> This is the Calcite plan
> {code}
> HiveProject.HIVE.[](input=rel#50:HiveAggregate.HIVE.[](input=rel#48:HiveProject.HIVE.[](input=rel#44:HiveAggregate.HIVE.[](input=rel#38:HiveProject.HIVE.[](input=rel#0:HiveTableScan.HIVE.[]
> (table=[druid_test_dst.test_base_table],table:alias=test_base_table)[false],$f0=$3,$f1=$1,$f2=$4),group={0,
> 1, 2},groups=[{0, 1}, {0, 2}],$g=GROUPING($0, $1,
> $2)),$f0=$0,$f1=$1,$f2=$2,$g_1==($3, 1),$g_2==($3,
> 2)),group={0},agg#0=count($1) FILTER $3,agg#1=count($2) FILTER
> $4),_o__c0=$1,_o__c1=$2)
> {code}
> This is the exception stack
> {code}
> 2018-05-17T08:46:48,604 ERROR [649a61b0-d8c7-45d8-962d-b1d38397feb4 main]
> ql.Driver: FAILED: SemanticException Line 0:-1 Argument type mismatch 'zone':
> The first argument to grouping() must be an int/long. Got: STRING
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Argument type
> mismatch 'zone': The first argument to grouping() must be an int/long. Got:
> STRING
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1467)
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:89)
> at
> org.apache.hadoop.hive.ql.lib.ExpressionWalker.walk(ExpressionWalker.java:76)
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:120)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:239)
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory.genExprNode(TypeCheckProcFactory.java:185)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genAllExprNodeDesc(SemanticAnalyzer.java:12566)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genExprNodeDesc(SemanticAnalyzer.java:12521)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:4525)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genSelectPlan(SemanticAnalyzer.java:4298)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPostGroupByBodyPlan(SemanticAnalyzer.java:10487)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genBodyPlan(SemanticAnalyzer.java:10426)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11339)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11196)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11223)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.genPlan(SemanticAnalyzer.java:11209)
> at
> org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:517)
> at
> org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:12074)
>
[jira] [Updated] (HIVE-19601) Unsupported Post join function 'IS NOT DISTINCT FROM'
[
https://issues.apache.org/jira/browse/HIVE-19601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
slim bouguerra updated HIVE-19601:
--
Summary: Unsupported Post join function 'IS NOT DISTINCT FROM' (was:
Unsupported Post join function)
> Unsupported Post join function 'IS NOT DISTINCT FROM'
> --
>
> Key: HIVE-19601
> URL: https://issues.apache.org/jira/browse/HIVE-19601
> Project: Hive
> Issue Type: Sub-task
>Reporter: slim bouguerra
>Priority: Major
>
> h1. As part of trying to use the Calcite rule {code}
> org.apache.calcite.rel.rules.AggregateExpandDistinctAggregatesRule#JOIN {code}
> i got the following Calcite plan
> {code}
> 2018-05-17T09:26:02,781 DEBUG [80d6d405-ed78-4f60-bd93-b3e08e424f73 main]
> translator.PlanModifierForASTConv: Final plan after modifier
> HiveProject(_c0=[$1], _c1=[$2])
> HiveProject(zone=[$0], $f1=[$1], $f2=[$3])
> HiveJoin(condition=[IS NOT DISTINCT FROM($0, $2)], joinType=[inner],
> algorithm=[none], cost=[not available])
> HiveProject(zone=[$0], $f1=[$1])
> HiveAggregate(group=[{0}], agg#0=[count($1)])
> HiveProject(zone=[$0], interval_marker=[$1])
> HiveAggregate(group=[{0, 1}])
> HiveProject(zone=[$3], interval_marker=[$1])
> HiveTableScan(table=[[druid_test_dst.test_base_table]],
> table:alias=[test_base_table])
> HiveProject(zone=[$0], $f1=[$1])
> HiveAggregate(group=[{0}], agg#0=[count($1)])
> HiveProject(zone=[$0], dim=[$1])
> HiveAggregate(group=[{0, 1}])
> HiveProject(zone=[$3], dim=[$4])
> HiveTableScan(table=[[druid_test_dst.test_base_table]],
> table:alias=[test_base_table])
> {code}
> I run into this issue
> {code}
> 2018-05-17T09:26:02,876 ERROR [80d6d405-ed78-4f60-bd93-b3e08e424f73 main]
> parse.CalcitePlanner: CBO failed, skipping CBO.
> org.apache.hadoop.hive.ql.parse.SemanticException: Line 0:-1 Invalid function
> 'IS NOT DISTINCT FROM'
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.getXpathOrFuncExprNodeDesc(TypeCheckProcFactory.java:1069)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.parse.TypeCheckProcFactory$DefaultExprProcessor.process(TypeCheckProcFactory.java:1464)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:105)
> ~[hive-exec-3.1.0-SNAPSHOT.jar:3.1.0-SNAPSHOT]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19588) Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader
[ https://issues.apache.org/jira/browse/HIVE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479818#comment-16479818 ] Eugene Koifman commented on HIVE-19588: --- +1 pending tests > Several invocation of file listing when creating > VectorizedOrcAcidRowBatchReader > > > Key: HIVE-19588 > URL: https://issues.apache.org/jira/browse/HIVE-19588 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19588.1.patch, Screen Shot 2018-05-16 at 2.23.25 > PM.png > > > Looks like we are doing file listing several times when creating one instance > of VectorizedOrcAcidRowBatchReader > AcidUtils.parseBaseOrDeltaBucketFilename() does full file listing (when > there are files with bucket_* prefix) just to get a single file out of a path > to figure out if it has ACID schema (as part of HIVE-18190). > There is full file listing where we populate > 1) ColumnizedDeleteEventRegistry > 2) SortMergedDeleteEventRegistry > 3) Twice in computeOffsetAndBucket() > > Attaching profiles which [~gopalv] took while debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laszlo Bodor updated HIVE-17852: Attachment: HIVE-17852.03.patch > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch, > HIVE-17852.03.patch > > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-17852) remove support for list bucketing "stored as directories" in 3.0
[ https://issues.apache.org/jira/browse/HIVE-17852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479819#comment-16479819 ] Laszlo Bodor commented on HIVE-17852: - [~ashutoshc] : uploaded 03.patch > remove support for list bucketing "stored as directories" in 3.0 > > > Key: HIVE-17852 > URL: https://issues.apache.org/jira/browse/HIVE-17852 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Laszlo Bodor >Priority: Major > Attachments: HIVE-17852.01.patch, HIVE-17852.02.patch, > HIVE-17852.03.patch > > > From the email thread: > 1) LB, when stored as directories, adds a lot of low-level complexity to Hive > tables that has to be accounted for in many places in the code where the > files are written or modified - from FSOP to ACID/replication/export. > 2) While working on some FSOP code I noticed that some of that logic is > broken - e.g. the duplicate file removal from tasks, a pretty fundamental > correctness feature in Hive, may be broken. LB also doesn’t appear to be > compatible with e.g. regular bucketing. > 3) The feature hasn’t seen development activity in a while; it also doesn’t > appear to be used a lot. > Keeping with the theme of cleaning up “legacy” code for 3.0, I was proposing > we remove it. > (2) also suggested that, if needed, it might be easier to implement similar > functionality by adding some flexibility to partitions (which LB directories > look like anyway); that would also keep the logic on a higher level of > abstraction (split generation, partition pruning) as opposed to many > low-level places like FSOP, etc. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19588) Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader
[ https://issues.apache.org/jira/browse/HIVE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479793#comment-16479793 ] Prasanth Jayachandran commented on HIVE-19588: -- [~ekoifman] can you please review the patch? > Several invocation of file listing when creating > VectorizedOrcAcidRowBatchReader > > > Key: HIVE-19588 > URL: https://issues.apache.org/jira/browse/HIVE-19588 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19588.1.patch, Screen Shot 2018-05-16 at 2.23.25 > PM.png > > > Looks like we are doing file listing several times when creating one instance > of VectorizedOrcAcidRowBatchReader > AcidUtils.parseBaseOrDeltaBucketFilename() does full file listing (when > there are files with bucket_* prefix) just to get a single file out of a path > to figure out if it has ACID schema (as part of HIVE-18190). > There is full file listing where we populate > 1) ColumnizedDeleteEventRegistry > 2) SortMergedDeleteEventRegistry > 3) Twice in computeOffsetAndBucket() > > Attaching profiles which [~gopalv] took while debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19588) Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader
[ https://issues.apache.org/jira/browse/HIVE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19588: - Status: Patch Available (was: Open) > Several invocation of file listing when creating > VectorizedOrcAcidRowBatchReader > > > Key: HIVE-19588 > URL: https://issues.apache.org/jira/browse/HIVE-19588 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19588.1.patch, Screen Shot 2018-05-16 at 2.23.25 > PM.png > > > Looks like we are doing file listing several times when creating one instance > of VectorizedOrcAcidRowBatchReader > AcidUtils.parseBaseOrDeltaBucketFilename() does full file listing (when > there are files with bucket_* prefix) just to get a single file out of a path > to figure out if it has ACID schema (as part of HIVE-18190). > There is full file listing where we populate > 1) ColumnizedDeleteEventRegistry > 2) SortMergedDeleteEventRegistry > 3) Twice in computeOffsetAndBucket() > > Attaching profiles which [~gopalv] took while debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19588) Several invocation of file listing when creating VectorizedOrcAcidRowBatchReader
[ https://issues.apache.org/jira/browse/HIVE-19588?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19588: - Attachment: HIVE-19588.1.patch > Several invocation of file listing when creating > VectorizedOrcAcidRowBatchReader > > > Key: HIVE-19588 > URL: https://issues.apache.org/jira/browse/HIVE-19588 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.1.0 >Reporter: Nita Dembla >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19588.1.patch, Screen Shot 2018-05-16 at 2.23.25 > PM.png > > > Looks like we are doing file listing several times when creating one instance > of VectorizedOrcAcidRowBatchReader > AcidUtils.parseBaseOrDeltaBucketFilename() does full file listing (when > there are files with bucket_* prefix) just to get a single file out of a path > to figure out if it has ACID schema (as part of HIVE-18190). > There is full file listing where we populate > 1) ColumnizedDeleteEventRegistry > 2) SortMergedDeleteEventRegistry > 3) Twice in computeOffsetAndBucket() > > Attaching profiles which [~gopalv] took while debugging. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19584) Dictionary encoding for string types
[
https://issues.apache.org/jira/browse/HIVE-19584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479783#comment-16479783
]
Hive QA commented on HIVE-19584:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
42s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
37s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
16s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
56s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
31s{color} | {color:blue} common in master has 62 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
49s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
7s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
8s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
18s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
43s{color} | {color:red} ql: The patch generated 122 new + 658 unchanged - 48
fixed = 780 total (was 706) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 4m
5s{color} | {color:red} ql generated 2 new + 2320 unchanged - 0 fixed = 2322
total (was 2320) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 25m 42s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:ql |
| | Unchecked/unconfirmed cast from org.apache.arrow.vector.FieldVector to
org.apache.arrow.vector.complex.MapVector in
org.apache.hadoop.hive.ql.io.arrow.Serializer.write(FieldVector, ColumnVector,
TypeInfo, int, boolean) At
Serializer.java:org.apache.arrow.vector.complex.MapVector in
org.apache.hadoop.hive.ql.io.arrow.Serializer.write(FieldVector, ColumnVector,
TypeInfo, int, boolean) At Serializer.java:[line 268] |
| | Unchecked/unconfirmed cast from org.apache.arrow.vector.complex.MapVector
to org.apache.arrow.vector.FieldVector in
org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(MapVector,
StructColumnVector, StructTypeInfo, int) At
Serializer.java:org.apache.arrow.vector.FieldVector in
org.apache.hadoop.hive.ql.io.arrow.Serializer.writeStruct(MapVector,
StructColumnVector, StructTypeInfo, int) At Serializer.java:[line 342] |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11035/dev-support/hive-personality.sh
|
| git revision | master / 0b6bea8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11035/yetus/diff-checkstyle-ql.txt
|
| findbugs |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11035/yetus/new-findbugs-ql.html
|
| modules | C: common ql U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11035/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Dictionary encoding for string
[jira] [Updated] (HIVE-19597) TestWorkloadManager sometimes hangs
[
https://issues.apache.org/jira/browse/HIVE-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19597:
Description:
Seems like the tests randomly get stuck after the lines like
{noformat}
2018-05-17T01:54:27,111 INFO [Workload management master] tez.WorkloadManager:
Processing current events
2018-05-17T01:54:27,603 INFO [TriggerValidator]
tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI of
null
{noformat}
Then they get killed by timeout. Happens in the same manner, to random tests in
a few separate runs.
was:
Seems like the tests randomly get stuck after the lines like
{noformat}
2018-05-17T01:54:27,111 INFO [Workload management master] tez.WorkloadManager:
Processing current events
2018-05-17T01:54:27,603 INFO [TriggerValidator]
tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI of
null
{noformat}
Then they get killed by timeout. Happens to random tests in a few separate runs.
> TestWorkloadManager sometimes hangs
> ---
>
> Key: HIVE-19597
> URL: https://issues.apache.org/jira/browse/HIVE-19597
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Seems like the tests randomly get stuck after the lines like
> {noformat}
> 2018-05-17T01:54:27,111 INFO [Workload management master]
> tez.WorkloadManager: Processing current events
> 2018-05-17T01:54:27,603 INFO [TriggerValidator]
> tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
> 2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI
> of null
> {noformat}
> Then they get killed by timeout. Happens in the same manner, to random tests
> in a few separate runs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19597) TestWorkloadManager sometimes hangs
[
https://issues.apache.org/jira/browse/HIVE-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479770#comment-16479770
]
Sergey Shelukhin commented on HIVE-19597:
-
The trigger thing just runs on timer, and therefore happens last after the test
itself is probably already stuck... appears to be a red herring.
> TestWorkloadManager sometimes hangs
> ---
>
> Key: HIVE-19597
> URL: https://issues.apache.org/jira/browse/HIVE-19597
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Seems like the tests randomly get stuck after the lines like
> {noformat}
> 2018-05-17T01:54:27,111 INFO [Workload management master]
> tez.WorkloadManager: Processing current events
> 2018-05-17T01:54:27,603 INFO [TriggerValidator]
> tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
> 2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI
> of null
> {noformat}
> Then they get killed by timeout. Happened in the same manner, to random tests
> in a few separate runs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-16295) Add support for using Hadoop's S3A OutputCommitter
[
https://issues.apache.org/jira/browse/HIVE-16295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sahil Takiar updated HIVE-16295:
Attachment: HIVE-16295.5.patch
> Add support for using Hadoop's S3A OutputCommitter
> --
>
> Key: HIVE-16295
> URL: https://issues.apache.org/jira/browse/HIVE-16295
> Project: Hive
> Issue Type: Sub-task
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-16295.1.WIP.patch, HIVE-16295.2.WIP.patch,
> HIVE-16295.3.WIP.patch, HIVE-16295.4.patch, HIVE-16295.5.patch
>
>
> Hive doesn't have integration with Hadoop's {{OutputCommitter}}, it uses a
> {{NullOutputCommitter}} and uses its own commit logic spread across
> {{FileSinkOperator}}, {{MoveTask}}, and {{Hive}}.
> The Hadoop community is building an {{OutputCommitter}} that integrates with
> S3Guard and does a safe, coordinate commit of data on S3 inside individual
> tasks (HADOOP-13786). If Hive can integrate with this new {{OutputCommitter}}
> there would be a lot of benefits to Hive-on-S3:
> * Data is only written once; directly committing data at a task level means
> no renames are necessary
> * The commit is done safely, in a coordinated manner; duplicate tasks (from
> task retries or speculative execution) should not step on each other
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19597) TestWorkloadManager sometimes hangs
[
https://issues.apache.org/jira/browse/HIVE-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19597:
Description:
Seems like the tests randomly get stuck after the lines like
{noformat}
2018-05-17T01:54:27,111 INFO [Workload management master] tez.WorkloadManager:
Processing current events
2018-05-17T01:54:27,603 INFO [TriggerValidator]
tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI of
null
{noformat}
Then they get killed by timeout. Happened in the same manner, to random tests
in a few separate runs.
was:
Seems like the tests randomly get stuck after the lines like
{noformat}
2018-05-17T01:54:27,111 INFO [Workload management master] tez.WorkloadManager:
Processing current events
2018-05-17T01:54:27,603 INFO [TriggerValidator]
tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI of
null
{noformat}
Then they get killed by timeout. Happens in the same manner, to random tests in
a few separate runs.
> TestWorkloadManager sometimes hangs
> ---
>
> Key: HIVE-19597
> URL: https://issues.apache.org/jira/browse/HIVE-19597
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Seems like the tests randomly get stuck after the lines like
> {noformat}
> 2018-05-17T01:54:27,111 INFO [Workload management master]
> tez.WorkloadManager: Processing current events
> 2018-05-17T01:54:27,603 INFO [TriggerValidator]
> tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
> 2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI
> of null
> {noformat}
> Then they get killed by timeout. Happened in the same manner, to random tests
> in a few separate runs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18916) SparkClientImpl doesn't error out if spark-submit fails
[
https://issues.apache.org/jira/browse/HIVE-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479722#comment-16479722
]
Hive QA commented on HIVE-18916:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923934/HIVE-18916.3.patch
{color:green}SUCCESS:{color} +1 due to 3 test(s) being added or modified.
{color:green}SUCCESS:{color} +1 due to 14411 tests passed
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11034/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11034/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11034/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923934 - PreCommit-HIVE-Build
> SparkClientImpl doesn't error out if spark-submit fails
> ---
>
> Key: HIVE-18916
> URL: https://issues.apache.org/jira/browse/HIVE-18916
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18916.1.WIP.patch, HIVE-18916.2.patch,
> HIVE-18916.3.patch
>
>
> If {{spark-submit}} returns a non-zero exit code, {{SparkClientImpl}} will
> simply log the exit code, but won't throw an error. Eventually, the
> connection timeout will get triggered and an exception like {{Timed out
> waiting for client connection}} will be logged, which is pretty misleading.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[
https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479719#comment-16479719
]
Sergey Shelukhin commented on HIVE-19258:
-
I was able to repro the failure of bucketizedhiveinputformat on the same query,
then I realized I forgot to actually apply the patch.
The failure, assuming it's the same one as here, is OOM in one of the stdouts:
{noformat}
[DEBUG] 2018-05-17 14:00:24.209 [IPC Client (928988232) connection to
localhost/127.0.0.1:50278 from sergey] Client - IPC Client (928988232)
connection to localhost/127.0.0.1:50278 from sergey: closed
[DEBUG] 2018-05-17 14:00:24.209 [IPC Client (928988232) connection to
localhost/127.0.0.1:50278 from sergey] Client - IPC Client (928988232)
connection to localhost/127.0.0.1:50278 from sergey: stopped, remaining
connections 0
#
# java.lang.OutOfMemoryError: GC overhead limit exceeded
# -XX:OnOutOfMemoryError="kill %p"
# Executing "kill 5333"...
[ERROR] 2018-05-17 14:01:39.613 [SIGTERM handler] CoarseGrainedExecutorBackend
- RECEIVED SIGNAL TERM
[DEBUG] 2018-05-17 14:01:39.613 [Executor task launch worker for task 18]
TaskMemoryManager - unreleased 268.0 MB memory from
org.apache.spark.util.collection.ExternalAppendOnlyMap@141a89ea
[ERROR] 2018-05-17 14:01:39.616 [Executor task launch worker for task 18]
Executor - Exception in task 0.0 in stage 2.2 (TID 18)
java.lang.OutOfMemoryError: GC overhead limit exceeded
at
org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:188)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.serializer.DeserializationStream$$anon$2.getNext(Serializer.scala:185)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.util.NextIterator.hasNext(NextIterator.scala:73)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:438)
~[scala-library-2.11.8.jar:?]
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
~[scala-library-2.11.8.jar:?]
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:37)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.util.collection.ExternalAppendOnlyMap.insertAll(ExternalAppendOnlyMap.scala:153)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.Aggregator.combineValuesByKey(Aggregator.scala:41)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:90)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.ShuffledRDD.compute(ShuffledRDD.scala:105)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:324)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.rdd.RDD.iterator(RDD.scala:288)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at org.apache.spark.scheduler.Task.run(Task.scala:109)
~[spark-core_2.11-2.3.0.jar:2.3.0]
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:345)
[spark-core_2.11-2.3.0.jar:2.3.0]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
[?:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[?:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_45]
[INFO ] 2018-05-17 14:01:39.617 [pool-8-thread-1] DiskBlockManager - Shutdown
hook called
{noformat}
cc [~xuefuz]
> add originals support to MM tables (and make the conversion a metadata only
> operation)
> --
>
> Key: HIVE-19258
> URL: https://issues.apache.org/jira/browse/HIVE-19258
> Project: Hive
> Issue Type: Bug
> Components: Transactions
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch,
[jira] [Commented] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[ https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479720#comment-16479720 ] Sergey Shelukhin commented on HIVE-19258: - WM tests are flaky: HIVE-19597 > add originals support to MM tables (and make the conversion a metadata only > operation) > -- > > Key: HIVE-19258 > URL: https://issues.apache.org/jira/browse/HIVE-19258 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, > HIVE-19258.03.patch, HIVE-19258.04.patch, HIVE-19258.05.patch, > HIVE-19258.06.patch, HIVE-19258.07.patch, HIVE-19258.08.patch, > HIVE-19258.08.patch, HIVE-19258.09.patch, HIVE-19258.10.patch, > HIVE-19258.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19258) add originals support to MM tables (and make the conversion a metadata only operation)
[ https://issues.apache.org/jira/browse/HIVE-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19258: Attachment: HIVE-19258.10.patch > add originals support to MM tables (and make the conversion a metadata only > operation) > -- > > Key: HIVE-19258 > URL: https://issues.apache.org/jira/browse/HIVE-19258 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19258.01.patch, HIVE-19258.02.patch, > HIVE-19258.03.patch, HIVE-19258.04.patch, HIVE-19258.05.patch, > HIVE-19258.06.patch, HIVE-19258.07.patch, HIVE-19258.08.patch, > HIVE-19258.08.patch, HIVE-19258.09.patch, HIVE-19258.10.patch, > HIVE-19258.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19599) Release Notes : Highlighting backwards incompatible changes
[ https://issues.apache.org/jira/browse/HIVE-19599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-19599: - > Release Notes : Highlighting backwards incompatible changes > --- > > Key: HIVE-19599 > URL: https://issues.apache.org/jira/browse/HIVE-19599 > Project: Hive > Issue Type: Bug > Components: Documentation >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Vineet Garg >Priority: Blocker > > We need to highlight backwards incompatible changes. A list Jira titles > won't be sufficient. > For example, tables with Acid V1 (pre 3.0) data has to be major compacted > before upgrade and may not process any update/delete/merge until after > upgrade. Not doing so may result in data corruption/loss. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19598) Acid V1 to V2 upgrade
[ https://issues.apache.org/jira/browse/HIVE-19598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19598: -- Target Version/s: 3.1.0 > Acid V1 to V2 upgrade > - > > Key: HIVE-19598 > URL: https://issues.apache.org/jira/browse/HIVE-19598 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > > The on-disk layout for full acid (transactional) tables has changed 3.0. > Any transactional table that has any update/delete events in any deltas that > have not been Major compacted, must go through a Major compaction before > upgrading to 3.0. No more update/delete/merge should be run after/during > major compaction. > Not doing so will result in data corruption/loss. > > Need to create a utility tool to help with this process. HIVE-19233 started > this but it needs more work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19598) Acid V1 to V2 upgrade
[ https://issues.apache.org/jira/browse/HIVE-19598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman updated HIVE-19598: -- Priority: Critical (was: Major) > Acid V1 to V2 upgrade > - > > Key: HIVE-19598 > URL: https://issues.apache.org/jira/browse/HIVE-19598 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Critical > > The on-disk layout for full acid (transactional) tables has changed 3.0. > Any transactional table that has any update/delete events in any deltas that > have not been Major compacted, must go through a Major compaction before > upgrading to 3.0. No more update/delete/merge should be run after/during > major compaction. > Not doing so will result in data corruption/loss. > > Need to create a utility tool to help with this process. HIVE-19233 started > this but it needs more work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19598) Acid V1 to V2 upgrade
[ https://issues.apache.org/jira/browse/HIVE-19598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-19598: - > Acid V1 to V2 upgrade > - > > Key: HIVE-19598 > URL: https://issues.apache.org/jira/browse/HIVE-19598 > Project: Hive > Issue Type: Bug > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Eugene Koifman >Assignee: Eugene Koifman >Priority: Major > > The on-disk layout for full acid (transactional) tables has changed 3.0. > Any transactional table that has any update/delete events in any deltas that > have not been Major compacted, must go through a Major compaction before > upgrading to 3.0. No more update/delete/merge should be run after/during > major compaction. > Not doing so will result in data corruption/loss. > > Need to create a utility tool to help with this process. HIVE-19233 started > this but it needs more work. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479678#comment-16479678 ] Prasanth Jayachandran commented on HIVE-18079: -- Minor change that is required for HIVE-19578. sparseMap access is replaced with getSparseMap() which will return merged (sparseMap + tempList) sparse map with HIVE-19578. This is required for the new tests to pass. > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079-branch-3.patch, HIVE-18079-branch-3.patch, > HIVE-18079-branch-3.patch, HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.15.patch, HIVE-18079.15.patch, > HIVE-18079.15.patch, HIVE-18079.16.patch, HIVE-18079.2.patch, > HIVE-18079.4.patch, HIVE-18079.5.patch, HIVE-18079.6.patch, > HIVE-18079.7.patch, HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18079: - Attachment: HIVE-18079.16.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079-branch-3.patch, HIVE-18079-branch-3.patch, > HIVE-18079-branch-3.patch, HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.15.patch, HIVE-18079.15.patch, > HIVE-18079.15.patch, HIVE-18079.16.patch, HIVE-18079.2.patch, > HIVE-18079.4.patch, HIVE-18079.5.patch, HIVE-18079.6.patch, > HIVE-18079.7.patch, HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18079) Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator bit-size
[ https://issues.apache.org/jira/browse/HIVE-18079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-18079: - Attachment: HIVE-18079-branch-3.patch > Statistics: Allow HyperLogLog to be merged to the lowest-common-denominator > bit-size > > > Key: HIVE-18079 > URL: https://issues.apache.org/jira/browse/HIVE-18079 > Project: Hive > Issue Type: Improvement > Components: Standalone Metastore, Statistics >Affects Versions: 3.0.0 >Reporter: Gopal V >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-18079-branch-3.patch, HIVE-18079-branch-3.patch, > HIVE-18079-branch-3.patch, HIVE-18079.1.patch, HIVE-18079.10.patch, > HIVE-18079.11.patch, HIVE-18079.12.patch, HIVE-18079.13.patch, > HIVE-18079.14.patch, HIVE-18079.15.patch, HIVE-18079.15.patch, > HIVE-18079.15.patch, HIVE-18079.16.patch, HIVE-18079.2.patch, > HIVE-18079.4.patch, HIVE-18079.5.patch, HIVE-18079.6.patch, > HIVE-18079.7.patch, HIVE-18079.8.patch, HIVE-18079.9.patch > > > HyperLogLog can merge a 14 bit HLL into a 10 bit HLL bitset, because of its > mathematical hash distribution & construction. > Allow the squashing of a 14 bit HLL -> 10 bit HLL without needing a second > scan over the data-set. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19578) HLL merges tempList on every add
[ https://issues.apache.org/jira/browse/HIVE-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479669#comment-16479669 ] Prasanth Jayachandran commented on HIVE-19578: -- [~sershe] can you look at .2 patch? > HLL merges tempList on every add > > > Key: HIVE-19578 > URL: https://issues.apache.org/jira/browse/HIVE-19578 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19578.1.patch, HIVE-19578.2.patch, Screen Shot > 2018-05-16 at 15.29.12 .png > > > See comments on HIVE-18866; this has significant perf overhead after the > even bigger overhead from hashing is removed. !Screen Shot 2018-05-16 at > 15.29.12 .png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19578) HLL merges tempList on every add
[ https://issues.apache.org/jira/browse/HIVE-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Prasanth Jayachandran updated HIVE-19578: - Attachment: HIVE-19578.2.patch > HLL merges tempList on every add > > > Key: HIVE-19578 > URL: https://issues.apache.org/jira/browse/HIVE-19578 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19578.1.patch, HIVE-19578.2.patch, Screen Shot > 2018-05-16 at 15.29.12 .png > > > See comments on HIVE-18866; this has significant perf overhead after the > even bigger overhead from hashing is removed. !Screen Shot 2018-05-16 at > 15.29.12 .png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19578) HLL merges tempList on every add
[
https://issues.apache.org/jira/browse/HIVE-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479665#comment-16479665
]
Prasanth Jayachandran commented on HIVE-19578:
--
New benchmark run with and without .2 patch
{code:java}
BenchmarkMode Cnt Score Error Units
HyperLogLogAdd.testHLLAddHiveWithPatch avgt 10 7.110 ± 0.085 ns/op
HyperLogLogAdd.testHLLAddHiveWithoutPath avgt 10 46.276 ± 2.767 ns/op{code}
> HLL merges tempList on every add
>
>
> Key: HIVE-19578
> URL: https://issues.apache.org/jira/browse/HIVE-19578
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Prasanth Jayachandran
>Priority: Major
> Attachments: HIVE-19578.1.patch, HIVE-19578.2.patch, Screen Shot
> 2018-05-16 at 15.29.12 .png
>
>
> See comments on HIVE-18866; this has significant perf overhead after the
> even bigger overhead from hashing is removed. !Screen Shot 2018-05-16 at
> 15.29.12 .png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-19579) remove HBase transitive dependency that drags in some snapshot
[
https://issues.apache.org/jira/browse/HIVE-19579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19579:
Attachment: HIVE-19579.01.patch
> remove HBase transitive dependency that drags in some snapshot
> --
>
> Key: HIVE-19579
> URL: https://issues.apache.org/jira/browse/HIVE-19579
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19579.01.patch, HIVE-19579.patch
>
>
> This:
> {noformat}
> [INFO] org.apache.hive:hive-llap-server:jar:3.1.0-SNAPSHOT
> [INFO] \- org.apache.hbase:hbase-server:jar:2.0.0-alpha4:compile
> [INFO]\- org.glassfish.web:javax.servlet.jsp:jar:2.3.2:compile
> [INFO] \- org.glassfish:javax.el:jar:3.0.1-b10:compile (version
> selected from constraint [3.0.0,))
> {noformat}
> occasionally results in bizarre build failures like this:
> {noformat}
> Failed to read artifact descriptor for
> org.glassfish:javax.el:jar:3.0.1-b06-SNAPSHOT: Could not transfer artifact
> org.glassfish:javax.el:pom:3.0.1-b06-SNAPSHOT from/to jvnet-nexus-snapshots
> (https://maven.java.net/content/repositories/snapshots): Failed to transfer
> file:
> https://maven.java.net/content/repositories/snapshots/org/glassfish/javax.el/3.0.1-b06-SNAPSHOT/javax.el-3.0.1-b06-SNAPSHOT.pom.
> Return code is: 402 , ReasonPhrase:Payment Required.
> {noformat}
> cc [~enis] [~stack] if you guys want to take a look at HBase side
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HIVE-19028) Insert values query gives Failed to retrieve table statistics
[ https://issues.apache.org/jira/browse/HIVE-19028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan resolved HIVE-19028. - Resolution: Duplicate Dupe of HIVE-19390 > Insert values query gives Failed to retrieve table statistics > - > > Key: HIVE-19028 > URL: https://issues.apache.org/jira/browse/HIVE-19028 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Thejas M Nair >Assignee: Jesus Camacho Rodriguez >Priority: Major > > INSERT INTO `VERSION` VALUES (1, '3.0.0', 'Hive release version 3.0.0') gives > error like - > 2018-03-22T17:09:42,680 ERROR [946f78ac-24e9-493b-b877-336d3006a9b5 > HiveServer2-Handler-Pool: Thread-87]: stats.StatsUtils (:()) - Failed to > retrieve table statistics: > org.apache.hadoop.hive.ql.metadata.HiveException: > NoSuchObjectException(message:Specified database/table does not exist : > _dummy_database._dummy_table) > The query proceeds, but this can be confusing to users, support team etc > Full stack is here - > https://gist.github.com/thejasmn/7e33ad38e7bc552e3a659adc041ba200 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18117) Create TestCliDriver for HDFS EC
[ https://issues.apache.org/jira/browse/HIVE-18117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Sherman updated HIVE-18117: -- Attachment: HIVE-18117.8.patch > Create TestCliDriver for HDFS EC > > > Key: HIVE-18117 > URL: https://issues.apache.org/jira/browse/HIVE-18117 > Project: Hive > Issue Type: Sub-task >Reporter: Sahil Takiar >Assignee: Andrew Sherman >Priority: Major > Attachments: HIVE-18117.1.patch, HIVE-18117.2.patch, > HIVE-18117.3.patch, HIVE-18117.4.patch, HIVE-18117.5.patch, > HIVE-18117.6.patch, HIVE-18117.7.patch, HIVE-18117.8.patch > > > Should be able to do something similar to what we do for HDFS encryption. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19597) TestWorkloadManager sometimes hangs
[
https://issues.apache.org/jira/browse/HIVE-19597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin reassigned HIVE-19597:
---
> TestWorkloadManager sometimes hangs
> ---
>
> Key: HIVE-19597
> URL: https://issues.apache.org/jira/browse/HIVE-19597
> Project: Hive
> Issue Type: Bug
>Reporter: Sergey Shelukhin
>Assignee: Sergey Shelukhin
>Priority: Major
>
> Seems like the tests randomly get stuck after the lines like
> {noformat}
> 2018-05-17T01:54:27,111 INFO [Workload management master]
> tez.WorkloadManager: Processing current events
> 2018-05-17T01:54:27,603 INFO [TriggerValidator]
> tez.PerPoolTriggerValidatorRunnable: Creating trigger validator for pool: llap
> 2018-05-17T01:54:37,090 DEBUG [Thread-28] conf.HiveConf: Found metastore URI
> of null
> {noformat}
> Then they get killed by timeout. Happens to random tests in a few separate
> runs.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HIVE-18916) SparkClientImpl doesn't error out if spark-submit fails
[
https://issues.apache.org/jira/browse/HIVE-18916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479651#comment-16479651
]
Hive QA commented on HIVE-18916:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
37s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
24s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
43s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
41s{color} | {color:blue} itests/util in master has 55 extant Findbugs
warnings. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 3m
52s{color} | {color:blue} ql in master has 2320 extant Findbugs warnings.
{color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
23s{color} | {color:blue} spark-client in master has 15 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
24s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
41s{color} | {color:red} ql: The patch generated 2 new + 15 unchanged - 2 fixed
= 17 total (was 17) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
10s{color} | {color:red} spark-client: The patch generated 2 new + 28 unchanged
- 0 fixed = 30 total (was 28) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
19s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
23s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
12s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 28m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/data/hiveptest/working/yetus_PreCommit-HIVE-Build-11034/dev-support/hive-personality.sh
|
| git revision | master / 0b6bea8 |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11034/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11034/yetus/diff-checkstyle-spark-client.txt
|
| asflicense |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11034/yetus/patch-asflicense-problems.txt
|
| modules | C: itests itests/util ql spark-client U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-11034/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> SparkClientImpl doesn't error out if spark-submit fails
> ---
>
> Key: HIVE-18916
> URL: https://issues.apache.org/jira/browse/HIVE-18916
> Project: Hive
> Issue Type: Sub-task
> Components: Spark
>Reporter: Sahil Takiar
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HIVE-18916.1.WIP.patch, HIVE-18916.2.patch,
> HIVE-18916.3.patch
>
>
>
[jira] [Commented] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts
[
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479634#comment-16479634
]
Sergey Shelukhin commented on HIVE-19516:
-
Unrelated failures, just rerunning. [~jcamachorodriguez] since this one only
touched unrelated itests, all of the tests that failed above are probably good
candidates for flaky
> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19516.01.patch, HIVE-19516.02.patch,
> HIVE-19516.03.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you
> run in batch on local system
> {noformat}
> mvn -B test -Dtest.groups= -Dtest=TestNegativeCliDriver
>
[jira] [Updated] (HIVE-19516) TestNegative merge_negative_5 and mm_concatenate are causing timeouts
[
https://issues.apache.org/jira/browse/HIVE-19516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HIVE-19516:
Attachment: HIVE-19516.03.patch
> TestNegative merge_negative_5 and mm_concatenate are causing timeouts
> -
>
> Key: HIVE-19516
> URL: https://issues.apache.org/jira/browse/HIVE-19516
> Project: Hive
> Issue Type: Bug
> Components: Testing Infrastructure
>Reporter: Vineet Garg
>Assignee: Sergey Shelukhin
>Priority: Major
> Attachments: HIVE-19516.01.patch, HIVE-19516.02.patch,
> HIVE-19516.03.patch, HIVE-19516.patch
>
>
> I haven't tried to reproduce this in isolation but it is reproducible if you
> run in batch on local system
> {noformat}
> mvn -B test -Dtest.groups= -Dtest=TestNegativeCliDriver
>
[jira] [Commented] (HIVE-19418) add background stats updater similar to compactor
[ https://issues.apache.org/jira/browse/HIVE-19418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479621#comment-16479621 ] Sergey Shelukhin commented on HIVE-19418: - Test failures looks bogus. Given the state of HiveQA it makes no sense to rerun for now, I'm assuming there will anyway be some changes based on the review. > add background stats updater similar to compactor > - > > Key: HIVE-19418 > URL: https://issues.apache.org/jira/browse/HIVE-19418 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19418.01.patch, HIVE-19418.02.patch, > HIVE-19418.patch > > > There's a JIRA HIVE-19416 to add snapshot version to stats for MM/ACID tables > to make them usable in a transaction without breaking ACID (for metadata-only > optimization). However, stats for ACID tables can still become unusable if > e.g. two parallel inserts run - neither sees the data written by the other, > so after both finish, the snapshots on either set of stats won't match the > current snapshot and the stats will be unusable. > Additionally, for ACID and non-ACID tables alike, a lot of the stats, with > some exceptions like numRows, cannot be aggregated (i.e. you cannot combine > ndvs from two inserts), and for ACID even less can be aggregated (you cannot > derive min/max if some rows are deleted but you don't scan the rest of the > dataset). > Therefore we will add background logic to metastore (similar to, and > partially inside, the ACID compactor) to update stats. > It will have 3 modes of operation. > 1) Off. > 2) Update only the stats that exist but are out of date (generating stats can > be expensive, so if the user is only analyzing a subset of tables it should > be able to only update that subset). We can simply look at existing stats and > only analyze for the relevant partitions and columns. > 3) On: 2 + create stats for all tables and columns missing stats. > There will also be a table parameter to skip stats update. > In phase 1, the process will operate outside of compactor, and run analyze > command on the table. The analyze command will automatically save the stats > with ACID snapshot information if needed, based on HIVE-19416, so we don't > need to do any special state management and this will work for all table > types. However it's also more expensive. > In phase 2, we can explore adding stats collection during MM compaction that > uses a temp table. If we don't have open writers during major compaction (so > we overwrite all of the data), the temp table stats can simply be copied over > to the main table with correct snapshot information, saving us a table scan. > In phase 3, we can add custom stats collection logic to full ACID compactor > that is not query based, the same way as we'd do for (2). Alternatively we > can wait for ACID compactor to become query based and just reuse (2). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19578) HLL merges tempList on every add
[ https://issues.apache.org/jira/browse/HIVE-19578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19578: Target Version/s: 3.1.0 > HLL merges tempList on every add > > > Key: HIVE-19578 > URL: https://issues.apache.org/jira/browse/HIVE-19578 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Prasanth Jayachandran >Priority: Major > Attachments: HIVE-19578.1.patch, Screen Shot 2018-05-16 at 15.29.12 > .png > > > See comments on HIVE-18866; this has significant perf overhead after the > even bigger overhead from hashing is removed. !Screen Shot 2018-05-16 at > 15.29.12 .png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path
[ https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18866: Attachment: HIVE-18866.01.patch > Semijoin: Implement a Long -> Hash64 vector fast-path > - > > Key: HIVE-18866 > URL: https://issues.apache.org/jira/browse/HIVE-18866 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Sergey Shelukhin >Priority: Major > Labels: performance > Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, > HIVE-18866.patch, perf-hash64-long.png > > > A significant amount of CPU is wasted with JMM restrictions on byte[] arrays. > To transform from one Long -> another Long, this goes into a byte[] array, > which shows up as a hotspot. > !perf-hash64-long.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path
[ https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479618#comment-16479618 ] Sergey Shelukhin commented on HIVE-18866: - Removed the imports. Test failures look bogus, trying again. > Semijoin: Implement a Long -> Hash64 vector fast-path > - > > Key: HIVE-18866 > URL: https://issues.apache.org/jira/browse/HIVE-18866 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Sergey Shelukhin >Priority: Major > Labels: performance > Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, > HIVE-18866.patch, perf-hash64-long.png > > > A significant amount of CPU is wasted with JMM restrictions on byte[] arrays. > To transform from one Long -> another Long, this goes into a byte[] array, > which shows up as a hotspot. > !perf-hash64-long.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path
[ https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18866: Attachment: (was: HIVE-18866.01.patch) > Semijoin: Implement a Long -> Hash64 vector fast-path > - > > Key: HIVE-18866 > URL: https://issues.apache.org/jira/browse/HIVE-18866 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Sergey Shelukhin >Priority: Major > Labels: performance > Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, > HIVE-18866.patch, perf-hash64-long.png > > > A significant amount of CPU is wasted with JMM restrictions on byte[] arrays. > To transform from one Long -> another Long, this goes into a byte[] array, > which shows up as a hotspot. > !perf-hash64-long.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19370) Issue: ADD Months function on timestamp datatype fields in hive
[
https://issues.apache.org/jira/browse/HIVE-19370?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479611#comment-16479611
]
Hive QA commented on HIVE-19370:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12923930/HIVE-19370.03.patch
{color:green}SUCCESS:{color} +1 due to 2 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 10 failed/errored test(s), 14336 tests
executed
*Failed tests:*
{noformat}
TestBeeLineExceptionHandling - did not produce a TEST-*.xml file (likely timed
out) (batchId=191)
TestBeeLineHistory - did not produce a TEST-*.xml file (likely timed out)
(batchId=191)
TestBeelineArgParsing - did not produce a TEST-*.xml file (likely timed out)
(batchId=191)
TestClientCommandHookFactory - did not produce a TEST-*.xml file (likely timed
out) (batchId=191)
TestHiveCli - did not produce a TEST-*.xml file (likely timed out) (batchId=191)
TestHiveSchemaTool - did not produce a TEST-*.xml file (likely timed out)
(batchId=191)
TestIncrementalRows - did not produce a TEST-*.xml file (likely timed out)
(batchId=191)
TestShutdownHook - did not produce a TEST-*.xml file (likely timed out)
(batchId=191)
TestTableOutputFormat - did not produce a TEST-*.xml file (likely timed out)
(batchId=191)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[localtimezone]
(batchId=55)
{noformat}
Test results:
https://builds.apache.org/job/PreCommit-HIVE-Build/11033/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/11033/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-11033/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 10 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12923930 - PreCommit-HIVE-Build
> Issue: ADD Months function on timestamp datatype fields in hive
> ---
>
> Key: HIVE-19370
> URL: https://issues.apache.org/jira/browse/HIVE-19370
> Project: Hive
> Issue Type: Bug
>Reporter: Amit Chauhan
>Assignee: Bharathkrishna Guruvayoor Murali
>Priority: Major
> Attachments: HIVE-19370.01.patch, HIVE-19370.02.patch,
> HIVE-19370.03.patch
>
>
> *Issue:*
> while using ADD_Months function on a timestamp datatype column the output
> omits the time part[HH:MM:SS] part from output.
> which should not be the case.
> *query:* EMAIL_FAILURE_DTMZ is of datatype timestamp in hive.
> hive> select CUSTOMER_ID,EMAIL_FAILURE_DTMZ,ADD_MONTHS (EMAIL_FAILURE_DTMZ ,
> 1) from TABLE1 where CUSTOMER_ID=125674937;
> OK
> 125674937 2015-12-09 12:25:53 2016-01-09
> *hiver version :*
> hive> !hive --version;
> Hive 1.2.1000.2.5.6.0-40
>
> can you please help if somehow I can get below as output:
>
> 125674937 2015-12-09 12:25:53 2016-01-09 12:25:53
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path
[ https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-18866: Target Version/s: 3.1.0 > Semijoin: Implement a Long -> Hash64 vector fast-path > - > > Key: HIVE-18866 > URL: https://issues.apache.org/jira/browse/HIVE-18866 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Sergey Shelukhin >Priority: Major > Labels: performance > Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, > HIVE-18866.patch, perf-hash64-long.png > > > A significant amount of CPU is wasted with JMM restrictions on byte[] arrays. > To transform from one Long -> another Long, this goes into a byte[] array, > which shows up as a hotspot. > !perf-hash64-long.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19417) Modify metastore to have/access persistent tables for stats
[ https://issues.apache.org/jira/browse/HIVE-19417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479600#comment-16479600 ] Vineet Garg commented on HIVE-19417: The patch is touching wrong metastore upgrade scripts and schema scripts e.g. upgrade-2.3.0-to-3.0.0.derby because we have already cut off 3.0 this is wrong. Depending upon what version this patch is targeted for appropriate upgrade/schema scripts should be added and edited. If this is targeted for 3.1 please add upgrade-3.0.0-to-3.1.0..xx scripts for all databases plus hive-schema-3.1.0 for all databases. > Modify metastore to have/access persistent tables for stats > --- > > Key: HIVE-19417 > URL: https://issues.apache.org/jira/browse/HIVE-19417 > Project: Hive > Issue Type: Sub-task > Components: Transactions >Affects Versions: 3.0.0 >Reporter: Steve Yeom >Assignee: Steve Yeom >Priority: Major > Attachments: HIVE-19417.01.patch, HIVE-19417.02.patch, > HIVE-19417.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-18866) Semijoin: Implement a Long -> Hash64 vector fast-path
[ https://issues.apache.org/jira/browse/HIVE-18866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-18866: Attachment: HIVE-18866.01.patch > Semijoin: Implement a Long -> Hash64 vector fast-path > - > > Key: HIVE-18866 > URL: https://issues.apache.org/jira/browse/HIVE-18866 > Project: Hive > Issue Type: Improvement > Components: Vectorization >Reporter: Gopal V >Assignee: Sergey Shelukhin >Priority: Major > Labels: performance > Attachments: 0001-hash64-WIP.patch, HIVE-18866.01.patch, > HIVE-18866.patch, perf-hash64-long.png > > > A significant amount of CPU is wasted with JMM restrictions on byte[] arrays. > To transform from one Long -> another Long, this goes into a byte[] array, > which shows up as a hotspot. > !perf-hash64-long.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19591) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19591: Status: Patch Available (was: Open) > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19591 > URL: https://issues.apache.org/jira/browse/HIVE-19591 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19591.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19594) CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19594: Status: Patch Available (was: Open) > CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on > HiveQA > > > Key: HIVE-19594 > URL: https://issues.apache.org/jira/browse/HIVE-19594 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19594.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19589: Status: Patch Available (was: Open) > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19589 > URL: https://issues.apache.org/jira/browse/HIVE-19589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19589.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19590: Status: Patch Available (was: Open) > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19592: Status: Patch Available (was: Open) > CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19592 > URL: https://issues.apache.org/jira/browse/HIVE-19592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19592.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19593) CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19593: Status: Patch Available (was: Open) > CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19593 > URL: https://issues.apache.org/jira/browse/HIVE-19593 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19593.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19589) NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19589: Attachment: HIVE-19589.patch > NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19589 > URL: https://issues.apache.org/jira/browse/HIVE-19589 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19589.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19592: Attachment: HIVE-19592.patch > CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19592 > URL: https://issues.apache.org/jira/browse/HIVE-19592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19592.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19590) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19590: Attachment: HIVE-19590.patch > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19590 > URL: https://issues.apache.org/jira/browse/HIVE-19590 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19590.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19462) Fix mapping for char_length function to enable pushdown to Druid.
[ https://issues.apache.org/jira/browse/HIVE-19462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashutosh Chauhan updated HIVE-19462: Target Version/s: 3.1.0 > Fix mapping for char_length function to enable pushdown to Druid. > -- > > Key: HIVE-19462 > URL: https://issues.apache.org/jira/browse/HIVE-19462 > Project: Hive > Issue Type: Improvement > Components: Druid integration >Reporter: slim bouguerra >Assignee: slim bouguerra >Priority: Major > Attachments: HIVE-19462.patch > > > currently char_length is not push down to Druid because of missing mapping > form/to calcite > This patch will add this mapping. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19591) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19591: Attachment: HIVE-19591.patch > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19591 > URL: https://issues.apache.org/jira/browse/HIVE-19591 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19591.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19596) CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19596: Attachment: HIVE-19596.patch > CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests > are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19593) CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19593: Attachment: HIVE-19593.patch > CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19593 > URL: https://issues.apache.org/jira/browse/HIVE-19593 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19593.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19594) CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19594: Attachment: HIVE-19594.patch > CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on > HiveQA > > > Key: HIVE-19594 > URL: https://issues.apache.org/jira/browse/HIVE-19594 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19594.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19595) CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19595: Attachment: HIVE-19595.patch > CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are > flaky on HiveQA > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19596) CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19596: Status: Patch Available (was: Open) > CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests > are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19596.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (HIVE-19595) CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin updated HIVE-19595: Status: Patch Available (was: Open) > CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are > flaky on HiveQA > > > Key: HIVE-19595 > URL: https://issues.apache.org/jira/browse/HIVE-19595 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > Attachments: HIVE-19595.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19596) CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19596: --- > CLONE - CLONE - CLONE - CLONE - CLONE - CLONE - NOOP jira to see which tests > are flaky on HiveQA > > > Key: HIVE-19596 > URL: https://issues.apache.org/jira/browse/HIVE-19596 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HIVE-19460) Improve stats estimations for NOT IN operator
[ https://issues.apache.org/jira/browse/HIVE-19460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16479594#comment-16479594 ] Ashutosh Chauhan commented on HIVE-19460: - Lets break this in 2 patches : one without bitvectors and one with it. > Improve stats estimations for NOT IN operator > - > > Key: HIVE-19460 > URL: https://issues.apache.org/jira/browse/HIVE-19460 > Project: Hive > Issue Type: Bug >Reporter: Zoltan Haindrich >Assignee: Zoltan Haindrich >Priority: Major > Attachments: HIVE-19460.01wip01.patch, HIVE-19460.01wip02.patch, > HIVE-19460.01wip03.patch, HIVE-19460.02.patch, HIVE-19460.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19591) CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19591: --- > CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19591 > URL: https://issues.apache.org/jira/browse/HIVE-19591 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (HIVE-19592) CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA
[ https://issues.apache.org/jira/browse/HIVE-19592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Shelukhin reassigned HIVE-19592: --- > CLONE - CLONE - NOOP jira to see which tests are flaky on HiveQA > > > Key: HIVE-19592 > URL: https://issues.apache.org/jira/browse/HIVE-19592 > Project: Hive > Issue Type: Bug >Reporter: Sergey Shelukhin >Assignee: Sergey Shelukhin >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
