[jira] [Commented] (HIVE-18048) Support Struct type with vectorization for Parquet file
[
https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303116#comment-16303116
]
Hive QA commented on HIVE-18048:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903605/HIVE-18048.001.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 38 failed/errored test(s), 11544 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[smb_mapjoin_25]
(batchId=7)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_reference_windowed]
(batchId=32)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[mergejoin]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_part_all_complex]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_complex]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_complex]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_complex_all]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniSparkOnYarnCliDriver.testCliDriver[bucketizedhiveinputformat]
(batchId=177)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedColumnReader.structReadSomeNull
(batchId=267)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedColumnReader.testNestedStructRead
(batchId=267)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedColumnReader.testStructRead
(batchId=267)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedDictionaryEncodingColumnReader.structReadSomeNull
(batchId=266)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedDictionaryEncodingColumnReader.testNestedStructRead
(batchId=266)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedDictionaryEncodingColumnReader.testStructRead
(batchId=266)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8374/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8374/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8374/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 38 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903605 - PreCommit-HIVE-Build
> Support Struct type with
[jira] [Commented] (HIVE-18048) Support Struct type with vectorization for Parquet file
[
https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303113#comment-16303113
]
Hive QA commented on HIVE-18048:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
30s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
20s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
10s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
1s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
39s{color} | {color:red} ql: The patch generated 3 new + 1004 unchanged - 1
fixed = 1007 total (was 1005) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
40s{color} | {color:red} root: The patch generated 3 new + 1004 unchanged - 1
fixed = 1007 total (was 1005) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
20s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 41m 35s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 035eca3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8374/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8374/yetus/diff-checkstyle-root.txt
|
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8374/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support Struct type with vectorization for Parquet file
> ---
>
> Key: HIVE-18048
> URL: https://issues.apache.org/jira/browse/HIVE-18048
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18048.001.patch
>
>
> Struct type is not supported in MapWork with vectorization, it should be
> supported to improve the performance.
> The following improvements will be implemented:
> * Add fields of struct type to VectorizedRowBatchCtx.
> * Improve the VectorizedParquetRecordReader to support the struct type for
> parquet file.
> Note:
> * Orc file won't be supported.
> * Filter operator won't be supported.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (HIVE-18048) Support Struct type with vectorization for Parquet file
[
https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303086#comment-16303086
]
Hive QA commented on HIVE-18048:
Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12903605/HIVE-18048.001.patch
{color:green}SUCCESS:{color} +1 due to 1 test(s) being added or modified.
{color:red}ERROR:{color} -1 due to 37 failed/errored test(s), 11544 tests
executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[auto_join25] (batchId=72)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[mapjoin_hook]
(batchId=12)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=71)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[ppd_join5] (batchId=35)
org.apache.hadoop.hive.cli.TestCliDriver.testCliDriver[vector_outer_reference_windowed]
(batchId=32)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[llap_smb]
(batchId=150)
org.apache.hadoop.hive.cli.TestMiniLlapCliDriver.testCliDriver[parquet_complex_types_vectorization]
(batchId=149)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[bucketsortoptimize_insert_2]
(batchId=151)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[cbo_rp_lineage2]
(batchId=152)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[hybridgrace_hashjoin_2]
(batchId=156)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[insert_values_orig_table_use_metadata]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage2]
(batchId=163)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[lineage3]
(batchId=160)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[llap_acid_fast]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_orc_vec_part_all_complex]
(batchId=162)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vec_part_all_complex]
(batchId=158)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[schema_evol_text_vecrow_part_all_complex]
(batchId=168)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[sysdb]
(batchId=159)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_complex_all]
(batchId=164)
org.apache.hadoop.hive.cli.TestMiniLlapLocalCliDriver.testCliDriver[vector_reduce_groupby_duplicate_cols]
(batchId=158)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[authorization_part]
(batchId=93)
org.apache.hadoop.hive.cli.TestNegativeCliDriver.testCliDriver[stats_aggregator_error_1]
(batchId=93)
org.apache.hadoop.hive.cli.TestSparkCliDriver.testCliDriver[ppd_join5]
(batchId=120)
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testTransactionalValidation
(batchId=213)
org.apache.hadoop.hive.ql.TestAcidOnTez.testMapJoinOnTez (batchId=222)
org.apache.hadoop.hive.ql.io.TestDruidRecordWriter.testWrite (batchId=253)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedColumnReader.structReadSomeNull
(batchId=267)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedColumnReader.testNestedStructRead
(batchId=267)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedColumnReader.testStructRead
(batchId=267)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedDictionaryEncodingColumnReader.structReadSomeNull
(batchId=266)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedDictionaryEncodingColumnReader.testNestedStructRead
(batchId=266)
org.apache.hadoop.hive.ql.io.parquet.TestVectorizedDictionaryEncodingColumnReader.testStructRead
(batchId=266)
org.apache.hadoop.hive.ql.parse.TestReplicationScenarios.testConstraints
(batchId=225)
org.apache.hive.jdbc.TestSSL.testConnectionMismatch (batchId=231)
org.apache.hive.jdbc.TestSSL.testConnectionWrongCertCN (batchId=231)
org.apache.hive.jdbc.TestSSL.testMetastoreConnectionWrongCertCN (batchId=231)
{noformat}
Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/8373/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/8373/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-8373/
Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 37 tests failed
{noformat}
This message is automatically generated.
ATTACHMENT ID: 12903605 - PreCommit-HIVE-Build
> Support Struct type with vectorization for Parquet file
> ---
>
>
[jira] [Commented] (HIVE-18048) Support Struct type with vectorization for Parquet file
[
https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303074#comment-16303074
]
Hive QA commented on HIVE-18048:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
|| || || || {color:brown} Prechecks {color} ||
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
1s{color} | {color:blue} Findbugs executables are not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} master Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
42s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
19s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
14s{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
58s{color} | {color:green} master passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
19s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m
13s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m
38s{color} | {color:red} ql: The patch generated 3 new + 1004 unchanged - 1
fixed = 1007 total (was 1005) {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 1m
38s{color} | {color:red} root: The patch generated 3 new + 1004 unchanged - 1
fixed = 1007 total (was 1005) {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 6m
21s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 1s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Optional Tests | asflicense javac javadoc findbugs checkstyle compile |
| uname | Linux hiveptest-server-upstream 3.16.0-4-amd64 #1 SMP Debian
3.16.36-1+deb8u1 (2016-09-03) x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /data/hiveptest/working/yetus/dev-support/hive-personality.sh |
| git revision | master / 035eca3 |
| Default Java | 1.8.0_111 |
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8373/yetus/diff-checkstyle-ql.txt
|
| checkstyle |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8373/yetus/diff-checkstyle-root.txt
|
| modules | C: ql . itests U: . |
| Console output |
http://104.198.109.242/logs//PreCommit-HIVE-Build-8373/yetus.txt |
| Powered by | Apache Yetushttp://yetus.apache.org |
This message was automatically generated.
> Support Struct type with vectorization for Parquet file
> ---
>
> Key: HIVE-18048
> URL: https://issues.apache.org/jira/browse/HIVE-18048
> Project: Hive
> Issue Type: Sub-task
>Reporter: Colin Ma
>Assignee: Colin Ma
> Attachments: HIVE-18048.001.patch
>
>
> Struct type is not supported in MapWork with vectorization, it should be
> supported to improve the performance.
> The following improvements will be implemented:
> * Add fields of struct type to VectorizedRowBatchCtx.
> * Improve the VectorizedParquetRecordReader to support the struct type for
> parquet file.
> Note:
> * Orc file won't be supported.
> * Filter operator won't be supported.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Updated] (HIVE-18338) [Client, JDBC] Asynchronous interface through hive JDBC.
[ https://issues.apache.org/jira/browse/HIVE-18338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amruth S updated HIVE-18338: Attachment: HIVE-18338.patch.0 > [Client, JDBC] Asynchronous interface through hive JDBC. > > > Key: HIVE-18338 > URL: https://issues.apache.org/jira/browse/HIVE-18338 > Project: Hive > Issue Type: Improvement > Components: Clients, JDBC >Affects Versions: 2.3.2 >Reporter: Amruth S >Assignee: Amruth S >Priority: Minor > Attachments: HIVE-18338.patch.0 > > > Lot of users are struggling and rewriting a lot of boiler plate over thrift > to get pure asynchronous capability. > The idea is to expose operation handle, so that clients can persist it and > later can latch on to the same execution. > Let me know your ideas around this. We have solved this already at our org by > tweaking HiveStatement.java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18338) [Client, JDBC] Asynchronous interface through hive JDBC.
[ https://issues.apache.org/jira/browse/HIVE-18338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amruth S updated HIVE-18338: Affects Version/s: 2.3.2 Status: Patch Available (was: Open) > [Client, JDBC] Asynchronous interface through hive JDBC. > > > Key: HIVE-18338 > URL: https://issues.apache.org/jira/browse/HIVE-18338 > Project: Hive > Issue Type: Improvement > Components: Clients, JDBC >Affects Versions: 2.3.2 >Reporter: Amruth S >Assignee: Amruth S >Priority: Minor > > Lot of users are struggling and rewriting a lot of boiler plate over thrift > to get pure asynchronous capability. > The idea is to expose operation handle, so that clients can persist it and > later can latch on to the same execution. > Let me know your ideas around this. We have solved this already at our org by > tweaking HiveStatement.java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18338) [Client, JDBC] Asynchronous interface through hive JDBC.
[ https://issues.apache.org/jira/browse/HIVE-18338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Amruth S reassigned HIVE-18338: --- > [Client, JDBC] Asynchronous interface through hive JDBC. > > > Key: HIVE-18338 > URL: https://issues.apache.org/jira/browse/HIVE-18338 > Project: Hive > Issue Type: Improvement > Components: Clients, JDBC >Reporter: Amruth S >Assignee: Amruth S >Priority: Minor > > Lot of users are struggling and rewriting a lot of boiler plate over thrift > to get pure asynchronous capability. > The idea is to expose operation handle, so that clients can persist it and > later can latch on to the same execution. > Let me know your ideas around this. We have solved this already at our org by > tweaking HiveStatement.java. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18048) Support Struct type with vectorization for Parquet file
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303047#comment-16303047 ] Colin Ma commented on HIVE-18048: - [~Ferd], the initial patch is finished, can you help to review, thanks. > Support Struct type with vectorization for Parquet file > --- > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-18048.001.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > The following improvements will be implemented: > * Add fields of struct type to VectorizedRowBatchCtx. > * Improve the VectorizedParquetRecordReader to support the struct type for > parquet file. > Note: > * Orc file won't be supported. > * Filter operator won't be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18048) Support Struct type with vectorization for Parquet file
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Status: Patch Available (was: Open) > Support Struct type with vectorization for Parquet file > --- > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-18048.001.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > The following improvements will be implemented: > * Add fields of struct type to VectorizedRowBatchCtx. > * Improve the VectorizedParquetRecordReader to support the struct type for > parquet file. > Note: > * Orc file won't be supported. > * Filter operator won't be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18048) Support Struct type with vectorization for Parquet file
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Description: Struct type is not supported in MapWork with vectorization, it should be supported to improve the performance. The following improvements will be implemented: * Add fields of struct type to VectorizedRowBatchCtx. * Improve the VectorizedParquetRecordReader to support the struct type for parquet file. Note: * Orc file won't be supported. * Filter operator won't be supported. was: Struct type is not supported in MapWork with vectorization, it should be supported to improve the performance. The following improvements will be implemented: 1. Add fields of struct type to VectorizedRowBatchCtx. 2. Improve the VectorizedParquetRecordReader to support the struct type for parquet file. Note: 1. Orc file won't be supported. 2. Filter operator won't be supported. > Support Struct type with vectorization for Parquet file > --- > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-18048.001.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > The following improvements will be implemented: > * Add fields of struct type to VectorizedRowBatchCtx. > * Improve the VectorizedParquetRecordReader to support the struct type for > parquet file. > Note: > * Orc file won't be supported. > * Filter operator won't be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18048) Support Struct type with vectorization for Parquet file
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Attachment: HIVE-18048.001.patch > Support Struct type with vectorization for Parquet file > --- > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > Attachments: HIVE-18048.001.patch > > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > The following improvements will be implemented: > * Add fields of struct type to VectorizedRowBatchCtx. > * Improve the VectorizedParquetRecordReader to support the struct type for > parquet file. > Note: > * Orc file won't be supported. > * Filter operator won't be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18048) Support Struct type with vectorization for Parquet file
[ https://issues.apache.org/jira/browse/HIVE-18048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Ma updated HIVE-18048: Description: Struct type is not supported in MapWork with vectorization, it should be supported to improve the performance. The following improvements will be implemented: 1. Add fields of struct type to VectorizedRowBatchCtx. 2. Improve the VectorizedParquetRecordReader to support the struct type for parquet file. Note: 1. Orc file won't be supported. 2. Filter operator won't be supported. was:Struct type is not supported in MapWork with vectorization, it should be supported to improve the performance. > Support Struct type with vectorization for Parquet file > --- > > Key: HIVE-18048 > URL: https://issues.apache.org/jira/browse/HIVE-18048 > Project: Hive > Issue Type: Sub-task >Reporter: Colin Ma >Assignee: Colin Ma > > Struct type is not supported in MapWork with vectorization, it should be > supported to improve the performance. > The following improvements will be implemented: > 1. Add fields of struct type to VectorizedRowBatchCtx. > 2. Improve the VectorizedParquetRecordReader to support the struct type > for parquet file. > Note: 1. Orc file won't be supported. > 2. Filter operator won't be supported. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18337) upgrade pig to 0.17.0
[ https://issues.apache.org/jira/browse/HIVE-18337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18337: Status: Patch Available (was: Open) > upgrade pig to 0.17.0 > - > > Key: HIVE-18337 > URL: https://issues.apache.org/jira/browse/HIVE-18337 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > Attachments: HIVE-18337.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18337) upgrade pig to 0.17.0
[ https://issues.apache.org/jira/browse/HIVE-18337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18337: Attachment: HIVE-18337.1.patch > upgrade pig to 0.17.0 > - > > Key: HIVE-18337 > URL: https://issues.apache.org/jira/browse/HIVE-18337 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > Attachments: HIVE-18337.1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18337) upgrade pig to 0.17.0
[ https://issues.apache.org/jira/browse/HIVE-18337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18337: Attachment: (was: 1.patch) > upgrade pig to 0.17.0 > - > > Key: HIVE-18337 > URL: https://issues.apache.org/jira/browse/HIVE-18337 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-18337) upgrade pig to 0.17.0
[ https://issues.apache.org/jira/browse/HIVE-18337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang updated HIVE-18337: Attachment: 1.patch > upgrade pig to 0.17.0 > - > > Key: HIVE-18337 > URL: https://issues.apache.org/jira/browse/HIVE-18337 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > Attachments: 1.patch > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (HIVE-18324) drop database failed for not empty tables, but acid table info in metastore is still deleted
[ https://issues.apache.org/jira/browse/HIVE-18324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303019#comment-16303019 ] J.P Feng edited comment on HIVE-18324 at 12/25/17 2:22 AM: --- I think it also need to check event status in onDropTable and onDropPartition of AcidEventListener before actually dropping table and dropping partition. was (Author: snodawn): I think it should also check event status in onDropTable and onDropPartition of AcidEventListener before actually dropping table and dropping partition. > drop database failed for not empty tables, but acid table info in metastore > is still deleted > - > > Key: HIVE-18324 > URL: https://issues.apache.org/jira/browse/HIVE-18324 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 2.1.1, 2.3.2 >Reporter: J.P Feng > > I use "drop database hive_test" to drop database (without cascade) , because > it's not empty, my operation runs fail, and throws exception -> > InvalidOperationException(message:Database hive_test is not empty. One or > more tables exist.) > but acid tables info in > TXN_COMPONENTS,COMPLETED_TXN_COMPONENTS,COMPACTION_QUEUE,COMPLETED_COMPACTIONS > is still deleted by AcidEventListener. > So I advise that, onDropDatabase in AcidEventListener need to judge > DropDatabaseEvent.getStatus before delete acid table info. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (HIVE-18324) drop database failed for not empty tables, but acid table info in metastore is still deleted
[ https://issues.apache.org/jira/browse/HIVE-18324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16303019#comment-16303019 ] J.P Feng commented on HIVE-18324: - I think it should also check event status in onDropTable and onDropPartition of AcidEventListener before actually dropping table and dropping partition. > drop database failed for not empty tables, but acid table info in metastore > is still deleted > - > > Key: HIVE-18324 > URL: https://issues.apache.org/jira/browse/HIVE-18324 > Project: Hive > Issue Type: Bug > Components: Hive, Transactions >Affects Versions: 2.1.1, 2.3.2 >Reporter: J.P Feng > > I use "drop database hive_test" to drop database (without cascade) , because > it's not empty, my operation runs fail, and throws exception -> > InvalidOperationException(message:Database hive_test is not empty. One or > more tables exist.) > but acid tables info in > TXN_COMPONENTS,COMPLETED_TXN_COMPONENTS,COMPACTION_QUEUE,COMPLETED_COMPACTIONS > is still deleted by AcidEventListener. > So I advise that, onDropDatabase in AcidEventListener need to judge > DropDatabaseEvent.getStatus before delete acid table info. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18337) upgrade pig to 0.17.0
[ https://issues.apache.org/jira/browse/HIVE-18337?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saijin Huang reassigned HIVE-18337: --- > upgrade pig to 0.17.0 > - > > Key: HIVE-18337 > URL: https://issues.apache.org/jira/browse/HIVE-18337 > Project: Hive > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: Saijin Huang >Assignee: Saijin Huang >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (HIVE-18336) add Safe Mode
[ https://issues.apache.org/jira/browse/HIVE-18336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eugene Koifman reassigned HIVE-18336: - > add Safe Mode > - > > Key: HIVE-18336 > URL: https://issues.apache.org/jira/browse/HIVE-18336 > Project: Hive > Issue Type: Bug > Components: Transactions >Reporter: Eugene Koifman >Assignee: Eugene Koifman > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (HIVE-11139) Emit more lineage information
[ https://issues.apache.org/jira/browse/HIVE-11139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wang Haihua updated HIVE-11139: --- Description: # HIVE-1131 emits some column lineage info. But it doesn't support INSERT statements, or CTAS statements. It doesn't emit the predicate information either. We can enhance and use the dependency information created in HIVE-1131, generate more complete lineage info. was: HIVE-1131 emits some column lineage info. But it doesn't support INSERT statements, or CTAS statements. It doesn't emit the predicate information either. We can enhance and use the dependency information created in HIVE-1131, generate more complete lineage info. > Emit more lineage information > - > > Key: HIVE-11139 > URL: https://issues.apache.org/jira/browse/HIVE-11139 > Project: Hive > Issue Type: Improvement >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang > Fix For: 1.3.0, 2.0.0 > > Attachments: HIVE-11139.1.patch, HIVE-11139.2.patch, > HIVE-11139.3.patch > > > # HIVE-1131 emits some column lineage info. But it doesn't support INSERT > statements, or CTAS statements. It doesn't emit the predicate information > either. > We can enhance and use the dependency information created in HIVE-1131, > generate more complete lineage info. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
