[Impala-ASF-CR] IMPALA-8736: test hive parquet codec interop fails on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13781 ) Change subject: IMPALA-8736: test_hive_parquet_codec_interop fails on S3 .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8eee8ba44e6acf4a069467f749756f9040ef641 Gerrit-Change-Number: 13781 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 04 Jul 2019 00:50:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8736: test hive parquet codec interop fails on S3
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13781 ) Change subject: IMPALA-8736: test_hive_parquet_codec_interop fails on S3 .. IMPALA-8736: test_hive_parquet_codec_interop fails on S3 The test case fails (hanging untill the timeout is hit), when trying to do hive insert on S3. Support for hive insert is flaky in S3 and so the testcase should be disabled for S3. Change-Id: Ie8eee8ba44e6acf4a069467f749756f9040ef641 Reviewed-on: http://gerrit.cloudera.org:8080/13781 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M tests/custom_cluster/test_hive_parquet_codec_interop.py 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie8eee8ba44e6acf4a069467f749756f9040ef641 Gerrit-Change-Number: 13781 Gerrit-PatchSet: 4 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8729: [DOCS] Describe on-demand metadata feature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13802 ) Change subject: IMPALA-8729: [DOCS] Describe on-demand metadata feature .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/378/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13802 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64261625c1d9b122c7cca59f9b004dda05810351 Gerrit-Change-Number: 13802 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Thu, 04 Jul 2019 00:22:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8729: [DOCS] Describe on-demand metadata feature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13802 ) Change subject: IMPALA-8729: [DOCS] Describe on-demand metadata feature .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/378/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13802 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I64261625c1d9b122c7cca59f9b004dda05810351 Gerrit-Change-Number: 13802 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 23:58:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8729: [DOCS] Describe on-demand metadata feature
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13802 Change subject: IMPALA-8729: [DOCS] Describe on-demand metadata feature .. IMPALA-8729: [DOCS] Describe on-demand metadata feature - Overview of on-demand metadata. - Config flags to enable/disable on-demand metadata. Change-Id: I64261625c1d9b122c7cca59f9b004dda05810351 --- M docs/topics/impala_metadata.xml 1 file changed, 102 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/13802/1 -- To view, visit http://gerrit.cloudera.org:8080/13802 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I64261625c1d9b122c7cca59f9b004dda05810351 Gerrit-Change-Number: 13802 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] Revert "build: use thin static archives"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13798 ) Change subject: Revert "build: use thin static archives" .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 Gerrit-Change-Number: 13798 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 03 Jul 2019 23:32:07 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Bump CDP BUILD NUMBER to 1235229
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13799 ) Change subject: [WIP] Bump CDP_BUILD_NUMBER to 1235229 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3822/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d0fa856fa6c7cb1f4c9ee0e4250cfa1e8a14595 Gerrit-Change-Number: 13799 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 22:25:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3821/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 22:09:39 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Bump CDP BUILD NUMBER to 1235229
Vihang Karajgaonkar has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/13799 ) Change subject: [WIP] Bump CDP_BUILD_NUMBER to 1235229 .. [WIP] Bump CDP_BUILD_NUMBER to 1235229 The newer CDP build includes fix for HIVE-21932 which is required to fix test failures on CDP builds. Also, fixed the MetastoreEventsProcessorTest on CDP caused due to the serialization difference of the event messages between CDH and CDP builds. [WIP] Testing: 1. Build against CDP and CDH builds 2. Run the previously failing tests on CDP which were caused by HIVE-21932 3. Run full tests with CDP and CDH builds Change-Id: I6d0fa856fa6c7cb1f4c9ee0e4250cfa1e8a14595 --- M bin/impala-config.sh M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 34 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/13799/2 -- To view, visit http://gerrit.cloudera.org:8080/13799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6d0fa856fa6c7cb1f4c9ee0e4250cfa1e8a14595 Gerrit-Change-Number: 13799 Gerrit-PatchSet: 2 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Jiawei Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13801 Change subject: IMPALA-5149: Provide query profile in JSON format .. IMPALA-5149: Provide query profile in JSON format Today there is a text and Thrift version of the query profile, but it would be useful to have a JSON version for portability and machine consumption. The ultimate goal is to have a Download JSON format profile link along with the other two formats. Design Document: https://docs.google.com/document/d/ 15P_Lmjf1rlZUD4PZLXdUwTE8Lv_ME3NQ9Yf4MITzO2M/edit Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 --- M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-http-handler.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/runtime-profile-counters.h M be/src/util/runtime-profile.cc M be/src/util/runtime-profile.h M common/thrift/RuntimeProfile.thrift M www/query_profile.tmpl 11 files changed, 363 insertions(+), 11 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/13801/1 -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-5149: Provide query profile in JSON format
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13801 ) Change subject: IMPALA-5149: Provide query profile in JSON format .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13801/1/be/src/util/runtime-profile.cc File be/src/util/runtime-profile.cc: http://gerrit.cloudera.org:8080/#/c/13801/1/be/src/util/runtime-profile.cc@718 PS1, Line 718: const string& counter_name, const CounterMap& counter_map, line too long (99 > 90) -- To view, visit http://gerrit.cloudera.org:8080/13801 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8181ac818bf22207ca1deabd9220c397ae723ec1 Gerrit-Change-Number: 13801 Gerrit-PatchSet: 1 Gerrit-Owner: Jiawei Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 21:29:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] A typo fix in impala metadata
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13800 ) Change subject: [DOCS] A typo fix in impala_metadata .. [DOCS] A typo fix in impala_metadata Change-Id: I12c6a9777622e0142a17e58629bb852d8a9aa934 Reviewed-on: http://gerrit.cloudera.org:8080/13800 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_metadata.xml 1 file changed, 2 insertions(+), 2 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I12c6a9777622e0142a17e58629bb852d8a9aa934 Gerrit-Change-Number: 13800 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] A typo fix in impala metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13800 ) Change subject: [DOCS] A typo fix in impala_metadata .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/377/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12c6a9777622e0142a17e58629bb852d8a9aa934 Gerrit-Change-Number: 13800 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 20:49:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13736 ) Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 20:46:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13736 ) Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. IMPALA-8681: Only show ValidWriteIdLists for Acid tables Lists ValidWriteIds for transactional tables in profile. If a query does not trigger any transactional table loading, the query profile will not have the "Loaded ValidWriteIdLists" timeline. Tests: Manual tests. Fixed StmtMetadataLoaderTest. Added acid_profile test Sample output: Query Compilation: 3s525ms - Metadata load started: 37.369ms (37.369ms) - Metadata load finished. loaded-tables=1/1 ... - Loaded ValidWriteIdLists for transactional tables: functional.insert_only_transactional_table:0:9223372036854775807:: : 3s312ms (551.463us) - Analysis finished: 3s370ms (58.110ms) ... Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Reviewed-on: http://gerrit.cloudera.org:8080/13736 Reviewed-by: Csaba Ringhofer Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java A testdata/workloads/functional-query/queries/QueryTest/acid-profile.test M tests/common/skip.py M tests/query_test/test_acid.py 5 files changed, 46 insertions(+), 10 deletions(-) Approvals: Csaba Ringhofer: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 5 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13665 ) Change subject: IMPALA-8663 : FileMetadataLoader should skip hidden and tmp directories .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/13665/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java File fe/src/main/java/org/apache/impala/common/FileSystemUtil.java: http://gerrit.cloudera.org:8080/#/c/13665/8/fe/src/main/java/org/apache/impala/common/FileSystemUtil.java@545 PS8, Line 545: all underlying files (except which are :* in the ignored directories) > hrm, were the semantics already broken then? See the unit test AcidUtilsTes yeah, it looks like its broken currently for this case. I will file a JIRA. The AcidUtils test works because it works off the List not the ones which are actually returned by the FileMetadataLoader. Specifically, if the table is transactional we cannot rely on using listFiles API since it will skip empty directories (Or we should implement a version of it for own purposes). Not sure how to handle the S3 recursive listFiles case though. Created https://issues.apache.org/jira/browse/IMPALA-8739 as suggested -- To view, visit http://gerrit.cloudera.org:8080/13665 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2c4a22908304fe9e377d77d6c18d401c3f3294aa Gerrit-Change-Number: 13665 Gerrit-PatchSet: 8 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 03 Jul 2019 20:45:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] Bump CDP BUILD NUMBER to 1235229
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13799 ) Change subject: [WIP] Bump CDP_BUILD_NUMBER to 1235229 .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3820/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d0fa856fa6c7cb1f4c9ee0e4250cfa1e8a14595 Gerrit-Change-Number: 13799 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 20:36:30 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A typo fix in impala metadata
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13800 Change subject: [DOCS] A typo fix in impala_metadata .. [DOCS] A typo fix in impala_metadata Change-Id: I12c6a9777622e0142a17e58629bb852d8a9aa934 --- M docs/topics/impala_metadata.xml 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/13800/1 -- To view, visit http://gerrit.cloudera.org:8080/13800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I12c6a9777622e0142a17e58629bb852d8a9aa934 Gerrit-Change-Number: 13800 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] A typo fix in impala metadata
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13800 ) Change subject: [DOCS] A typo fix in impala_metadata .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12c6a9777622e0142a17e58629bb852d8a9aa934 Gerrit-Change-Number: 13800 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 20:26:31 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A typo fix in impala metadata
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13800 ) Change subject: [DOCS] A typo fix in impala_metadata .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/377/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13800 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I12c6a9777622e0142a17e58629bb852d8a9aa934 Gerrit-Change-Number: 13800 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 20:26:16 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] Bump CDP BUILD NUMBER to 1235229
Vihang Karajgaonkar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13799 Change subject: [WIP] Bump CDP_BUILD_NUMBER to 1235229 .. [WIP] Bump CDP_BUILD_NUMBER to 1235229 The newer CDP build includes fix for HIVE-21932 which is required to fix test failures on CDP builds. Also, fixed the MetastoreEventsProcessorTest on CDP caused due to the serialization difference of the event messages between CDH and CDP builds. [WIP] Testing: 1. Build against CDP and CDH builds 2. Run the previously failing tests on CDP which were caused by HIVE-21932 3. Run full tests with CDP and CDH builds Change-Id: I6d0fa856fa6c7cb1f4c9ee0e4250cfa1e8a14595 --- M bin/impala-config.sh M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/test/java/org/apache/impala/catalog/events/MetastoreEventsProcessorTest.java 4 files changed, 33 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/99/13799/1 -- To view, visit http://gerrit.cloudera.org:8080/13799 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6d0fa856fa6c7cb1f4c9ee0e4250cfa1e8a14595 Gerrit-Change-Number: 13799 Gerrit-PatchSet: 1 Gerrit-Owner: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8736: test hive parquet codec interop fails on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13781 ) Change subject: IMPALA-8736: test_hive_parquet_codec_interop fails on S3 .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3819/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8eee8ba44e6acf4a069467f749756f9040ef641 Gerrit-Change-Number: 13781 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 03 Jul 2019 19:42:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8427: [DOCS] Document the new startup flag IMPALA-7800 introduced
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13762 ) Change subject: IMPALA-8427: [DOCS] Document the new startup flag IMPALA-7800 introduced .. IMPALA-8427: [DOCS] Document the new startup flag IMPALA-7800 introduced - Added a new doc impala_client.xml as the overview of Impala impala client access. Change-Id: I1a4c1975721c32a78a003d91babc5d2bb90f3949 Reviewed-on: http://gerrit.cloudera.org:8080/13762 Tested-by: Impala Public Jenkins Reviewed-by: Michael Ho --- M docs/impala.ditamap A docs/topics/impala_client.xml M docs/topics/impala_config_options.xml 3 files changed, 153 insertions(+), 55 deletions(-) Approvals: Impala Public Jenkins: Verified Michael Ho: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/13762 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I1a4c1975721c32a78a003d91babc5d2bb90f3949 Gerrit-Change-Number: 13762 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho
[Impala-ASF-CR] IMPALA-8736: test hive parquet codec interop fails on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13781 ) Change subject: IMPALA-8736: test_hive_parquet_codec_interop fails on S3 .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8eee8ba44e6acf4a069467f749756f9040ef641 Gerrit-Change-Number: 13781 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 03 Jul 2019 19:08:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8736: test hive parquet codec interop fails on S3
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13781 ) Change subject: IMPALA-8736: test_hive_parquet_codec_interop fails on S3 .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4581/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8eee8ba44e6acf4a069467f749756f9040ef641 Gerrit-Change-Number: 13781 Gerrit-PatchSet: 3 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 03 Jul 2019 19:08:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8736: test hive parquet codec interop fails on S3
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13781 ) Change subject: IMPALA-8736: test_hive_parquet_codec_interop fails on S3 .. Patch Set 2: Code-Review+2 Thanks for fixing this -- To view, visit http://gerrit.cloudera.org:8080/13781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie8eee8ba44e6acf4a069467f749756f9040ef641 Gerrit-Change-Number: 13781 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 03 Jul 2019 19:05:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8736: test hive parquet codec interop fails on S3
Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13781 Change subject: IMPALA-8736: test_hive_parquet_codec_interop fails on S3 .. IMPALA-8736: test_hive_parquet_codec_interop fails on S3 The test case fails (hanging untill the timeout is hit), when trying to do hive insert on S3. Support for hive insert is flaky in S3 and so the testcase should be disabled for S3. Change-Id: Ie8eee8ba44e6acf4a069467f749756f9040ef641 --- M tests/custom_cluster/test_hive_parquet_codec_interop.py 1 file changed, 2 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/81/13781/2 -- To view, visit http://gerrit.cloudera.org:8080/13781 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ie8eee8ba44e6acf4a069467f749756f9040ef641 Gerrit-Change-Number: 13781 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13777 ) Change subject: IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL .. IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL - Also made a few formatting changes. - Removed the Preview Release note for Invalidation of Metadata cache. Change-Id: I36cfc7e592ed2588a8c1f8375033d60492b27a4f Reviewed-on: http://gerrit.cloudera.org:8080/13777 Reviewed-by: Vihang Karajgaonkar Tested-by: Impala Public Jenkins --- M docs/topics/impala_metadata.xml 1 file changed, 68 insertions(+), 34 deletions(-) Approvals: Vihang Karajgaonkar: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13777 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I36cfc7e592ed2588a8c1f8375033d60492b27a4f Gerrit-Change-Number: 13777 Gerrit-PatchSet: 7 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13777 ) Change subject: IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL .. Patch Set 6: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/376/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13777 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36cfc7e592ed2588a8c1f8375033d60492b27a4f Gerrit-Change-Number: 13777 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 03 Jul 2019 18:48:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13777 ) Change subject: IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13777 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36cfc7e592ed2588a8c1f8375033d60492b27a4f Gerrit-Change-Number: 13777 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 03 Jul 2019 18:39:29 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "build: use thin static archives"
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13798 ) Change subject: Revert "build: use thin static archives" .. Revert "build: use thin static archives" This reverts commit 78c55230288f8874bcd16454eb3c55277211719a. Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 Reviewed-on: http://gerrit.cloudera.org:8080/13798 Reviewed-by: Joe McDonnell Tested-by: Joe McDonnell --- M CMakeLists.txt 1 file changed, 0 insertions(+), 9 deletions(-) Approvals: Joe McDonnell: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/13798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 Gerrit-Change-Number: 13798 Gerrit-PatchSet: 2 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell
[Impala-ASF-CR] Revert "build: use thin static archives"
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13798 ) Change subject: Revert "build: use thin static archives" .. Patch Set 1: Verified+1 This is reverting a small CMake change that broke Centos 6. Impala has been building without this change for a long time (and the verification jobs are past the build stage), so I'm just going to verify and push this in. -- To view, visit http://gerrit.cloudera.org:8080/13798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 Gerrit-Change-Number: 13798 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 03 Jul 2019 18:32:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13777 ) Change subject: IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL .. Patch Set 6: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/376/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13777 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36cfc7e592ed2588a8c1f8375033d60492b27a4f Gerrit-Change-Number: 13777 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 03 Jul 2019 18:25:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13777 ) Change subject: IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml File docs/topics/impala_metadata.xml: http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml@137 PS5, Line 137: table > In this case, it actually refreshes the partition not the table Done http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml@216 PS5, Line 216: operation > Using APIs instead of operation is better in my opinion. Done http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml@218 PS5, Line 218: /tmp/spark_unpart > I think it would be easier to understand if the example provides a realisti Done http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml@258 PS5, Line 258: hive.metastore.dml.events > We should modify this text to say that this configuration should also be se Done -- To view, visit http://gerrit.cloudera.org:8080/13777 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36cfc7e592ed2588a8c1f8375033d60492b27a4f Gerrit-Change-Number: 13777 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 03 Jul 2019 18:25:43 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL
Hello Bharath Vissapragada, Vihang Karajgaonkar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13777 to look at the new patch set (#6). Change subject: IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL .. IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL - Also made a few formatting changes. - Removed the Preview Release note for Invalidation of Metadata cache. Change-Id: I36cfc7e592ed2588a8c1f8375033d60492b27a4f --- M docs/topics/impala_metadata.xml 1 file changed, 68 insertions(+), 34 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/13777/6 -- To view, visit http://gerrit.cloudera.org:8080/13777 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I36cfc7e592ed2588a8c1f8375033d60492b27a4f Gerrit-Change-Number: 13777 Gerrit-PatchSet: 6 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] Revert "build: use thin static archives"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13798 ) Change subject: Revert "build: use thin static archives" .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3818/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 Gerrit-Change-Number: 13798 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 03 Jul 2019 18:23:12 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "build: use thin static archives"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13798 ) Change subject: Revert "build: use thin static archives" .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4580/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 Gerrit-Change-Number: 13798 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 03 Jul 2019 18:04:20 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "build: use thin static archives"
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/13798 ) Change subject: Revert "build: use thin static archives" .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 Gerrit-Change-Number: 13798 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Wed, 03 Jul 2019 18:03:54 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "build: use thin static archives"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13798 ) Change subject: Revert "build: use thin static archives" .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4579/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/13798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 Gerrit-Change-Number: 13798 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 17:43:56 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "build: use thin static archives"
Joe McDonnell has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13798 Change subject: Revert "build: use thin static archives" .. Revert "build: use thin static archives" This reverts commit 78c55230288f8874bcd16454eb3c55277211719a. Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 --- M CMakeLists.txt 1 file changed, 0 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/13798/1 -- To view, visit http://gerrit.cloudera.org:8080/13798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I777ca8b4e862c8bfd8c06135834161fe91fefe67 Gerrit-Change-Number: 13798 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell
[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
Greg Rahn has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 ) Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function .. Patch Set 1: > (1 comment) I'll clarify in the commit msg and docs -- this is the restricted DL (or OSA) as you pointed out, not the full DL. -- To view, visit http://gerrit.cloudera.org:8080/13794 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d Gerrit-Change-Number: 13794 Gerrit-PatchSet: 1 Gerrit-Owner: Greg Rahn Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 17:41:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8341: [DOCS] Added a note about the requirement for existing dirs
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13793 ) Change subject: IMPALA-8341: [DOCS] Added a note about the requirement for existing dirs .. IMPALA-8341: [DOCS] Added a note about the requirement for existing dirs Change-Id: I5feff3ab7c09ee681098ec3e630977cb92f1 Reviewed-on: http://gerrit.cloudera.org:8080/13793 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_data_cache.xml 1 file changed, 3 insertions(+), 2 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13793 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I5feff3ab7c09ee681098ec3e630977cb92f1 Gerrit-Change-Number: 13793 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL
Vihang Karajgaonkar has posted comments on this change. ( http://gerrit.cloudera.org:8080/13777 ) Change subject: IMPALA-8519: [DOCS] Doc the limitation in insert events from SparkSQL .. Patch Set 5: (4 comments) http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml File docs/topics/impala_metadata.xml: http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml@137 PS5, Line 137: table In this case, it actually refreshes the partition not the table http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml@216 PS5, Line 216: operation Using APIs instead of operation is better in my opinion. http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml@218 PS5, Line 218: /tmp/spark_unpart I think it would be easier to understand if the example provides a realistic table location like /user/hive/warehouse/spark_etl.db/customers/date=01012019 http://gerrit.cloudera.org:8080/#/c/13777/5/docs/topics/impala_metadata.xml@258 PS5, Line 258: hive.metastore.dml.events We should modify this text to say that this configuration should also be set to true in the hive-site.xml used by the spark applications (typically /etc/hive/conf/hive-site.xml) so that INSERT events are generated when the spark application inserts data into existing tables and partitions. -- To view, visit http://gerrit.cloudera.org:8080/13777 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I36cfc7e592ed2588a8c1f8375033d60492b27a4f Gerrit-Change-Number: 13777 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 03 Jul 2019 17:36:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] A format fix in impala txtfile
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/13797 ) Change subject: [DOCS] A format fix in impala_txtfile .. [DOCS] A format fix in impala_txtfile Change-Id: I63c2d4dfb4985bc2560fb559bee3f4987d415405 Reviewed-on: http://gerrit.cloudera.org:8080/13797 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_txtfile.xml 1 file changed, 2 insertions(+), 3 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/13797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I63c2d4dfb4985bc2560fb559bee3f4987d415405 Gerrit-Change-Number: 13797 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] A format fix in impala txtfile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13797 ) Change subject: [DOCS] A format fix in impala_txtfile .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/375/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/13797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63c2d4dfb4985bc2560fb559bee3f4987d415405 Gerrit-Change-Number: 13797 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 17:20:05 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A format fix in impala txtfile
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/13797 Change subject: [DOCS] A format fix in impala_txtfile .. [DOCS] A format fix in impala_txtfile Change-Id: I63c2d4dfb4985bc2560fb559bee3f4987d415405 --- M docs/topics/impala_txtfile.xml 1 file changed, 2 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/13797/1 -- To view, visit http://gerrit.cloudera.org:8080/13797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I63c2d4dfb4985bc2560fb559bee3f4987d415405 Gerrit-Change-Number: 13797 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] A format fix in impala txtfile
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/13797 ) Change subject: [DOCS] A format fix in impala_txtfile .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/13797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63c2d4dfb4985bc2560fb559bee3f4987d415405 Gerrit-Change-Number: 13797 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 17:08:30 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A format fix in impala txtfile
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13797 ) Change subject: [DOCS] A format fix in impala_txtfile .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/375/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/13797 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I63c2d4dfb4985bc2560fb559bee3f4987d415405 Gerrit-Change-Number: 13797 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 17:08:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13736 ) Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3817/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 15:13:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13559 ) Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables .. Patch Set 11: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/3816/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/13559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad Gerrit-Change-Number: 13559 Gerrit-PatchSet: 11 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 15:07:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13736 ) Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4578/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 15:01:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13736 ) Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. Patch Set 4: Code-Review+2 Thanks for the changes! -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 14:47:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 ) Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc@4003 PS1, Line 4003: TestErrorString("damerau_levenshtein('z', repeat('x', 256))", : "damerau-levenshtein argument exceeds maximum length of 255 characters\n"); : TestErrorString("damerau_levenshtein(repeat('x', 256), 'z')", : "damerau-levenshtein argument exceeds maximum length of 255 characters\n"); It could be useful to test the 255*255 case to ensure that there are no allocation issues. http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc File be/src/exprs/string-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc@1177 PS1, Line 1177: int d[s1len + 1][s2len + 1]; This worries me a bit: 1. The s1len > 255 || s2len > 255 does not protect from the allocation itself, so it is possible that a call with huge strings will lead to stack overflow. A solution that seems safe enough to me is creating an inner function that is called after the check and declare / use the array there, 2. Even if we exclude the s1len > 255 || s2len > 255 case, 64K on the stack seems too much to me, StringFunctions::Levenshtein uses dynamic allocation for much shorter arrays. -- To view, visit http://gerrit.cloudera.org:8080/13794 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d Gerrit-Change-Number: 13794 Gerrit-PatchSet: 1 Gerrit-Owner: Greg Rahn Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 14:42:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13736 ) Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. Patch Set 4: (1 comment) The comment for skipping is added. http://gerrit.cloudera.org:8080/#/c/13736/2/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/13736/2/tests/query_test/test_acid.py@55 PS2, Line 55: # that Catalog loads tables by itself, the q > Can you add this information in a comment? Done -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 14:34:31 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Hello Zoltan Borok-Nagy, Todd Lipcon, Csaba Ringhofer, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13736 to look at the new patch set (#4). Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. IMPALA-8681: Only show ValidWriteIdLists for Acid tables Lists ValidWriteIds for transactional tables in profile. If a query does not trigger any transactional table loading, the query profile will not have the "Loaded ValidWriteIdLists" timeline. Tests: Manual tests. Fixed StmtMetadataLoaderTest. Added acid_profile test Sample output: Query Compilation: 3s525ms - Metadata load started: 37.369ms (37.369ms) - Metadata load finished. loaded-tables=1/1 ... - Loaded ValidWriteIdLists for transactional tables: functional.insert_only_transactional_table:0:9223372036854775807:: : 3s312ms (551.463us) - Analysis finished: 3s370ms (58.110ms) ... Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 --- M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/test/java/org/apache/impala/analysis/StmtMetadataLoaderTest.java A testdata/workloads/functional-query/queries/QueryTest/acid-profile.test M tests/common/skip.py M tests/query_test/test_acid.py 5 files changed, 46 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/36/13736/4 -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13559 ) Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables .. Patch Set 10: (13 comments) http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.h File be/src/exec/hdfs-table-sink.h: http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.h@264 PS10, Line 264: long > int64_t? Done http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.cc File be/src/exec/hdfs-table-sink.cc: http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.cc@500 PS10, Line 500: // However, for transactional tables we should create a new empty base directory. > Why? I assume there is some good reason but it's not immediately obvious to Sorry, I left out some information. We only need to do that in case of INSERT OVERWRITEs with empty result sets. For insert-only ACID tables we don't delete old files, we just create a new base directory. Extended the comment and modified the condition. http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/exec/hdfs-table-sink.cc@700 PS10, Line 700: if (IsTransactional()) return true; > This one seems more obvious to me, but makes me think that the class commen Extended the class comment. http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/runtime/coordinator.cc File be/src/runtime/coordinator.cc: http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/runtime/coordinator.cc@571 PS10, Line 571: Status Coordinator::FinalizeHdfsInsert() { > I think we should probably do the transaction abort in this function, since Thanks, done. http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/runtime/coordinator.cc@571 PS10, Line 571: FinalizeHdfsInsert > We should maybe rename this to FinalizeHdfsDml(), now or later. Done http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.h File be/src/service/client-request-state.h: http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.h@307 PS10, Line 307: /// True if there is an open transaction. > Hive ACID transaction, just to be clear. Since then removed this bool because it caused more issues than solved because the handover of the transaction ownership between the FE and BE should happen immediately. Now I'm using a member of FinalizeParams to detect whether we are in a transaction or not although it might not be the cleanest solution. http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.h@308 PS10, Line 308: bool in_transaction_ = false; > I think this would best fit in DmlExecState. We'll have transactions for queries as well. http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.cc File be/src/service/client-request-state.cc: http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.cc@720 PS10, Line 720: DCHECK(exec_request().__isset.transaction_id); > I think we would prefer to abort the transaction earlier in the query lifec Thanks for the suggestion, moved it to FinalizeHdfsDml() http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/client-request-state.cc@820 PS10, Line 820: //TODO: HMS is not updated at this point, only in UpdateCatalog(). But > Zoli explained this to me out-of-band. I think I agree that having the cata Added commit to catalog. Was also thinking about adding abort as well, but the error handling in CatalogOpExecutor.UpdateCatalog() was inconsistent (sometimes an exception is thrown, sometimes we set an error in the result object), and in some cases we still need to abort the transaction from the BE. http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/frontend.h File be/src/service/frontend.h: http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/frontend.h@170 PS10, Line 170: /// Commits transaction with the given transaction id. > Are there any invariants we should document for these methods? I guess we'r Removed it since now we commit through the Catalog. http://gerrit.cloudera.org:8080/#/c/13559/10/be/src/service/frontend.h@171 PS10, Line 171: long > I think we want to use int64_t, to match the thrift definition and our styl Done. I think I got used to using longs in Java code. http://gerrit.cloudera.org:8080/#/c/13559/10/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java File fe/src/main/java/org/apache/impala/analysis/InsertStmt.java: http://gerrit.cloudera.org:8080/#/c/13559/10/fe/src/main/java/org/apache/impala/analysis/InsertStmt.java@179 PS10, Line 179: private Long writeId_; > This is just a nit, but it would be slightly more intuitive to me to use -1 Done http://gerrit.cloudera.org:8080/#/c/13559/10/testdata/workloads/functional-query/queries/QueryTest/acid-insert.test File testdata/workloads/functional-query/queries/QueryTest/acid-insert.test:
[Impala-ASF-CR] IMPALA-8636: Implement INSERT for insert-only ACID tables
Hello Csaba Ringhofer, Todd Lipcon, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13559 to look at the new patch set (#11). Change subject: IMPALA-8636: Implement INSERT for insert-only ACID tables .. IMPALA-8636: Implement INSERT for insert-only ACID tables This commit adds INSERT support for insert-only ACID tables. The Frontend opens a transaction for queries that refer to transactional tables. For INSERT statements that write insert-only ACID tables it also allocates a write ID. The Frontend aborts the transaction if an error occurs during analysis/planning. The Backend gets the transaction id in TExecRequestState and the write id is set for the HDFS table sinks. The sinks write the files at their final destination which is an ACID base/delta directory. There is no need for finalization of transactional INSERTS. ClientRequestState commits the transaction in WaitInternal() if everything went well. If the transaction is still open in Done(), it means there was an error, therefore the transaction needs to be aborted. The Backend commits/aborts the transaction by calling the Frontend via JNI. Testing: * added new tables during dataload * added acid-insert.test file with INSERT statements against the new tables * added integration test with Hive to test_hms_integration.py. The test inserts data with Impala and reads with Hive. (These integration tests only run with exhaustive exploration strategy) TODO in following commits: * add locks and heartbeats * implement TRUNCATE (maybe in another commit) * CTAS creates files in the 'root' directory of the table/partition. It is handled correctly during SELECT, but would be better to create a base directory from the beginning. Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad --- M be/src/exec/hdfs-table-sink.cc M be/src/exec/hdfs-table-sink.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/frontend.cc M be/src/service/frontend.h M be/src/util/jni-util.h M common/thrift/CatalogService.thrift M common/thrift/DataSinks.thrift M common/thrift/Frontend.thrift M common/thrift/ImpalaInternalService.thrift M fe/src/compat-hive-2/java/org/apache/impala/compat/MetastoreShim.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/common/TransactionException.java M fe/src/main/java/org/apache/impala/planner/HdfsTableSink.java M fe/src/main/java/org/apache/impala/planner/TableSink.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M testdata/datasets/functional/functional_schema_template.sql M testdata/datasets/functional/schema_constraints.csv A testdata/workloads/functional-query/queries/QueryTest/acid-insert.test M tests/metadata/test_hms_integration.py M tests/query_test/test_insert.py 28 files changed, 765 insertions(+), 126 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/13559/11 -- To view, visit http://gerrit.cloudera.org:8080/13559 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id6c36fa6902676f06b4e38730f737becfc7c06ad Gerrit-Change-Number: 13559 Gerrit-PatchSet: 11 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8681: Only show ValidWriteIdLists for Acid tables
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13736 ) Change subject: IMPALA-8681: Only show ValidWriteIdLists for Acid tables .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/13736/2/tests/query_test/test_acid.py File tests/query_test/test_acid.py: http://gerrit.cloudera.org:8080/#/c/13736/2/tests/query_test/test_acid.py@55 PS2, Line 55: @SkipIfCatalogV2.hms_event_polling_enabled() > When local catalogV2 combines with hms_enent_polling enabled, it seems cata Can you add this information in a comment? -- To view, visit http://gerrit.cloudera.org:8080/13736 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcc31c7ddcfc471b0e5308f7e4aaadfa8189a905 Gerrit-Change-Number: 13736 Gerrit-PatchSet: 2 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 13:09:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/13722/2/be/src/runtime/datetime-iso-sql-format-parser.h File be/src/runtime/datetime-iso-sql-format-parser.h: http://gerrit.cloudera.org:8080/#/c/13722/2/be/src/runtime/datetime-iso-sql-format-parser.h@39 PS2, Line 39: ParseDateTime I could be mistaken but I couldn't find any BE-tests for this function. http://gerrit.cloudera.org:8080/#/c/13722/2/be/src/runtime/datetime-iso-sql-format-tokenizer.h File be/src/runtime/datetime-iso-sql-format-tokenizer.h: http://gerrit.cloudera.org:8080/#/c/13722/2/be/src/runtime/datetime-iso-sql-format-tokenizer.h@51 PS2, Line 51: Tokenize I could be mistaken but I couldn't find any BE-tests for this and the other functions in this class. http://gerrit.cloudera.org:8080/#/c/13722/2/be/src/runtime/datetime-simple-date-format-parser.h File be/src/runtime/datetime-simple-date-format-parser.h: http://gerrit.cloudera.org:8080/#/c/13722/2/be/src/runtime/datetime-simple-date-format-parser.h@68 PS2, Line 68: /// Constants to hold default format lengths. Maybe the stuff in this file could be moved to separate classes, like SimpleDateFormatParser and SimpleDateFormatTokenizer ? I think that naming would make the distinction between code dealing with simple-date format and code dealing with iso-sql format cleaner. -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 13:04:39 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/13558 ) Change subject: IMPALA-8593: Prohibit write operations for bucketed tables .. Patch Set 4: Code-Review+2 I give +2, but I would prefer to run https://jenkins.impala.io/job/ubuntu-16.04-from-scratch-cdp-hive/ with the change before merging it. The cdp build is still red, but there are only a few ( 3 ) errors left on master. -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 12:56:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/13722 ) Change subject: IMPALA-8703: ISO:SQL:2016 datetime patterns - Milestone 1 .. Patch Set 2: (7 comments) http://gerrit.cloudera.org:8080/#/c/13722/2/fe/src/main/java/org/apache/impala/analysis/CastExpr.java File fe/src/main/java/org/apache/impala/analysis/CastExpr.java: http://gerrit.cloudera.org:8080/#/c/13722/2/fe/src/main/java/org/apache/impala/analysis/CastExpr.java@51 PS2, Line 51: private final String castFormat_; Add a comment to describe the new member. http://gerrit.cloudera.org:8080/#/c/13722/2/fe/src/main/java/org/apache/impala/analysis/CastExpr.java@208 PS2, Line 208: castFormat_ What if 'castFormat_' includes an apostrophe? http://gerrit.cloudera.org:8080/#/c/13722/2/fe/src/main/java/org/apache/impala/analysis/CastExpr.java@226 PS2, Line 226: boolean isIntermediateStep = : type_.getPrimitiveType() == PrimitiveType.CHAR && : children_.get(0).getType().getPrimitiveType() == PrimitiveType.STRING; The naming of this variable is a bit confusing. 'isIntermediateStep' is set to true even if we have a simple CAST(string_col AS char(20)) expression. I would just put the right-hand side directly into the if condition. http://gerrit.cloudera.org:8080/#/c/13722/2/fe/src/main/java/org/apache/impala/analysis/CastExpr.java@280 PS2, Line 280: if (null != castFormat_ && !isIntermediateStepNeeded) { : if (!(type_.isDateOrTimeType() && getChild(0).getType().isStringType()) && : !(type_.isStringType() && getChild(0).getType().isDateOrTimeType())) { : // FORMAT clause works only for casting between date types and string types : throw new AnalysisException("FORMAT clause is not applicable from " + : getChild(0).getType() + " to " + type_); : } : if (castFormat_.isEmpty()) { : throw new AnalysisException("FORMAT clause can't be empty"); : } : } This block can be used as the else branch of he previous 'if' statement. And then you don't need to introduce the 'isIntermediateStepNeeded' variable. http://gerrit.cloudera.org:8080/#/c/13722/2/testdata/workloads/functional-query/queries/QueryTest/date.test File testdata/workloads/functional-query/queries/QueryTest/date.test: http://gerrit.cloudera.org:8080/#/c/13722/2/testdata/workloads/functional-query/queries/QueryTest/date.test@629 PS2, Line 629: # : # QUERY : #select cast("2014/-; ,11'/:05" as date format "-MM-DD"); : # RESULTS : #2014-11-05 : # TYPES : #DATE This should work right? Any reason it is commented out? http://gerrit.cloudera.org:8080/#/c/13722/2/testdata/workloads/functional-query/queries/QueryTest/date.test@841 PS2, Line 841: Extra space http://gerrit.cloudera.org:8080/#/c/13722/2/tests/query_test/test_cast_with_format.py File tests/query_test/test_cast_with_format.py: http://gerrit.cloudera.org:8080/#/c/13722/2/tests/query_test/test_cast_with_format.py@22 PS2, Line 22: TestCastWithFormat Could the tests in this script be moved to cast_format.test? Or to another '.test' file? -- To view, visit http://gerrit.cloudera.org:8080/13722 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I19d8d097a45ae6f103b6cd1b2d81aad38dfd9e23 Gerrit-Change-Number: 13722 Gerrit-PatchSet: 2 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 03 Jul 2019 12:50:07 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 ) Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc@3999 PS1, Line 3999: TestValue("damerau_levenshtein('CA', 'ABC')", TYPE_INT, 3); Based on https://en.wikipedia.org/wiki/Damerau%E2%80%93Levenshtein_distance#Algorithm the used algorithm computes "optimal string alignment distance" or "restricted edit distance" which is not the "true" Damerau–Levenshtein distance because that should return 2 in this case (CA -> AC -> ABC). I'm not sure how to resolve it, I think you know it best, but I think at least some comments needed to clarify the semantics. -- To view, visit http://gerrit.cloudera.org:8080/13794 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d Gerrit-Change-Number: 13794 Gerrit-PatchSet: 1 Gerrit-Owner: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 12:37:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8593: Prohibit write operations for bucketed tables
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13558 ) Change subject: IMPALA-8593: Prohibit write operations for bucketed tables .. Patch Set 4: Hive will treat all acid table as managed and all managed are needed to be transactional. Impala created tables will be external tables. The changing and discussion is still going on. This patch is just first step, we can make more fixes later. Following are related docs for HMS translation layer: Hive: https://docs.google.com/document/d/1ZmtMlKvnSz-EFGuCuYfPVA-2kfgHOWenL8sW47vkhnw/edit# Impala: https://docs.google.com/document/d/1lgcD_ZqIFVydLdRPQhkYBdFu2C0mH1A8VnXpcSs9psA/edit#heading=h.l9b51051hhma Three engines: https://docs.google.com/document/d/1fPIn1WW-tRRSmM-aGEVl0DI-TByIswo5HrLdgSPcw9s/edit?ts=5d1baa75 -- To view, visit http://gerrit.cloudera.org:8080/13558 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia08d01168660830b6e0d08b55a95eac129889cec Gerrit-Change-Number: 13558 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Sudhanshu Arora Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Reviewer: Yongzhi Chen Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Wed, 03 Jul 2019 12:14:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function
Zoltan Borok-Nagy has posted comments on this change. ( http://gerrit.cloudera.org:8080/13794 ) Change subject: IMPALA-8709: Add Damerau-Levenshtein edit distance built-in function .. Patch Set 1: Code-Review+1 (2 comments) Found some nits, but LGTM. http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc File be/src/exprs/expr-test.cc: http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/expr-test.cc@3984 PS1, Line 3984: TestValue("damerau_levenshtein('', '')", TYPE_INT, 0); : TestValue("damerau_levenshtein('abc', 'abc')", TYPE_INT, 0); : TestValue("damerau_levenshtein('a', 'b')", TYPE_INT, 1); : TestValue("damerau_levenshtein('a', '')", TYPE_INT, 1); : TestValue("damerau_levenshtein('aabc', 'abc')", TYPE_INT, 1); : TestValue("damerau_levenshtein('abcc', 'abc')", TYPE_INT, 1); : TestValue("damerau_levenshtein('', 'a')", TYPE_INT, 1); : TestValue("damerau_levenshtein('abc', 'abcc')", TYPE_INT, 1); : TestValue("damerau_levenshtein('abc', 'aabc')", TYPE_INT, 1); : TestValue("damerau_levenshtein('teh', 'the')", TYPE_INT, 1); : TestValue("damerau_levenshtein('tets', 'test')", TYPE_INT, 1); : TestValue("damerau_levenshtein('fuor', 'four')", TYPE_INT, 1); : TestValue("damerau_levenshtein('kitten', 'sitting')", TYPE_INT, 3); : TestValue("damerau_levenshtein('Saturday', 'Sunday')", TYPE_INT, 3); : TestValue("damerau_levenshtein('rosettacode', 'raisethysword')", TYPE_INT, 8); : TestValue("damerau_levenshtein('CA', 'ABC')", TYPE_INT, 3); : TestIsNull("damerau_levenshtein('foo', NULL)", TYPE_INT); : TestIsNull("damerau_levenshtein(NULL, 'foo')", TYPE_INT); : TestIsNull("damerau_levenshtein(NULL, NULL)", TYPE_INT); : TestErrorString("damerau_levenshtein('z', repeat('x', 256))", : "damerau-levenshtein argument exceeds maximum length of 255 characters\n"); : TestErrorString("damerau_levenshtein(repeat('x', 256), 'z')", : "damerau-levenshtein argument exceeds maximum length of 255 characters\n"); : : TestValue("dle_dst('', '')", TYPE_INT, 0); : TestValue("dle_dst('abc', 'abc')", TYPE_INT, 0); : TestValue("dle_dst('a', 'b')", TYPE_INT, 1); : TestValue("dle_dst('a', '')", TYPE_INT, 1); : TestValue("dle_dst('aabc', 'abc')", TYPE_INT, 1); : TestValue("dle_dst('abcc', 'abc')", TYPE_INT, 1); : TestValue("dle_dst('', 'a')", TYPE_INT, 1); : TestValue("dle_dst('abc', 'abcc')", TYPE_INT, 1); : TestValue("dle_dst('abc', 'aabc')", TYPE_INT, 1); : TestValue("dle_dst('teh', 'the')", TYPE_INT, 1); : TestValue("dle_dst('tets', 'test')", TYPE_INT, 1); : TestValue("dle_dst('fuor', 'four')", TYPE_INT, 1); : TestValue("dle_dst('kitten', 'sitting')", TYPE_INT, 3); : TestValue("dle_dst('Saturday', 'Sunday')", TYPE_INT, 3); : TestValue("dle_dst('rosettacode', 'raisethysword')", TYPE_INT, 8); : TestValue("dle_dst('CA', 'ABC')", TYPE_INT, 3); : TestIsNull("dle_dst('foo', NULL)", TYPE_INT); : TestIsNull("dle_dst(NULL, 'foo')", TYPE_INT); : TestIsNull("dle_dst(NULL, NULL)", TYPE_INT); : TestErrorString("dle_dst('z', repeat('x', 256))", : "damerau-levenshtein argument exceeds maximum length of 255 characters\n"); : TestErrorString("dle_dst(repeat('x', 256), 'z')", : "damerau-levenshtein argument exceeds maximum length of 255 characters\n"); nit: instead of duplicating maybe you could just write something like for (string fn_name : {"damerau_levenshtein", "dle_dst"}) { TestValue(fn_name + "('', '')" ...); ... } http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc File be/src/exprs/string-functions-ir.cc: http://gerrit.cloudera.org:8080/#/c/13794/1/be/src/exprs/string-functions-ir.cc@1218 PS1, Line 1218: d[i][j] = std::min(d[i][j], :d[i - 2][j - 2] + l_cost // transposition : ); nit: fits single line -- To view, visit http://gerrit.cloudera.org:8080/13794 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib759817ec15e7075bf49d51e494e45c8af4db94d Gerrit-Change-Number: 13794 Gerrit-PatchSet: 1 Gerrit-Owner: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13765 ) Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3815/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 03 Jul 2019 09:47:16 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Yongzhi Chen has posted comments on this change. ( http://gerrit.cloudera.org:8080/13765 ) Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. Patch Set 3: (5 comments) Answer Sahil's question: IsNull Check may be necessary for other places, but use IsNullPredicate in requiresNullWrapping is not required. We can workaround it. Attach patch 4 with fixed tests. Run all core tests. http://gerrit.cloudera.org:8080/#/c/13765/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13765/3//COMMIT_MSG@16 PS3, Line 16: cannot both be null and non-null. But during its > This line is a bit confusing. Can you clarify what you mean here? Although we do not have the predict to tell if a value is null, we know that a valid should be either null or nonnull and cannot be both. http://gerrit.cloudera.org:8080/#/c/13765/3//COMMIT_MSG@17 PS3, Line 17: checking whether the wrapper needed, because of code > what do you mean by "because of code limitation"? We do not have IsNullPredicate for complex type, it is mentioned in the latter part of the message. I will make a change to add it here too. http://gerrit.cloudera.org:8080/#/c/13765/3/testdata/workloads/functional-query/queries/QueryTest/complex_joins.test File testdata/workloads/functional-query/queries/QueryTest/complex_joins.test: http://gerrit.cloudera.org:8080/#/c/13765/3/testdata/workloads/functional-query/queries/QueryTest/complex_joins.test@5 PS3, Line 5: create table array_tbl (col1 string, col2 array ) stored as parquet; > databases such as "functional_parquet" already have tables such as "complex This is the test case that the customer provided. And our functional table does not have data. http://gerrit.cloudera.org:8080/#/c/13765/3/testdata/workloads/functional-query/queries/QueryTest/complex_joins.test@35 PS3, Line 35: select t.col1, rank() over(order by t.col1) as rnk > is the analytical function necessary to re-produce the error? otherwise I t Yes, it is the customer's case, I must make sure the tests passed with/without data. Why create two: The testing framework parallels the tests which make test flaky if switching between impala and hive. http://gerrit.cloudera.org:8080/#/c/13765/3/tests/query_test/test_complex.py File tests/query_test/test_complex.py: http://gerrit.cloudera.org:8080/#/c/13765/3/tests/query_test/test_complex.py@23 PS3, Line 23: class TestComplex(ImpalaTestSuite): > You can just add this test to test_nested_types.py instead of creating a ne I will move -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 3 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen Gerrit-Comment-Date: Wed, 03 Jul 2019 09:11:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column
Hello Bharath Vissapragada, Sahil Takiar, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13765 to look at the new patch set (#4). Change subject: IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column .. IMPALA-8718: Fixed AnalysisException in inline view with outer join complex type column For Inline view with left outer join, in order to make right side exprs nullable, the planner makes a wrapper for the expr that return non-null when all its contained slotref is null. The wrap is TupleIsNullPredicate. Complex type does not need the wrapper for its value cannot both be null and non-null. But during its checking whether the wrapper needed, because of code limitation(IsNullPredicate is not supported for complex types), AnalysisException is thrown. This fix overcomes the limitation by avoiding to use unsupported IsNullPredicate for complex type. It should be a safe change for: 1. Other data types such as int return false after a call to backend to check isNotNull(nullvalue). 2.Left outer join with complex type works fine for the queries without inline view. Join code can handle complex type correctly without the wrapper. Tests: Added ee tests to test left outer joins. Manual tests. Ran all core tests. Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 --- M fe/src/main/java/org/apache/impala/analysis/TupleIsNullPredicate.java A testdata/workloads/functional-query/queries/QueryTest/complex_joins.test M tests/query_test/test_nested_types.py 3 files changed, 51 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/65/13765/4 -- To view, visit http://gerrit.cloudera.org:8080/13765 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ida65503ca4b1342b8fe0049753bc664da227dca9 Gerrit-Change-Number: 13765 Gerrit-PatchSet: 4 Gerrit-Owner: Yongzhi Chen Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Yongzhi Chen
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/3814/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 03 Jul 2019 07:03:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 5: (2 comments) http://gerrit.cloudera.org:8080/#/c/13746/5/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/13746/5/shell/impala_client.py@392 PS5, Line 392: n flake8: E501 line too long (93 > 90 characters) http://gerrit.cloudera.org:8080/#/c/13746/5/tests/run-tests.py File tests/run-tests.py: http://gerrit.cloudera.org:8080/#/c/13746/5/tests/run-tests.py@233 PS5, Line 233: \ flake8: E502 the backslash is redundant between brackets -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 03 Jul 2019 06:24:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/13746 ) Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. Patch Set 5: (6 comments) http://gerrit.cloudera.org:8080/#/c/13746/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/13746/4//COMMIT_MSG@17 PS4, Line 17: impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) > I kinda feel like we should be able to enable this with a single flag. That Good point. Makes it simpler. http://gerrit.cloudera.org:8080/#/c/13746/4/be/src/service/impala-server.cc File be/src/service/impala-server.cc: http://gerrit.cloudera.org:8080/#/c/13746/4/be/src/service/impala-server.cc@2080 PS4, Line 2080: << " to server " << connection_context.server_name << " with " > I think it might still be useful to log the number of associated sessions e Done. Removed L2085 since we have "closing session" logs for each session anyway. http://gerrit.cloudera.org:8080/#/c/13746/4/shell/impala_client.py File shell/impala_client.py: http://gerrit.cloudera.org:8080/#/c/13746/4/shell/impala_client.py@367 PS4, Line 367: if connect_timeout_ms > 0: > nit: prefer format(), e.g. Done http://gerrit.cloudera.org:8080/#/c/13746/4/shell/impala_client.py@377 PS4, Line 377: # THttpClient relies on the URI scheme (http vs https) to open an appropriate > Same comment about format() Done http://gerrit.cloudera.org:8080/#/c/13746/4/shell/impala_client.py@379 PS4, Line 379: if self.use_ssl: > Same comment about format() Done http://gerrit.cloudera.org:8080/#/c/13746/4/shell/impala_client.py@383 PS4, Line 383: else: > Same comment about format() Done -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 03 Jul 2019 06:24:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8717: impala-shell support for HS2 HTTP endpoint
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/13746 to look at the new patch set (#5). Change subject: IMPALA-8717: impala-shell support for HS2 HTTP endpoint .. IMPALA-8717: impala-shell support for HS2 HTTP endpoint Adds impala-shell support to connect to HiveServer2 HTTP endpoint. Relies on toolchain change at https://gerrit.cloudera.org/#/c/13725/. Use --protocol='hs2-http' to enable this behavior. Example usages: --- impala-shell --protocol='hs2-http' (No auth) impala-shell --protocol='hs2-http' --ldap -u. (PLAIN auth) impala-shell --protocol-'hs2-http' --ssl --ca_cert... (TLS) impala-shell --protocol='hs2-http' --ldap --ssl --ca_cert... (LDAP + TLS) Limitations: --- - Does not support Kerberos (-k) due to lack ot SPNEGO support. Testing: - Parameterized existing shell tests to support this combination. - Manually tested plain LDAP and LDAP + TLS combinations due to lack of LDAP support in tests. Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 --- M be/src/service/impala-server.cc M bin/impala-config.sh M shell/impala_client.py M shell/impala_shell.py M shell/option_parser.py M tests/common/impala_cluster.py M tests/common/impala_service.py M tests/common/impala_test_suite.py M tests/common/test_dimensions.py M tests/conftest.py M tests/custom_cluster/test_client_ssl.py M tests/run-tests.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py M tests/shell/util.py 15 files changed, 134 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/13746/5 -- To view, visit http://gerrit.cloudera.org:8080/13746 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8323950857dfe1c1dfd5377fde79f87bc2ce9534 Gerrit-Change-Number: 13746 Gerrit-PatchSet: 5 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong