[Impala-ASF-CR] IMPALA-8291: Show constraints in DESCRIBE FORMATTED
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16428 ) Change subject: IMPALA-8291: Show constraints in DESCRIBE FORMATTED .. IMPALA-8291: Show constraints in DESCRIBE FORMATTED Support for displaying primary and foreign key constraints in describe formatted output. The output attempts to be as close to Hive's implementation as possible. Also includes constraint definitions for the TPC-DS test workload. Testing: * Fresh load of testdata * Metadata query tests comparing the output between Impala and Hive Change-Id: I676b69c465c46491f870d7fdc894e7474c030356 Reviewed-on: http://gerrit.cloudera.org:8080/16428 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/compat-hive-3/java/org/apache/impala/compat/HiveMetadataFormatUtils.java M fe/src/compat-hive-3/java/org/apache/impala/compat/MetastoreShim.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java M testdata/datasets/tpcds/tpcds_schema_template.sql M tests/metadata/test_metadata_query_statements.py 5 files changed, 788 insertions(+), 533 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/16428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I676b69c465c46491f870d7fdc894e7474c030356 Gerrit-Change-Number: 16428 Gerrit-PatchSet: 4 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Vihang Karajgaonkar
[Impala-ASF-CR] IMPALA-8291: Show constraints in DESCRIBE FORMATTED
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16428 ) Change subject: IMPALA-8291: Show constraints in DESCRIBE FORMATTED .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b69c465c46491f870d7fdc894e7474c030356 Gerrit-Change-Number: 16428 Gerrit-PatchSet: 3 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 17 Sep 2020 05:25:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10078: Proper codegen for KuduPartitionExpr
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/16419 ) Change subject: IMPALA-10078: Proper codegen for KuduPartitionExpr .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16419/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16419/3//COMMIT_MSG@12 PS3, Line 12: - TODO: I don't think we have good test coverage on this, I tried to > Would it be possible to write a test to call both the non-llvm and the llvm I think it would be simplest if the number of partitions and impalads was the same. That's what I'm assuming about the hash function, yeah. -- To view, visit http://gerrit.cloudera.org:8080/16419 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ifcae34f71b407837e2c5f1b97aa230e490a268df Gerrit-Change-Number: 16419 Gerrit-PatchSet: 3 Gerrit-Owner: Daniel Becker Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Daniel Becker Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Thomas Tauber-Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 17 Sep 2020 03:32:48 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 21: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6436/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 21 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 17 Sep 2020 03:30:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 21: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 21 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 17 Sep 2020 03:30:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16406 ) Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. Patch Set 8: Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6434/ -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 17 Sep 2020 03:24:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16449 ) Change subject: IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7189/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16449 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15f256b4e3f5206c7140746694106e03b0a4ad92 Gerrit-Change-Number: 16449 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 17 Sep 2020 03:09:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16449 ) Change subject: IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page .. Patch Set 3: (14 comments) http://gerrit.cloudera.org:8080/#/c/16449/2/be/src/catalog/catalog-json.cc File be/src/catalog/catalog-json.cc: http://gerrit.cloudera.org:8080/#/c/16449/2/be/src/catalog/catalog-json.cc@134 PS2, Line 134: Value type_name(_TTableType_VALUES_TO_NAMES.find(table.table_type)->second, > line too long (91 > 90) Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@349 PS2, Line 349: t > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@352 PS2, Line 352: t > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@363 PS2, Line 363: t > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@367 PS2, Line 367: " > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@373 PS2, Line 373: t > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@377 PS2, Line 377: s > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@379 PS2, Line 379: o > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@381 PS2, Line 381: " > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@382 PS2, Line 382: n > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@383 PS2, Line 383: o > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@384 PS2, Line 384: a > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@385 PS2, Line 385: > flake8: W601 .has_key() is deprecated, use 'in' Done http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@386 PS2, Line 386: a > flake8: W601 .has_key() is deprecated, use 'in' Done -- To view, visit http://gerrit.cloudera.org:8080/16449 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15f256b4e3f5206c7140746694106e03b0a4ad92 Gerrit-Change-Number: 16449 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Thu, 17 Sep 2020 02:48:38 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16449 to look at the new patch set (#3). Change subject: IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page .. IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page Catalogd has a debug page at '/catalog_object' showing catalog objects in thrift debug strings. It's inconvenient for tests to parse the thrift string and get interested infos. This patch extend this page to support returning JSON results, which eases tests to extract complex infos from the catalog objects, e.g. partition ids of a hdfs table. Currently, not all the thrift fields are exposed since we are writing the converting codes by ourselves. If there is a tool/library for converting thrift objects to json objects, we can switch to that solution in the future. Tests: - Add tests to verified json fields existence. Change-Id: I15f256b4e3f5206c7140746694106e03b0a4ad92 --- M be/src/catalog/CMakeLists.txt A be/src/catalog/catalog-json.cc M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M tests/webserver/test_web_pages.py 5 files changed, 411 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16449/3 -- To view, visit http://gerrit.cloudera.org:8080/16449 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I15f256b4e3f5206c7140746694106e03b0a4ad92 Gerrit-Change-Number: 16449 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-10076: Reduce partition level update logs
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16375 ) Change subject: IMPALA-10076: Reduce partition level update logs .. Patch Set 4: > Patch Set 4: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6431/ Hit IMPALA-9923 -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 17 Sep 2020 01:18:00 + Gerrit-HasComments: No
[Impala-ASF-CR] Generate test-classpath.txt
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/16450 ) Change subject: Generate test-classpath.txt .. Patch Set 1: Code-Review+2 This makes sense to me. Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/16450 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47e4340fde9a5678d94459f676d8894dc5da14dc Gerrit-Change-Number: 16450 Gerrit-PatchSet: 1 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Comment-Date: Thu, 17 Sep 2020 00:56:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16406 ) Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. Patch Set 8: (1 comment) http://gerrit.cloudera.org:8080/#/c/16406/6/tests/custom_cluster/test_shell_interactive.py File tests/custom_cluster/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/16406/6/tests/custom_cluster/test_shell_interactive.py@87 PS6, Line 87: proc.sendline(profile_cmd) > Done. pexpect.expect takes a regex, so I just used regexes instead. Cool! -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Thu, 17 Sep 2020 00:36:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8291: Show constraints in DESCRIBE FORMATTED
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16428 ) Change subject: IMPALA-8291: Show constraints in DESCRIBE FORMATTED .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b69c465c46491f870d7fdc894e7474c030356 Gerrit-Change-Number: 16428 Gerrit-PatchSet: 3 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 17 Sep 2020 00:14:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8291: Show constraints in DESCRIBE FORMATTED
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16428 ) Change subject: IMPALA-8291: Show constraints in DESCRIBE FORMATTED .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6435/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b69c465c46491f870d7fdc894e7474c030356 Gerrit-Change-Number: 16428 Gerrit-PatchSet: 3 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 17 Sep 2020 00:14:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8291: Show constraints in DESCRIBE FORMATTED
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16428 ) Change subject: IMPALA-8291: Show constraints in DESCRIBE FORMATTED .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b69c465c46491f870d7fdc894e7474c030356 Gerrit-Change-Number: 16428 Gerrit-PatchSet: 2 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Thu, 17 Sep 2020 00:10:38 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16406 ) Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6434/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 16 Sep 2020 23:21:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16406 ) Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. Patch Set 8: Hit IMPALA-9923 -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 16 Sep 2020 23:21:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9382: part 1: transposed profile prototype
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/15798 ) Change subject: IMPALA-9382: part 1: transposed profile prototype .. Patch Set 20: Code-Review+2 This is looking good to me. -- To view, visit http://gerrit.cloudera.org:8080/15798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I0838c6a0872f57c696267ff4e92d29c08748eb7a Gerrit-Change-Number: 15798 Gerrit-PatchSet: 20 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Kurt Deschler Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Wed, 16 Sep 2020 23:13:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8291: Show constraints in DESCRIBE FORMATTED
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16428 ) Change subject: IMPALA-8291: Show constraints in DESCRIBE FORMATTED .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/16428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b69c465c46491f870d7fdc894e7474c030356 Gerrit-Change-Number: 16428 Gerrit-PatchSet: 2 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 16 Sep 2020 21:34:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16406 ) Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6432/ -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 16 Sep 2020 18:24:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10170: Data race on Webserver::UrlHandler::is on nav bar
Wenzhe Zhou has posted comments on this change. ( http://gerrit.cloudera.org:8080/16459 ) Change subject: IMPALA-10170: Data race on Webserver::UrlHandler::is_on_nav_bar_ .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/16459/1/be/src/util/webserver.h File be/src/util/webserver.h: http://gerrit.cloudera.org:8080/#/c/16459/1/be/src/util/webserver.h@160 PS1, Line 160: const bool is_on_nav_bar_; It seems that we need to acquire url_handlers_lock_ in Webserver::GetCommonJson() when accessing url_handlers_. -- To view, visit http://gerrit.cloudera.org:8080/16459 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6923af4754e3fe72b8b04c5303a1e7a79da7613a Gerrit-Change-Number: 16459 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Wenzhe Zhou Gerrit-Comment-Date: Wed, 16 Sep 2020 16:59:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8291: Show constraints in DESCRIBE FORMATTED
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16428 ) Change subject: IMPALA-8291: Show constraints in DESCRIBE FORMATTED .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6433/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b69c465c46491f870d7fdc894e7474c030356 Gerrit-Change-Number: 16428 Gerrit-PatchSet: 2 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 16 Sep 2020 16:23:26 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8291: Show constraints in DESCRIBE FORMATTED
David Rorke has posted comments on this change. ( http://gerrit.cloudera.org:8080/16428 ) Change subject: IMPALA-8291: Show constraints in DESCRIBE FORMATTED .. Patch Set 2: Code-Review+1 Checked the constraint definitions in the test workload. They LGTM. -- To view, visit http://gerrit.cloudera.org:8080/16428 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I676b69c465c46491f870d7fdc894e7474c030356 Gerrit-Change-Number: 16428 Gerrit-PatchSet: 2 Gerrit-Owner: Shant Hovsepian Gerrit-Reviewer: David Rorke Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Shant Hovsepian Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 16 Sep 2020 16:07:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10170: Data race on Webserver::UrlHandler::is on nav bar
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16459 ) Change subject: IMPALA-10170: Data race on Webserver::UrlHandler::is_on_nav_bar_ .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7188/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16459 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6923af4754e3fe72b8b04c5303a1e7a79da7613a Gerrit-Change-Number: 16459 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 16 Sep 2020 15:24:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10170: Data race on Webserver::UrlHandler::is on nav bar
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16459 Change subject: IMPALA-10170: Data race on Webserver::UrlHandler::is_on_nav_bar_ .. IMPALA-10170: Data race on Webserver::UrlHandler::is_on_nav_bar_ This data race can be reproduced by TestCompactCatalogUpdates.test_restart_catalogd, although it does not seem to always occur. The data race was originally reported in a Jenkins job, but I could not reproduce it locally. The fix is to mark a few variables in Webserver::UrlHandler as const. These variables are set during construction time, and never modified afterwards. Testing: * Ran be and custom cluster TSAN tests Change-Id: I6923af4754e3fe72b8b04c5303a1e7a79da7613a --- M be/src/util/webserver.h 1 file changed, 2 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/16459/1 -- To view, visit http://gerrit.cloudera.org:8080/16459 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6923af4754e3fe72b8b04c5303a1e7a79da7613a Gerrit-Change-Number: 16459 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16406 ) Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7187/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 16 Sep 2020 14:45:26 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1
Yida Wu has posted comments on this change. ( http://gerrit.cloudera.org:8080/16318 ) Change subject: WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1 .. Patch Set 10: (10 comments) http://gerrit.cloudera.org:8080/#/c/16318/10//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/16318/10//COMMIT_MSG@43 PS10, Line 43:A local buffer file can be evicted if it is in status REMOTE or it > typo REMOTE -> UPLOADED Done http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/hdfs-fs-cache.h File be/src/runtime/hdfs-fs-cache.h: http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/hdfs-fs-cache.h@57 PS10, Line 57: > Maybe use a typedef for the vector> to improve th Done http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/io/disk-io-mgr.cc File be/src/runtime/io/disk-io-mgr.cc: http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/io/disk-io-mgr.cc@110 PS10, Line 110: DEFINE_int32(num_remote_hdfs_file_oper_io_threads, 2, > Add a comment for the new startup flag. Done. Modified the default value for operation io threads either. http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/io/disk-io-mgr.cc@240 PS10, Line 240: ScopedHistogramTimer > Should the write timer also involve the lock acquisition delays? Done. Yes, it is better to include the lock delays. http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/io/disk-io-mgr.cc@285 PS10, Line 285: offset_ = file_offset; : disk_id_ = disk_id; : tmp_file_ = tmp_file; : io_mgr_ = io_mgr; > We could move all these to the initializer's list in the constructor. Done http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/io/disk-io-mgr.cc@294 PS10, Line 294: disk_id_ = disk_id; : file_path_ = file_path; : io_mgr_ = io_mgr; > Same here, better to move to initializer's list. Done http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/io/request-context.h File be/src/runtime/io/request-context.h: http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/io/request-context.h@296 PS10, Line 296: void RemoteOperDone(RemoteOperRange* oper_range, const Status& write_status); > Would be good to add comments here. Done http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/io/request-ranges.h File be/src/runtime/io/request-ranges.h: http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/io/request-ranges.h@372 PS10, Line 372: bool > Adding comments here for the new functions would be good. Done http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/tmp-file-mgr.cc File be/src/runtime/tmp-file-mgr.cc: http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/tmp-file-mgr.cc@152 PS10, Line 152: > Don't think we're using this variable? Done http://gerrit.cloudera.org:8080/#/c/16318/10/be/src/runtime/tmp-file-mgr.cc@391 PS10, Line 391: s3a_options_ > Should we set s3a_options_ only for S3 and not for hdfs? Done. Move to the S3 path logic. -- To view, visit http://gerrit.cloudera.org:8080/16318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I419b1d5dbbfe35334d9f964c4b65e553579fdc89 Gerrit-Change-Number: 16318 Gerrit-PatchSet: 10 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Yida Wu Gerrit-Comment-Date: Wed, 16 Sep 2020 14:33:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16318 ) Change subject: WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1 .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7186/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I419b1d5dbbfe35334d9f964c4b65e553579fdc89 Gerrit-Change-Number: 16318 Gerrit-PatchSet: 11 Gerrit-Owner: Yida Wu Gerrit-Reviewer: Abhishek Rawat Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 16 Sep 2020 14:31:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16406 ) Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 16 Sep 2020 14:26:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16406 ) Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6432/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 8 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 16 Sep 2020 14:26:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16406 ) Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. Patch Set 7: Code-Review+2 Carrying +2. -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Wed, 16 Sep 2020 14:26:32 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/16406 )
Change subject: IMPALA-9229: impala-shell 'profile' to show original and
retried queries
..
Patch Set 6:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/16406/6/tests/custom_cluster/test_shell_interactive.py
File tests/custom_cluster/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/16406/6/tests/custom_cluster/test_shell_interactive.py@87
PS6, Line 87: proc.expect("Query State: FINISHED")
> Can we verify the absence of the original profile? I.e. something like this
Done. pexpect.expect takes a regex, so I just used regexes instead.
--
To view, visit http://gerrit.cloudera.org:8080/16406
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65
Gerrit-Change-Number: 16406
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Quanlong Huang
Gerrit-Reviewer: Sahil Takiar
Gerrit-Comment-Date: Wed, 16 Sep 2020 14:24:53 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-9229: impala-shell 'profile' to show original and retried queries
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16406 to look at the new patch set (#7). Change subject: IMPALA-9229: impala-shell 'profile' to show original and retried queries .. IMPALA-9229: impala-shell 'profile' to show original and retried queries Currently, the impala-shell 'profile' command only returns the profile for the most recent profile attempt. There is no way to get the original query profile (the profile of the first query attempt that failed) from the impala-shell. This patch modifies TGetRuntimeProfileReq and TGetRuntimeProfileResp to add support for returning both the original and retried profiles for a retried query. When a query is retried, TGetRuntimeProfileResp currently contains the profile for the most recent query attempt. TGetRuntimeProfileReq has a new field called 'include_query_attempts' and when it is set to true, the TGetRuntimeProfileResp will include all failed profiles in a new field called failed_profiles / failed_thrift_profiles. impala-shell has been modified so the 'profile' command has a new set of options. The syntax is now: PROFILE [ALL | LATEST | ORIGINAL] If 'ALL' is specified, both the latest and original profiles are printed. If 'LATEST' is specified, only the latest profile is printed. If 'ORIGINAL' is printed, only the original profile is printed. The default behavior is equivalent to specifying 'LATEST' (which is the current behavior before this patch as well). Support for this has only been added to HS2 given that Beeswax is being deprecated soon. The new 'profile' options have no affect when the Beeswax protocol is used. Most of the code change is in impala-hs2-server and impala-server; a lot of the GetRuntimeProfile code has been re-factored. Testing: * Added new impala-shell tests * Ran core tests Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 --- M be/src/service/client-request-state.cc M be/src/service/client-request-state.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-hs2-server.cc M be/src/service/impala-http-handler.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M common/thrift/ImpalaService.thrift M shell/impala_client.py M shell/impala_shell.py M tests/custom_cluster/test_shell_interactive.py M tests/shell/test_shell_commandline.py M tests/shell/util.py 13 files changed, 505 insertions(+), 125 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/16406/7 -- To view, visit http://gerrit.cloudera.org:8080/16406 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I89cee02947b311e7bf9c7274f47dfc7214c1bb65 Gerrit-Change-Number: 16406 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1
Yida Wu has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/16318 ) Change subject: WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1 .. WIP: IMPALA-9867: Add Support for Spilling to S3: Milestone 1 Major Features 1) Local files as buffers for spilling to S3. 2) Async Upload and Sync Fetching of remote files. 3) Sync remote files deletion after query ends. 4) Local buffer files management. 5) Compatibility of spilling to local and remote. 6) All the errors from hdfs/s3 should terminate the query. Implementation Details: 1) An new enum type is added to specify the disk type of local files, including TmpFileDiskType::LOCAL/DFS/S3. Also, startup option remote_tmp_file_read_by_file is added to specify the implementation of reading pages from the remote. If set to true, the entire file would be fetched to the local buffer during reading(pinning) if it was evicted. If set to false, only a page is read for each reading. 2) Two disk queues have been added to do the file operation jobs. Queue name: RemoteS3DiskFileOper/RemoteDfsDiskFileOper File operations on the remote disk like upload and fetch should be done in these queues. The purpose of the queues is to seperate long run operations with short ones, and also to have a more accurate control on the thread number working on these file operation jobs, sometimes we might not want too many upload and fetch jobs working in the same time. RemoteOperRange is the new type to carry the file operation jobs. Previously,we have request types of READ and WRITE. Now FETCH/UPLOAD have been added. 3) The tmp files are deleted when the tmp file group is deconstructing. For remote files, the entire directory would be deleted. 4) The local buffer files management is to control the total size of local buffer files and evict files if needed. There are basically six status of a remote tmp file, IN_WRITING/DUMPED/IN_FETCHING/UPLOADED/DUMPED_UPLOADED/DELETED. A local buffer file can be evicted if it is in status UPLOADED or it has been all pinned. There are two modes to decide the sequence of choosing files to be evicted. Default is LIFO, the other is FIFO. It can be controlled by startup option remote_tmp_files_avail_pool_lifo. 5) Spilling to local has higher priority than spilling to remote. If no local scratch space is available, temporary data will be spilled to remote. Remote scratch space uses the highest priority local scratch dir as its buffer. If no local scratch space or only one has been configured, a default local buffer should be used. The purpose of the design is to simplify the implementation in milestone 1 with less changes on the configuration. Limitations: * Only one remote scratch dir is supported. * The highest priority local scratch dir is used for the buffer of remote scratch space if remote scratch dir exists. Testcases: * Ran Unit Tests: $IMPALA_HOME/be/build/debug/runtime/buffered-tuple-stream-test $IMPALA_HOME/be/build/debug/runtime/tmp-file-mgr-test $IMPALA_HOME/be/build/debug/runtime/bufferpool/buffer-pool-test $IMPALA_HOME/be/build/debug/runtime/io/disk-io-mgr-test * Some new testcases has been added to tmp-file-mgr-test. TODO: - New Testcases for Spilling to S3. - Upper and lower bounds of new options related to size. - Preserve memory buffer for block buffers on file upload and fetch. - Add some new metrics, like the rate of accessing local buffer. Change-Id: I419b1d5dbbfe35334d9f964c4b65e553579fdc89 --- M be/src/runtime/hdfs-fs-cache.cc M be/src/runtime/hdfs-fs-cache.h M be/src/runtime/io/CMakeLists.txt M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h A be/src/runtime/io/file-writer.h M be/src/runtime/io/hdfs-file-reader.cc A be/src/runtime/io/hdfs-file-writer.cc A be/src/runtime/io/hdfs-file-writer.h M be/src/runtime/io/local-file-system.cc M be/src/runtime/io/local-file-system.h A be/src/runtime/io/local-file-writer.cc A be/src/runtime/io/local-file-writer.h M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/query-state.cc M be/src/runtime/tmp-file-mgr-internal.h M be/src/runtime/tmp-file-mgr-test.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M common/thrift/metrics.json 26 files changed, 2,929 insertions(+), 237 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/16318/11 -- To view, visit http://gerrit.cloudera.org:8080/16318 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I419b1d5dbbfe35334d9f964c4b65e553579fdc89 Gerrit-Change-Number: 16318
[Impala-ASF-CR] IMPALA-10076: Reduce partition level update logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16375 ) Change subject: IMPALA-10076: Reduce partition level update logs .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/6431/ -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 16 Sep 2020 10:18:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16449 ) Change subject: IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7184/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16449 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15f256b4e3f5206c7140746694106e03b0a4ad92 Gerrit-Change-Number: 16449 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 16 Sep 2020 09:45:36 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] MPALA-10075: Reuse unchanged partition instances
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16392 ) Change subject: [WIP] MPALA-10075: Reuse unchanged partition instances .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/7185/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/16392 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dd645c260d271291021e52fdac4b74924df1170 Gerrit-Change-Number: 16392 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 16 Sep 2020 09:44:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] MPALA-10075: Reuse unchanged partition instances
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16392 ) Change subject: [WIP] MPALA-10075: Reuse unchanged partition instances .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/16392/3/tests/metadata/test_reuse_partitions.py File tests/metadata/test_reuse_partitions.py: http://gerrit.cloudera.org:8080/#/c/16392/3/tests/metadata/test_reuse_partitions.py@25 PS3, Line 25: class TestReusePartitions(ImpalaTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/16392 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dd645c260d271291021e52fdac4b74924df1170 Gerrit-Change-Number: 16392 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 16 Sep 2020 09:25:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] MPALA-10075: Reuse unchanged partition instances
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/16392 ) Change subject: [WIP] MPALA-10075: Reuse unchanged partition instances .. Patch Set 3: Thank Qifan's review. Still adding more tests. -- To view, visit http://gerrit.cloudera.org:8080/16392 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dd645c260d271291021e52fdac4b74924df1170 Gerrit-Change-Number: 16392 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Wed, 16 Sep 2020 09:24:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16449 ) Change subject: IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page .. Patch Set 2: (14 comments) http://gerrit.cloudera.org:8080/#/c/16449/2/be/src/catalog/catalog-json.cc File be/src/catalog/catalog-json.cc: http://gerrit.cloudera.org:8080/#/c/16449/2/be/src/catalog/catalog-json.cc@134 PS2, Line 134: Value type_name(_TTableType_VALUES_TO_NAMES.find(table.table_type)->second, allocator); line too long (91 > 90) http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py File tests/webserver/test_web_pages.py: http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@349 PS2, Line 349: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@352 PS2, Line 352: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@363 PS2, Line 363: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@367 PS2, Line 367: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@373 PS2, Line 373: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@377 PS2, Line 377: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@379 PS2, Line 379: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@381 PS2, Line 381: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@382 PS2, Line 382: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@383 PS2, Line 383: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@384 PS2, Line 384: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@385 PS2, Line 385: . flake8: W601 .has_key() is deprecated, use 'in' http://gerrit.cloudera.org:8080/#/c/16449/2/tests/webserver/test_web_pages.py@386 PS2, Line 386: . flake8: W601 .has_key() is deprecated, use 'in' -- To view, visit http://gerrit.cloudera.org:8080/16449 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I15f256b4e3f5206c7140746694106e03b0a4ad92 Gerrit-Change-Number: 16449 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Wed, 16 Sep 2020 09:24:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page
Quanlong Huang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/16449 ) Change subject: IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page .. IMPALA-10168(part-1): Expose JSON catalog objects in catalogd's debug page Catalogd has a debug page at '/catalog_object' showing catalog objects in thrift debug strings. It's inconvenient for tests to parse the thrift string and get interested infos. This patch extend this page to support returning JSON results, which eases tests to extract complex infos from the catalog objects, e.g. partition ids of a hdfs table. Currently, not all the thrift fields are exposed since we are writing the converting codes by ourselves. If there is a tool/library for converting thrift objects to json objects, we can switch to that solution in the future. Tests: - Add tests to verified json fields existence. Change-Id: I15f256b4e3f5206c7140746694106e03b0a4ad92 --- M be/src/catalog/CMakeLists.txt A be/src/catalog/catalog-json.cc M be/src/catalog/catalog-server.cc M be/src/catalog/catalog-server.h M tests/webserver/test_web_pages.py 5 files changed, 410 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/49/16449/2 -- To view, visit http://gerrit.cloudera.org:8080/16449 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I15f256b4e3f5206c7140746694106e03b0a4ad92 Gerrit-Change-Number: 16449 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] [WIP] MPALA-10075: Reuse unchanged partition instances
Hello Qifan Chen, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/16392 to look at the new patch set (#3). Change subject: [WIP] MPALA-10075: Reuse unchanged partition instances .. [WIP] MPALA-10075: Reuse unchanged partition instances Currently, we always update the partition instance when we reload a partition. If a partition remains the same after reloading, we should reuse the old partition instance. So we won't send redudant updates on these partitions. This reduces the size of the catalog topic update. When a huge table is REFRESHed, catalogd only propagates the changes partitions. Tests: - Manually verified that repeate REFERSHs won't trigger catalog updates on partitions. - Still adding more tests Change-Id: I2dd645c260d271291021e52fdac4b74924df1170 --- M fe/src/main/java/org/apache/impala/catalog/FileMetadataLoader.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/ParallelFileMetadataLoader.java A tests/metadata/test_reuse_partitions.py 5 files changed, 158 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/16392/3 -- To view, visit http://gerrit.cloudera.org:8080/16392 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2dd645c260d271291021e52fdac4b74924df1170 Gerrit-Change-Number: 16392 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Qifan Chen Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-10076: Reduce partition level update logs
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16375 ) Change subject: IMPALA-10076: Reduce partition level update logs .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6431/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/16375 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic48946b2f8b0be1e73988092d03a004836f1b368 Gerrit-Change-Number: 16375 Gerrit-PatchSet: 4 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Wed, 16 Sep 2020 06:22:54 + Gerrit-HasComments: No
