[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Fredy Wijaya has uploaded a new patch set (#4). ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. IMPALA-7144: Re-enable TestDescribeTableResults This patch makes the TestDescribeTableResults more robust by only comparing the information that the authorization cares about instead of comparing all output in DESCRIBE. This change will avoid any unnecessary changes to AuthorizationTest if HMS updates the DESCRIBE output. The test is also updated to support standalone execution without relying on other tests be executed first since it can cause the test to be flaky especially if the tests in AuthorizationTest are executed in parallel. Testing: - Ran all FE tests Cherry-picks: not for 2.x Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java 1 file changed, 172 insertions(+), 186 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/10643/4 -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. Patch Set 4: (5 comments) http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java: http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1811 PS3, Line 1811: > add a comment to clarify if these are column names or values. Done http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1812 PS3, Line 1812: ring> colu > add a comment, e.g., "the value of the describe result's location field" Done http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1950 PS3, Line 1950: > cleanup ws Done http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1961 PS3, Line 1961: [month,int,], > example makes it clearer. too not distract this method, pls minimize this a Done http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@2024 PS3, Line 2024: privi > moving this offset in a loop without a check is error prone. pls add a prec Done -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 05:39:33 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. Patch Set 3: (5 comments) several small suggestions to make things clearer, then lgtm. http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java: http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1811 PS3, Line 1811: columns_ add a comment to clarify if these are column names or values. http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1812 PS3, Line 1812: location_; add a comment, e.g., "the value of the describe result's location field" http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1950 PS3, Line 1950: cleanup ws http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1961 PS3, Line 1961: [#col_name,data_type example makes it clearer. too not distract this method, pls minimize this a bit.. for example, its not needed to have so many column example (two #col_name makes sense to illustrate the repetition). the location field should stay. a few fields that are not needed is also fine, but would be good to remove the majority of these. http://gerrit.cloudera.org:8080/#/c/10643/3/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@2024 PS3, Line 2024: rowIdx moving this offset in a loop without a check is error prone. pls add a precondition before each rows access that rowIdx < rows.size() -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 05:10:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10415 ) Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2626/ -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 4 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 04:54:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7044: Prevent overflow when computing Parquet block size
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10483 ) Change subject: IMPALA-7044: Prevent overflow when computing Parquet block size .. IMPALA-7044: Prevent overflow when computing Parquet block size When writing Parquet files we compute a minimum block size based on the number of columns in the target table: 3 * page_size * num_cols For tables with a large number of columns (> ~10k), this value will get larger than 2GB. When we pass it to hdfsOpenFile() in HdfsTableSink::CreateNewTmpFile() it gets cast to a signed int32 and can overflow. To fix this we return an error if we detect that the minimum block size exceed 2GB. This change adds a test using CTAS into a table with 12k columns, making sure that Impala returns the correct error. Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Reviewed-on: http://gerrit.cloudera.org:8080/10483 Reviewed-by: Lars Volker Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-parquet-table-writer.h M be/src/exec/hdfs-table-sink.cc M tests/query_test/test_insert_parquet.py 4 files changed, 52 insertions(+), 14 deletions(-) Approvals: Lars Volker: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Gerrit-Change-Number: 10483 Gerrit-PatchSet: 8 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7044: Prevent overflow when computing Parquet block size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10483 ) Change subject: IMPALA-7044: Prevent overflow when computing Parquet block size .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Gerrit-Change-Number: 10483 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 04:40:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10647 ) Change subject: IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build .. IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build The test is flaky in the erasure coding build. Let's disable it for now. Change-Id: Ic9a34a91eef40e1da9c7134cfb7054006d9115de Reviewed-on: http://gerrit.cloudera.org:8080/10647 Reviewed-by: Tianyi Wang Tested-by: Impala Public Jenkins --- M tests/query_test/test_mem_usage_scaling.py 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Tianyi Wang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic9a34a91eef40e1da9c7134cfb7054006d9115de Gerrit-Change-Number: 10647 Gerrit-PatchSet: 2 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang
[Impala-ASF-CR] IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10647 ) Change subject: IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9a34a91eef40e1da9c7134cfb7054006d9115de Gerrit-Change-Number: 10647 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 08 Jun 2018 04:15:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 2: clean up DiskIoMgr
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10479 ) Change subject: IMPALA-6953: part 2: clean up DiskIoMgr .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie82849652266b6660dd479689e5134273b172eb5 Gerrit-Change-Number: 10479 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 02:56:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 2: clean up DiskIoMgr
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10479 ) Change subject: IMPALA-6953: part 2: clean up DiskIoMgr .. IMPALA-6953: part 2: clean up DiskIoMgr Follow-on changes. Made further mechanical changes that involved a lot of code motion: * Move code that manipulates RequestContext internal state to RequestContext * Don't use protected friend pattern that doesn't work as I thought * Remove some unused member variables and arguments * Move PerDiskState definition into .cc Change-Id: Ie82849652266b6660dd479689e5134273b172eb5 Reviewed-on: http://gerrit.cloudera.org:8080/10479 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/hdfs-parquet-scanner.cc M be/src/exec/hdfs-parquet-scanner.h M be/src/exec/hdfs-scan-node-base.cc M be/src/exec/hdfs-scan-node.cc M be/src/exec/parquet-column-readers.cc M be/src/exec/scanner-context.cc M be/src/runtime/io/disk-io-mgr-stress.cc M be/src/runtime/io/disk-io-mgr-test.cc M be/src/runtime/io/disk-io-mgr.cc M be/src/runtime/io/disk-io-mgr.h M be/src/runtime/io/request-context.cc M be/src/runtime/io/request-context.h M be/src/runtime/io/request-ranges.h M be/src/runtime/io/scan-range.cc M be/src/runtime/tmp-file-mgr.cc M be/src/runtime/tmp-file-mgr.h 17 files changed, 527 insertions(+), 565 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ie82849652266b6660dd479689e5134273b172eb5 Gerrit-Change-Number: 10479 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. Patch Set 3: (11 comments) http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java: http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1810 PS2, Line 1810: > nit: remove (same for below) Done http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1810 PS2, Line 1810: > nit: remove (same for below) Done http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1879 PS2, Line 1879: > what data? It should be metadata. http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1890 PS2, Line 1890: "hdfs://localhost:20500/test-warehouse/alltypesagg" > is there a reason for skipping EXTENDED for alltypessmall? Nope. I'll add. http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1903 PS2, Line 1903: atements force > move to before use (L1809) Done http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1904 PS2, Line 1904: indepe > columns_ Done http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1905 PS2, Line 1905: ctional. > location_ Done http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1944 PS2, Line 1944: // > use a switch instead. Done http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1946 PS2, Line 1946: > should these be case insensitive? The column names in DESCRIBE output are always in lower case. I'll trim the output. http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1953 PS2, Line 1953: > does this case matter? We want to match the same exact thing. So it's better to match the output in DESCRIBE. http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1956 PS2, Line 1956: eak; > not obvious what is r doing... it looks like a row idx but gets bumped per I'll rename r to rowIdx and add more comments. -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 01:57:22 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. IMPALA-7144: Re-enable TestDescribeTableResults This patch makes the TestDescribeTableResults more robust by only comparing the information that the authorization cares about instead of comparing all output in DESCRIBE. This change will avoid any unnecessary changes to AuthorizationTest if HMS updates the DESCRIBE output. The test is also updated to support standalone execution without relying on other tests be executed first since it can cause the test to be flaky especially if the tests in AuthorizationTest are executed in parallel. Testing: - Ran all FE tests Cherry-picks: not for 2.x Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java 1 file changed, 211 insertions(+), 186 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/10643/3 -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10415 ) Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2626/ -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 4 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 01:38:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7044: Prevent overflow when computing Parquet block size
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10483 ) Change subject: IMPALA-7044: Prevent overflow when computing Parquet block size .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2625/ -- To view, visit http://gerrit.cloudera.org:8080/10483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Gerrit-Change-Number: 10483 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 01:20:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7044: Prevent overflow when computing Parquet block size
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/10483 ) Change subject: IMPALA-7044: Prevent overflow when computing Parquet block size .. Patch Set 7: Code-Review+2 (1 comment) Addressed one comment, rebased in PS7. Carrying Tim's +2. http://gerrit.cloudera.org:8080/#/c/10483/6/be/src/exec/hdfs-parquet-table-writer.cc File be/src/exec/hdfs-parquet-table-writer.cc: http://gerrit.cloudera.org:8080/#/c/10483/6/be/src/exec/hdfs-parquet-table-writer.cc@985 PS6, Line 985: > static_cast Done -- To view, visit http://gerrit.cloudera.org:8080/10483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Gerrit-Change-Number: 10483 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 01:19:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7044: Prevent overflow when computing Parquet block size
Hello Thomas Marshall, Tim Armstrong, Dan Hecht, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10483 to look at the new patch set (#7). Change subject: IMPALA-7044: Prevent overflow when computing Parquet block size .. IMPALA-7044: Prevent overflow when computing Parquet block size When writing Parquet files we compute a minimum block size based on the number of columns in the target table: 3 * page_size * num_cols For tables with a large number of columns (> ~10k), this value will get larger than 2GB. When we pass it to hdfsOpenFile() in HdfsTableSink::CreateNewTmpFile() it gets cast to a signed int32 and can overflow. To fix this we return an error if we detect that the minimum block size exceed 2GB. This change adds a test using CTAS into a table with 12k columns, making sure that Impala returns the correct error. Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 --- M be/src/exec/hdfs-parquet-table-writer.cc M be/src/exec/hdfs-parquet-table-writer.h M be/src/exec/hdfs-table-sink.cc M tests/query_test/test_insert_parquet.py 4 files changed, 52 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/83/10483/7 -- To view, visit http://gerrit.cloudera.org:8080/10483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Gerrit-Change-Number: 10483 Gerrit-PatchSet: 7 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10365 ) Change subject: IMPALA-6035: Add query options to limit thread reservation .. Patch Set 9: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2623/ -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 00:58:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5740: [DOCS] Correct the max length of STRING
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10511 ) Change subject: IMPALA-5740: [DOCS] Correct the max length of STRING .. IMPALA-5740: [DOCS] Correct the max length of STRING Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2 Reviewed-on: http://gerrit.cloudera.org:8080/10511 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M docs/topics/impala_string.xml 1 file changed, 157 insertions(+), 81 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2 Gerrit-Change-Number: 10511 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5740: [DOCS] Correct the max length of STRING
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10511 ) Change subject: IMPALA-5740: [DOCS] Correct the max length of STRING .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2 Gerrit-Change-Number: 10511 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 00:55:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10647 ) Change subject: IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2624/ -- To view, visit http://gerrit.cloudera.org:8080/10647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9a34a91eef40e1da9c7134cfb7054006d9115de Gerrit-Change-Number: 10647 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 08 Jun 2018 00:55:01 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5740: [DOCS] Correct the max length of STRING
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10511 ) Change subject: IMPALA-5740: [DOCS] Correct the max length of STRING .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-docs-submit/312/ -- To view, visit http://gerrit.cloudera.org:8080/10511 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I43c5a2819c8a3db33a8ce3a6bbde6a1d823ec9b2 Gerrit-Change-Number: 10511 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 00:51:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10647 ) Change subject: IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build .. Patch Set 1: Code-Review+2 It seems only the query memory limit is exceeded. In that case disabling this query is fine. -- To view, visit http://gerrit.cloudera.org:8080/10647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic9a34a91eef40e1da9c7134cfb7054006d9115de Gerrit-Change-Number: 10647 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 08 Jun 2018 00:52:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7149: Skip q7 in test mem usage scaling in erasure coding build
Taras Bobrovytsky has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10647 Change subject: IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build .. IMPALA-7149: Skip q7 in test_mem_usage_scaling in erasure coding build The test is flaky in the erasure coding build. Let's disable it for now. Change-Id: Ic9a34a91eef40e1da9c7134cfb7054006d9115de --- M tests/query_test/test_mem_usage_scaling.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/10647/1 -- To view, visit http://gerrit.cloudera.org:8080/10647 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ic9a34a91eef40e1da9c7134cfb7054006d9115de Gerrit-Change-Number: 10647 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-7099: Don't set FILESYSTEM PREFIX for s3
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10595 ) Change subject: IMPALA-7099: Don't set FILESYSTEM_PREFIX for s3 .. Patch Set 3: Validated using an S3 core build. The standard GVO wouldn't take this path, so it wouldn't help validate this much. -- To view, visit http://gerrit.cloudera.org:8080/10595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace5fc9557ff6e7d184573beafea694191424742 Gerrit-Change-Number: 10595 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 00:42:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7099: Don't set FILESYSTEM PREFIX for s3
Dan Hecht has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10595 ) Change subject: IMPALA-7099: Don't set FILESYSTEM_PREFIX for s3 .. IMPALA-7099: Don't set FILESYSTEM_PREFIX for s3 When S3 support was originally added to Impala, Impala still required that the defaultFs point to HDFS, and so the S3 tests were originally run as a "secondary" filesystem (which meant that all paths had to be fully qualified). Since then, the restriction on defaultFs has been addressed, and additionally, the S3 test configuration had been changed to run with S3 as the defaultFs. So, setting FILESYSTEM_PREFIX for S3 is no longer necessary. Let's remove that so that S3 tests run in a configuration more similar to HDFS, which will help prevent them from breaking as often (since precheckin tests run only in the HDFS configuration). Incidently, this addresses the problem with test_unsupported_text_compression on S3 since the problem is with the handling of fully qualified paths. Testing: - core on S3 Change-Id: Iace5fc9557ff6e7d184573beafea694191424742 Reviewed-on: http://gerrit.cloudera.org:8080/10595 Reviewed-by: Dan Hecht Tested-by: Dan Hecht --- M bin/impala-config.sh 1 file changed, 0 insertions(+), 1 deletion(-) Approvals: Dan Hecht: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iace5fc9557ff6e7d184573beafea694191424742 Gerrit-Change-Number: 10595 Gerrit-PatchSet: 4 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7099: Don't set FILESYSTEM PREFIX for s3
Hello Sailesh Mukil, Tim Armstrong, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10595 to look at the new patch set (#3). Change subject: IMPALA-7099: Don't set FILESYSTEM_PREFIX for s3 .. IMPALA-7099: Don't set FILESYSTEM_PREFIX for s3 When S3 support was originally added to Impala, Impala still required that the defaultFs point to HDFS, and so the S3 tests were originally run as a "secondary" filesystem (which meant that all paths had to be fully qualified). Since then, the restriction on defaultFs has been addressed, and additionally, the S3 test configuration had been changed to run with S3 as the defaultFs. So, setting FILESYSTEM_PREFIX for S3 is no longer necessary. Let's remove that so that S3 tests run in a configuration more similar to HDFS, which will help prevent them from breaking as often (since precheckin tests run only in the HDFS configuration). Incidently, this addresses the problem with test_unsupported_text_compression on S3 since the problem is with the handling of fully qualified paths. Testing: - core on S3 Change-Id: Iace5fc9557ff6e7d184573beafea694191424742 --- M bin/impala-config.sh 1 file changed, 0 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/95/10595/3 -- To view, visit http://gerrit.cloudera.org:8080/10595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iace5fc9557ff6e7d184573beafea694191424742 Gerrit-Change-Number: 10595 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7099: Don't set FILESYSTEM PREFIX for s3
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10595 ) Change subject: IMPALA-7099: Don't set FILESYSTEM_PREFIX for s3 .. Patch Set 2: Do you want to go ahead and start this merge? It would be good to get this in to reduce the probability of future issues. -- To view, visit http://gerrit.cloudera.org:8080/10595 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iace5fc9557ff6e7d184573beafea694191424742 Gerrit-Change-Number: 10595 Gerrit-PatchSet: 2 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 00:25:50 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7102: Disable support of erasure coding by default
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10646 ) Change subject: IMPALA-7102: Disable support of erasure coding by default .. Patch Set 1: Code-Review+1 -- To view, visit http://gerrit.cloudera.org:8080/10646 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icd3b1754541262467a6e67068b0b447882a40fb3 Gerrit-Change-Number: 10646 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 00:24:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7102: Disable support of erasure coding by default
Taras Bobrovytsky has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10646 Change subject: IMPALA-7102: Disable support of erasure coding by default .. IMPALA-7102: Disable support of erasure coding by default In this patch we add a query option ALLOW_ERASURE_CODED_FILES, that allows us to enable or disable the support of erasure coded files. At this time erasure coding is an experimental feature and is not supported by Impala, so we want to prevent users from using it without explictly enabling it. Cherry-picks: not for 2.x Change-Id: Icd3b1754541262467a6e67068b0b447882a40fb3 --- M be/src/service/query-options.cc M be/src/service/query-options.h M bin/run-all-tests.sh M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M testdata/bin/create-load-data.sh A testdata/workloads/functional-query/queries/QueryTest/hdfs-erasure-coding.test M tests/common/custom_cluster_test_suite.py M tests/common/skip.py M tests/query_test/test_observability.py M tests/query_test/test_scanners.py 12 files changed, 64 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/46/10646/1 -- To view, visit http://gerrit.cloudera.org:8080/10646 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Icd3b1754541262467a6e67068b0b447882a40fb3 Gerrit-Change-Number: 10646 Gerrit-PatchSet: 1 Gerrit-Owner: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-7071: make get fs path() idempotent
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10517 ) Change subject: IMPALA-7071: make get_fs_path() idempotent .. IMPALA-7071: make get_fs_path() idempotent This avoids future errors like IMPALA-7068. Change-Id: I10c7cbb36119883718e60660db2ac7313f14d388 Reviewed-on: http://gerrit.cloudera.org:8080/10517 Reviewed-by: Dan Hecht Tested-by: Impala Public Jenkins --- A tests/infra/test_utils.py M tests/util/filesystem_utils.py 2 files changed, 37 insertions(+), 2 deletions(-) Approvals: Dan Hecht: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I10c7cbb36119883718e60660db2ac7313f14d388 Gerrit-Change-Number: 10517 Gerrit-PatchSet: 5 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7071: make get fs path() idempotent
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10517 ) Change subject: IMPALA-7071: make get_fs_path() idempotent .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c7cbb36119883718e60660db2ac7313f14d388 Gerrit-Change-Number: 10517 Gerrit-PatchSet: 4 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 08 Jun 2018 00:14:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 3: Code-Review+2 thanks for the changes! -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 3 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 08 Jun 2018 00:10:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7137. Support configuring Frontend to use LocalCatalog
Hello Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10629 to look at the new patch set (#2). Change subject: IMPALA-7137. Support configuring Frontend to use LocalCatalog .. IMPALA-7137. Support configuring Frontend to use LocalCatalog This adds a new flag -use_local_catalog which is passed through to the frontend and causes it to use LocalCatalog instead of ImpaladCatalog. Additionally, when this flag is configured, the impalad does not subscribe to catalog topic updates from the statestore. The patch is slightly more complex than simply picking which class to instantiate, because the lifecycle is designed a bit differently between the two implementations: - LocalCatalog is instantiated once per query/request. - ImpaladCatalog is instantiated once and stateful across queries, except when a full catalog update is received. In order to abstract this difference in lifecycle from the frontend, I introduced a new FeCatalogManager class with different implementations for the two lifecycles. I also had to add a simple test implementation since some tests rely on directly injecting a Catalog implementation into the Frontend. This patch also includes a few small changes to the local catalog implementation objects to enable the impalad to start and accept connections. With this patch, I was able to manually test as follows: I started just the statestore and the impalad in the new mode: - ./bin/start-statestored.sh - ./bin/start-impalad.sh --use_local_catalog I connected with impala-shell as usual and was able to run the most simple queries: - SHOW DATABASES; - USE functional; - SHOW TABLES; All other functionality results in error messages due to the various TODOs in the current skeleton implementation. Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 --- M be/src/runtime/exec-env.cc M be/src/service/impala-server.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/service/DescribeResultFactory.java A fe/src/main/java/org/apache/impala/service/FeCatalogManager.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/main/java/org/apache/impala/service/MetadataOp.java 11 files changed, 232 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/10629/2 -- To view, visit http://gerrit.cloudera.org:8080/10629 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I8c9665bd031d23608740b23eef301970af9aa764 Gerrit-Change-Number: 10629 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7141. Extract interfaces for partition pruning without loading
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10631 to look at the new patch set (#2). Change subject: IMPALA-7141. Extract interfaces for partition pruning without loading .. IMPALA-7141. Extract interfaces for partition pruning without loading This adds a new PrunablePartition interface which HdfsPartition implements. The interface includes only the partition ID and the partition key/values. For the case of the current catalog implementation, this provides no benefit. However, for LocalCatalog, we want to defer loading partition information until after pruning. With this interface, we can construct PrunablePartition objects using just the partition names, and then load the partitions once pruning is complete. Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa --- M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/FeFsTable.java M fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java A fe/src/main/java/org/apache/impala/catalog/PrunablePartition.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionFilter.java M fe/src/main/java/org/apache/impala/planner/HdfsPartitionPruner.java M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/testutil/BlockIdGenerator.java 18 files changed, 181 insertions(+), 46 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10631/2 -- To view, visit http://gerrit.cloudera.org:8080/10631 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ifdcfd6cffcd298bbf44531e1ec2f47c3a5b7d1fa Gerrit-Change-Number: 10631 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7135. Skeleton implementation of LocalCatalog
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10627 to look at the new patch set (#2). Change subject: IMPALA-7135. Skeleton implementation of LocalCatalog .. IMPALA-7135. Skeleton implementation of LocalCatalog This adds some of the high level classes for implementing the local catalog: - LocalCatalog is the top level implementation. The plan is to instantiate this once per query, so that no thread safety is required. - It loads metadata from a MetaProvider interface. The current implementation fetches directly from HMS and provides no caching. A future subtask will add a CachingMetaProvider implementation. Separating out caching will make it easier to experiment with different policies or storage mechanisms. - It instantiates LocalDb and LocalTable objects to implement FeDb and FeTable. These are mostly stubbed out except for the most basic functionality. Functionality will be filled in incrementally in further patches. Since it's not yet possible to hook this up to most of the existing tests, a very simple new unit test is included to cover the bits of functionality that are not stubbed out. I didn't concentrate on too much test coverage here, since once we've implemented more functionality we can switch over all of the existing tests to get coverage of the new implementation. Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee --- M fe/src/main/java/org/apache/impala/catalog/Catalog.java A fe/src/main/java/org/apache/impala/catalog/local/DirectMetaProvider.java A fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java A fe/src/main/java/org/apache/impala/catalog/local/LocalCatalogException.java A fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java A fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java A fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java A fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 8 files changed, 766 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/27/10627/2 -- To view, visit http://gerrit.cloudera.org:8080/10627 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab653371188b21c72f50ee1ec4e94950aa6fb9ee Gerrit-Change-Number: 10627 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7128 (part 2): add an interface for data sources
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10626 to look at the new patch set (#2). Change subject: IMPALA-7128 (part 2): add an interface for data sources .. IMPALA-7128 (part 2): add an interface for data sources This changes most of the usage of DataSource and DataSourceTable to use interfaces instead of implementation classes. There are still various usages of the implementation for functionality like creating and dropping data sources. We'll address those as part of IMPALA-7131 at a later date. Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e --- M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/CreateTableDataSrcStmt.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/catalog/DataSource.java M fe/src/main/java/org/apache/impala/catalog/DataSourceTable.java M fe/src/main/java/org/apache/impala/catalog/FeCatalog.java A fe/src/main/java/org/apache/impala/catalog/FeDataSource.java A fe/src/main/java/org/apache/impala/catalog/FeDataSourceTable.java M fe/src/main/java/org/apache/impala/planner/DataSourceScanNode.java M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java 13 files changed, 99 insertions(+), 26 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/26/10626/2 -- To view, visit http://gerrit.cloudera.org:8080/10626 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ibe704197dc2ad7c09b8340865f17567096aa630e Gerrit-Change-Number: 10626 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog
Hello Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10630 to look at the new patch set (#2). Change subject: IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog .. IMPALA-7140 (part 1). Support fetching schema info in LocalCatalog This adds support for loading the Table object from HMS and parsing out the Column and partitioning information from it. With this change, I'm able to connect to an impalad running in "local catalog" mode and run DESCRIBE, DESCRIBE EXTENDED, and SHOW CREATE TABLE commands. Other commands like SHOW PARTITIONS don't work properly yet, and type-specific table functionality (eg views, HBase tables, etc) are not yet supported. Again a simple unit test is included to check that column information is loaded. More thorough testing is deferred until we've reached enough coverage that we can start running e2e tests against a cluster running in "local" mode. Change-Id: I640f27e36198955e057da62a3ce25a858406e496 --- A fe/src/main/java/org/apache/impala/catalog/FeCatalogUtils.java M fe/src/main/java/org/apache/impala/catalog/HdfsTable.java M fe/src/main/java/org/apache/impala/catalog/Table.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalogException.java M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/test/java/org/apache/impala/catalog/CatalogTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 8 files changed, 255 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/30/10630/2 -- To view, visit http://gerrit.cloudera.org:8080/10630 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I640f27e36198955e057da62a3ce25a858406e496 Gerrit-Change-Number: 10630 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Hello Impala Public Jenkins, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10611 to look at the new patch set (#3). Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition This refactors out interfaces in the frontend for the interaction between the analysis/planning code and the classes that implement these catalog objects. This takes care of the most commonly used objects but defers some others (e.g. functions, cache pools, data sources, etc) to follow-on patches. There are a few spots remaining in the frontend that still downcast to implementation classes, particularly where the frontend actually makes modifications to catalog objects in-place. I left TODOs in those spots and will come back later as necessary. Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 --- M fe/src/main/java/org/apache/impala/analysis/AlterTableAddDropRangePartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAddReplaceColsStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableAlterColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropColStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableDropPartitionStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableOrViewRenameStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetCachedStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetFileFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetLocationStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetRowFormatStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSetTblProperties.java M fe/src/main/java/org/apache/impala/analysis/AlterTableSortByStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterTableStmt.java M fe/src/main/java/org/apache/impala/analysis/AlterViewStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BaseTableRef.java M fe/src/main/java/org/apache/impala/analysis/ColumnLineageGraph.java M fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateFunctionStmtBase.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/CreateTableLikeStmt.java M fe/src/main/java/org/apache/impala/analysis/DescribeTableStmt.java M fe/src/main/java/org/apache/impala/analysis/DescriptorTable.java M fe/src/main/java/org/apache/impala/analysis/DropDbStmt.java M fe/src/main/java/org/apache/impala/analysis/DropFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/DropTableOrViewStmt.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/IsNullPredicate.java M fe/src/main/java/org/apache/impala/analysis/LoadDataStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/PartitionDef.java M fe/src/main/java/org/apache/impala/analysis/PartitionSet.java M fe/src/main/java/org/apache/impala/analysis/PartitionSpecBase.java M fe/src/main/java/org/apache/impala/analysis/Path.java M fe/src/main/java/org/apache/impala/analysis/PrivilegeSpec.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateFunctionStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowCreateTableStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowFilesStmt.java M fe/src/main/java/org/apache/impala/analysis/ShowStatsStmt.java M fe/src/main/java/org/apache/impala/analysis/SlotRef.java M fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java M fe/src/main/java/org/apache/impala/analysis/TableDef.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/TruncateStmt.java M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/main/java/org/apache/impala/catalog/Catalog.java M fe/src/main/java/org/apache/impala/catalog/Db.java A fe/src/main/java/org/apache/impala/catalog/FeCatalog.java A fe/src/main/java/org/apache/impala/catalog/FeDb.java A
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java File fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java@220 PS2, Line 220: HdfsPartition > I don't think this one merits a TODO because it's in test code that's speci yup.. meant to get rid of that as I was catching my self getting rid of the todo comment on the Db downcast in the other test. -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 23:48:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java File fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java@220 PS2, Line 220: HdfsPartition > nit: add a todo for the downcast. I don't think this one merits a TODO because it's in test code that's specific to the catalogd implementation -- the new implementation won't need TCatalogObject and friends since that's only used for the catalogd->impalad propagation protocol, right? -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 23:47:20 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 2: (3 comments) thanks for the explanations. the only open item is to move away from naming based on "hdfs"... I'm in favor of it but if we want to make a pass at that sort of thing afterwards, I'm ok with that as well. http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java File fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java@36 PS2, Line 36: Note: despite the "HDFS" nomenclature, this table type is also used for : * interacting with S3 or other Hadoop-compatible filesystems. > Done yes, I think its preferable to just do this now as additions are being made. http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@964 PS2, Line 964: KeyValueComparator > Partitions are used in Collections.sort, which requires the class to be com thanks for the explanation, makes sense. http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java File fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/test/java/org/apache/impala/catalog/CatalogObjectToFromThriftTest.java@220 PS2, Line 220: HdfsPartition nit: add a todo for the downcast. -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 23:46:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6953: part 2: clean up DiskIoMgr
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10479 ) Change subject: IMPALA-6953: part 2: clean up DiskIoMgr .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/31/ -- To view, visit http://gerrit.cloudera.org:8080/10479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie82849652266b6660dd479689e5134273b172eb5 Gerrit-Change-Number: 10479 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 23:37:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 2: clean up DiskIoMgr
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10479 ) Change subject: IMPALA-6953: part 2: clean up DiskIoMgr .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/31/ -- To view, visit http://gerrit.cloudera.org:8080/10479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie82849652266b6660dd479689e5134273b172eb5 Gerrit-Change-Number: 10479 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 23:37:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 2: clean up DiskIoMgr
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10479 ) Change subject: IMPALA-6953: part 2: clean up DiskIoMgr .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun-tarmstrong/30/ -- To view, visit http://gerrit.cloudera.org:8080/10479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie82849652266b6660dd479689e5134273b172eb5 Gerrit-Change-Number: 10479 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 23:36:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6953: part 2: clean up DiskIoMgr
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10479 ) Change subject: IMPALA-6953: part 2: clean up DiskIoMgr .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10479 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie82849652266b6660dd479689e5134273b172eb5 Gerrit-Change-Number: 10479 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 23:36:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10643 ) Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. Patch Set 2: (11 comments) http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java: http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1810 PS2, Line 1810: a nit: remove (same for below) http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1810 PS2, Line 1810: table nit: remove (same for below) http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1879 PS2, Line 1879: data what data? http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1890 PS2, Line 1890: style = TDescribeOutputStyle.MINIMAL; is there a reason for skipping EXTENDED for alltypessmall? http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1903 PS2, Line 1903: DescribeResult move to before use (L1809) http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1904 PS2, Line 1904: columns columns_ http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1905 PS2, Line 1905: location location_ also, what is this? an example of how describe output maps here would be useful. http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1944 PS2, Line 1944: if (sty use a switch instead. http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1946 PS2, Line 1946: getString_val should these be case insensitive? is trim no longer needed? http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1953 PS2, Line 1953: Location does this case matter? http://gerrit.cloudera.org:8080/#/c/10643/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@1956 PS2, Line 1956: r++; not obvious what is r doing... it looks like a row idx but gets bumped per column (L1960) -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 23:17:27 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10488 ) Change subject: IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10488 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie456c87c34e7d75eb7482f499e90d50de740df40 Gerrit-Change-Number: 10488 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 23:01:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5737: Tighten minicluster memory limit
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10277 ) Change subject: IMPALA-5737: Tighten minicluster memory limit .. Patch Set 8: > Patch Set 8: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2620/ I just noticed that jenkins.impala.io uses m4.4xlarge, which has much more cores than the machine I tested on. I will increase the memory limit in profile 2. -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 8 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 22:54:15 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5737: Tighten minicluster memory limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10277 ) Change subject: IMPALA-5737: Tighten minicluster memory limit .. Patch Set 8: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2620/ -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 8 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 22:48:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Print logs around removeDirective()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10640 ) Change subject: IMPALA-3040: Print logs around removeDirective() .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814 Gerrit-Change-Number: 10640 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 22:36:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Print logs around removeDirective()
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10640 ) Change subject: IMPALA-3040: Print logs around removeDirective() .. IMPALA-3040: Print logs around removeDirective() I observed that some caching directives on partitions don't get removed. HDFS would throw an exception if removeDirective() operation fails on its side, so I suspect that removeDirective() isn't called by catalogd. This patch prints logs around removeDirective() if it's skipped. Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814 Reviewed-on: http://gerrit.cloudera.org:8080/10640 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java 1 file changed, 22 insertions(+), 4 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814 Gerrit-Change-Number: 10640 Gerrit-PatchSet: 3 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 2: (10 comments) http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java File fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/analysis/StmtMetadataLoader.java@175 PS2, Line 175: SimpleCatalog > nit: stale Done http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeCatalog.java File fe/src/main/java/org/apache/impala/catalog/FeCatalog.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeCatalog.java@86 PS2, Line 86: HDFS > nit: FS ... I realize that some of the existing classes were misnamed HDFS, Done http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeDb.java File fe/src/main/java/org/apache/impala/catalog/FeDb.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeDb.java@27 PS2, Line 27: from the frontend > nit: rephrase as "Frontend interface for interacting with a database." Done http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/FeHdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsPartition.java@31 PS2, Line 31: between > remove Done http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsPartition.java@34 PS2, Line 34: Hdfs > I'd rename this to FS (couple comments about this) Done http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsPartition.java@61 PS2, Line 61: /** > nit: add a newline. several other cases below. Done http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java File fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java@36 PS2, Line 36: Note: despite the "HDFS" nomenclature, this table type is also used for : * interacting with S3 or other Hadoop-compatible filesystems. > I had tried to just be consistent that 'Foo' class turned into 'FeFoo' inte Done http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@961 PS2, Line 961: Comparison > nit: Comparator Done http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@563 PS2, Line 563: @Ov > nit: add a line Done http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@581 PS2, Line 581: @O > nit: add a line Done -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 22:29:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Fredy Wijaya has uploaded a new patch set (#14). ( http://gerrit.cloudera.org:8080/9933 ) Change subject: IMPALA-2195: Improper handling of comments in queries .. IMPALA-2195: Improper handling of comments in queries This patch fixes an issue where parseline is unable to deduce the correct command when a statement has a leading comment. Before: > -- comment > insert into table t values(100); Fetched 1 row(s) in 0.01s After: > -- comment > insert into table t values(100); Modified 1 row(s) in 0.01s Before (FE syntax error): > /*comment*/ help; After (show help correctly): > /*comment*/ help; Testing: - Added shell tests - Ran end-to-end shell tests on Python 2.6 and Python 2.7 Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 173 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/33/9933/14 -- To view, visit http://gerrit.cloudera.org:8080/9933 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7 Gerrit-Change-Number: 9933 Gerrit-PatchSet: 14 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Taras Bobrovytsky
[Impala-ASF-CR] IMPALA-2195: Improper handling of comments in queries
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/9933 )
Change subject: IMPALA-2195: Improper handling of comments in queries
..
Patch Set 14:
(8 comments)
Fixed all the flake8 errors and added some new tests.
http://gerrit.cloudera.org:8080/#/c/9933/13/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/9933/13/shell/impala_shell.py@325
PS13, Line 325: command, args),
> ./shell/impala_shell.py:325:50: E128 continuation line under-indented for v
Done
http://gerrit.cloudera.org:8080/#/c/9933/13/shell/impala_shell.py@1344
PS13, Line 1344: if isinstance(token, sqlparse.sql.Comment):
> ./shell/impala_shell.py:1344:51: E701 multiple statements on one line (colo
Done
http://gerrit.cloudera.org:8080/#/c/9933/13/shell/impala_shell.py@1346
PS13, Line 1346: for comment in sqlparse.tokens.Comment:
> ./shell/impala_shell.py:1346:36: E701 multiple statements on one line (colo
Done
http://gerrit.cloudera.org:8080/#/c/9933/13/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/9933/13/tests/shell/test_shell_interactive.py@481
PS13, Line 481: 'select * from
leading_comment;')
> What about a test case like:
Done
http://gerrit.cloudera.org:8080/#/c/9933/13/tests/shell/test_shell_interactive.py@482
PS13, Line 482: assert 'Fetched 1 row(s)' in result.stderr
> What about a test case like:
Done
http://gerrit.cloudera.org:8080/#/c/9933/13/tests/shell/test_shell_interactive.py@494
PS13, Line 494:
> Same as above except using help, not select.
Done
http://gerrit.cloudera.org:8080/#/c/9933/13/tests/shell/test_shell_interactive.py@610
PS13, Line 610: assert ('/*delete*/ ', 'select /*do not delete*/ 1') == \
> ./tests/shell/test_shell_interactive.py:610:10: E127 continuation line over
Done
http://gerrit.cloudera.org:8080/#/c/9933/13/tests/shell/test_shell_interactive.py@619
PS13, Line 619: assert ('/*delete*/\n', 'select c1 from\n'
> ./tests/shell/test_shell_interactive.py:619:10: E127 continuation line over
Done
--
To view, visit http://gerrit.cloudera.org:8080/9933
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I7ac7cb5a30e6dda73ebe761d9f0eb9ba038e14a7
Gerrit-Change-Number: 9933
Gerrit-PatchSet: 14
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Alex Behm
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: David Knupp
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Michael Brown
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Taras Bobrovytsky
Gerrit-Comment-Date: Thu, 07 Jun 2018 22:22:44 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6917: Implement COMMENT ON TABLE/VIEW
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10478 ) Change subject: IMPALA-6917: Implement COMMENT ON TABLE/VIEW .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I497c17342f79ff7c99931fd8a0ddec0c79303dbf Gerrit-Change-Number: 10478 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 22:11:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6802 (part 4): Clean up authorization tests
Fredy Wijaya has removed a vote on this change. Change subject: IMPALA-6802 (part 4): Clean up authorization tests .. Removed Code-Review+1 by Fredy Wijaya -- To view, visit http://gerrit.cloudera.org:8080/10442 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: deleteVote Gerrit-Change-Id: Ic4cf3a4751b41908ef81ec35d89a2713d9fa0dc4 Gerrit-Change-Number: 10442 Gerrit-PatchSet: 2 Gerrit-Owner: Adam Holley Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10611 ) Change subject: IMPALA-7128 (part 1) Refactor interfaces for Db, View, Table, Partition .. Patch Set 2: (4 comments) Responded to the comments that might merit discussion. I'll work on addressing the nits, etc. http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeDb.java File fe/src/main/java/org/apache/impala/catalog/FeDb.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeDb.java@31 PS2, Line 31: @return > Both styles exist (javadoc or not) in the code base, but would be preferabl k, will switch to javadoc. I think the lack of consistency here is because some comments were copy-pasted whereas others were written anew :) http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java File fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/FeHdfsTable.java@36 PS2, Line 36: Note: despite the "HDFS" nomenclature, this table type is also used for : * interacting with S3 or other Hadoop-compatible filesystems. > yup! shall we adopt the more general, preferred naming now? I had tried to just be consistent that 'Foo' class turned into 'FeFoo' interface, but if you think it's a better idea to bite the bullet and switch now I'm happy to do so. http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java File fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsPartition.java@964 PS2, Line 964: KeyValueComparator > this change can introduce differences depending on how partitions are dealt Partitions are used in Collections.sort, which requires the class to be comparable to itself. Previously HdfsPartition was Comparable, but with an interface, it becomes a bit trickier. One approach is to make FeHdfsPartition extend Comparable, but then all implementation classes need to implement compareTo(). And in fact all implementation classes will implement it exactly the same (ie using comparePartitionKeyValues, which only relies on the method that's part of the interface). If we were on Java 8, we could use a default method in the FeHdfsPartition interface to do this, but since we have to be Java 7-compatible, we can't do this. Given that, I felt like it was better to stop using the Comparable interface and instead use an explicit Comparator class. Regarding the concern about behavior differences in collections, I answered a similar one elsewhere but just to re-iterate here: I don't believe there are any collections in Java that would differ in behavior based on whether an object implements Comparable, unless they explicitly restrict themselves to be used with a Comparable object at compile-time. i.e any collections relying on Comparable would fail to compile (rather than change behavior) when I removed the Comparable interface from HdfsPartition. I did a quick check of the above by grepping the JDK source for 'instanceof Comparable' and didn't find any in any collection classes. http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java File fe/src/main/java/org/apache/impala/catalog/HdfsTable.java: http://gerrit.cloudera.org:8080/#/c/10611/2/fe/src/main/java/org/apache/impala/catalog/HdfsTable.java@2016 PS2, Line 2016: HdfsPartition.KV_COMPARATOR > how did you find all places to make this type of update, Collections.sort o The Collections.sort(List) signature only matches if T implements Comparable, so when I removed the Comparable interface from the HdfsPartition class, I got compilation errors at all the call sites where it was relying on it. The same would be true of other things like TreeMap -- it's not possible to instantiate one without either the class being Comparable or by providing a Comparator. -- To view, visit http://gerrit.cloudera.org:8080/10611 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id55f7d2e94d81e66ce720acb6315f15a89621b31 Gerrit-Change-Number: 10611 Gerrit-PatchSet: 2 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 21:50:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6035: Add query options to limit thread reservation
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10365 ) Change subject: IMPALA-6035: Add query options to limit thread reservation .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2623/ -- To view, visit http://gerrit.cloudera.org:8080/10365 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I5b5bbbdad5cd6b24442eb6c99a4d38c2ad710007 Gerrit-Change-Number: 10365 Gerrit-PatchSet: 9 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 21:30:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6917: Implement COMMENT ON TABLE/VIEW
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/10478 ) Change subject: IMPALA-6917: Implement COMMENT ON TABLE/VIEW .. IMPALA-6917: Implement COMMENT ON TABLE/VIEW This patch implements updating comment on a table or view. Syntax: COMMENT ON TABLE t IS 'comment' COMMENT ON VIEW v IS 'comment' Testing: - Added new front-end tests - Ran all front-end tests - Added new end-to-end tests - Ran end-to-end DDL tests Change-Id: I497c17342f79ff7c99931fd8a0ddec0c79303dbf --- M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup M fe/src/main/java/org/apache/impala/analysis/CommentOnDbStmt.java M fe/src/main/java/org/apache/impala/analysis/CommentOnStmt.java A fe/src/main/java/org/apache/impala/analysis/CommentOnTableOrViewStmt.java A fe/src/main/java/org/apache/impala/analysis/CommentOnTableStmt.java A fe/src/main/java/org/apache/impala/analysis/CommentOnViewStmt.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 13 files changed, 339 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/78/10478/7 -- To view, visit http://gerrit.cloudera.org:8080/10478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I497c17342f79ff7c99931fd8a0ddec0c79303dbf Gerrit-Change-Number: 10478 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-6917: Implement COMMENT ON TABLE/VIEW
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10478 ) Change subject: IMPALA-6917: Implement COMMENT ON TABLE/VIEW .. Patch Set 7: Rebased. Can anyone carry +2 for this? -- To view, visit http://gerrit.cloudera.org:8080/10478 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I497c17342f79ff7c99931fd8a0ddec0c79303dbf Gerrit-Change-Number: 10478 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 21:27:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10471 ) Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER .. Patch Set 9: Rebased -- To view, visit http://gerrit.cloudera.org:8080/10471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Gerrit-Change-Number: 10471 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 21:25:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7016: Implement ALTER DATABASE SET OWNER
Fredy Wijaya has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/10471 ) Change subject: IMPALA-7016: Implement ALTER DATABASE SET OWNER .. IMPALA-7016: Implement ALTER DATABASE SET OWNER Alter the database owner to either user or role. On database creation, the database owner will be set to the current user, which can be viewed via DESCRIBE DATABASE db command. Having an owner information allows implementing a feature where an owner can be given certain privileges automatically upon a database creation. The ALTER DATABASE SET OWNER will be a useful command for transferring ownership (a set of owner privileges) from the current owner to another owner. Syntax: ALTER DATABASE db SET OWNER USER user ALTER DATABASE db SET OWNER ROLE role Testing: - Added new front-end tests - Added new end-to-end tests Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 --- M common/thrift/CatalogService.thrift M common/thrift/JniCatalog.thrift M fe/src/main/cup/sql-parser.cup A fe/src/main/java/org/apache/impala/analysis/AlterDbSetOwnerStmt.java A fe/src/main/java/org/apache/impala/analysis/AlterDbStmt.java M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java A fe/src/main/java/org/apache/impala/analysis/Owner.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/Frontend.java M fe/src/main/java/org/apache/impala/util/MetaStoreUtil.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeDDLTest.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java M fe/src/test/java/org/apache/impala/analysis/ParserTest.java M tests/metadata/test_ddl.py M tests/metadata/test_ddl_base.py 15 files changed, 361 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10471/9 -- To view, visit http://gerrit.cloudera.org:8080/10471 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ie3b923021ebce5192d2d64784e7ddb952ba82bc3 Gerrit-Change-Number: 10471 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Alex Behm Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not required in comments
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10541 ) Change subject: IMPALA-2751: Matching quotes are not required in comments .. Patch Set 6: Rebased -- To view, visit http://gerrit.cloudera.org:8080/10541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c Gerrit-Change-Number: 10541 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 21:23:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not required in comments
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/10541 ) Change subject: IMPALA-2751: Matching quotes are not required in comments .. IMPALA-2751: Matching quotes are not required in comments This patch fixes the issue where non-matching quotes inside comments will cause the shell to not terminate. The fix is to strip any SQL comments before sending to shlex since shlex does not understand SQL comments and will raise an exception when it sees unmatched quotes regardless whether the quotes are in the comments or not. Testing: - Added new shell tests - Ran all end-to-end shell tests on Python 2.6 and Python 2.7 Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 26 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/41/10541/6 -- To view, visit http://gerrit.cloudera.org:8080/10541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c Gerrit-Change-Number: 10541 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. IMPALA-7106: Print rewritten SQL correctly when log trace is enabled toSql() method is used to print SQL string that is close to the original SQL string, which is something that users want to see. When debugging issues related to SQL rewrites, it can be very useful to be able to print/get the SQL string that is being rewritten. This patch adds a new method toRewrittenSql() to get the rewritten SQL string. The LOG.trace statement that prints the rewritten SQL is also updated to use toRewrittenSql(). Testing: - Added FE test for the rewritten SQL string - Ran all FE tests Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java 9 files changed, 145 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10571/8 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7106: Print rewritten SQL correctly when log trace is enabled
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Print rewritten SQL correctly when log trace is enabled .. Patch Set 8: Rebased. -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Thu, 07 Jun 2018 21:22:30 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7144: Re-enable TestDescribeTableResults
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10643 Change subject: IMPALA-7144: Re-enable TestDescribeTableResults .. IMPALA-7144: Re-enable TestDescribeTableResults This patch makes the TestDescribeTableResults more robust by only comparing the information that the authorization cares about instead of comparing all output in DESCRIBE. This change will avoid any unnecessary changes to AuthorizationTest if HMS updates the DESCRIBE output. The test is also updated to support standalone execution without relying on other tests be executed first since it can cause the test to be flaky especially if the tests in AuthorizationTest are executed in parallel. Testing: - Ran all FE tests Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java 1 file changed, 142 insertions(+), 182 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/43/10643/2 -- To view, visit http://gerrit.cloudera.org:8080/10643 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3aeaecf5b6d906a66d338e165a6d506e3964563f Gerrit-Change-Number: 10643 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Hello Tim Armstrong, Bikramjeet Vig, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10415 to look at the new patch set (#4). Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query .. IMPALA-6034: Add Cpu and scanned bytes limits per query To protect Impala from runaway queries add per query limits for Cpu and scanned bytes, limits are enforced via query options and applied to the entire query, not per backend like mem_limit. If a query exceeds any of the limits it will get cancelled. Query profile is updated to include query wide and per backend metrics for Cpu and scanned bytes. Testing: Added tests for various permutations for MAX_CPU_TIME_S and MAX_SCAN_BYTES Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 --- M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A testdata/workloads/functional-query/queries/QueryTest/query-resource-limits.test A tests/query_test/test_query_resource_limits.py 13 files changed, 376 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/15/10415/4 -- To view, visit http://gerrit.cloudera.org:8080/10415 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c6015e21da684bb9f33e236d71309dd4c178a20 Gerrit-Change-Number: 10415 Gerrit-PatchSet: 4 Gerrit-Owner: Mostafa Mokhtar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6034: Add Cpu and scanned bytes limits per query
Mostafa Mokhtar has posted comments on this change. (
http://gerrit.cloudera.org:8080/10415 )
Change subject: IMPALA-6034: Add Cpu and scanned bytes limits per query
..
Patch Set 3:
(22 comments)
http://gerrit.cloudera.org:8080/#/c/10415/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10415/3//COMMIT_MSG@10
PS3, Line 10: apply
> nit: applied
Done
http://gerrit.cloudera.org:8080/#/c/10415/3//COMMIT_MSG@14
PS3, Line 14: upated
> updated
Done
http://gerrit.cloudera.org:8080/#/c/10415/3//COMMIT_MSG@18
PS3, Line 18:
> nit: extra whitespace
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h
File be/src/runtime/coordinator-backend-state.h:
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h@113
PS3, Line 113: fragments
> nit: fragment
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h@200
PS3, Line 200: /// used by Coordinator::BackendState::AggregateBackendStats
> maybe mention units. (same for cpu_sys_)
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h@201
PS3, Line 201: cpu_user_
> nit: how about cpu_user_time_
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.h@207
PS3, Line 207: BYTES_READ_COUNTERs in profile_
> nit: Collection of BYTES_READ_COUNTERs of all the scan nodes in this fragme
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@213
PS3, Line 213: void Coordinator::BackendState::GetBackendResourceUtilization(
> We could just return BackendResourceUtilization by value and achieve the sa
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@341
PS3, Line 341: AggregateBackendStats
> do we need this call here? are any of the instance stats used in AggregateB
Coordinator::ComputeQuerySummary calls
Coordinator::BackendState::UpdateExecStats before the final metrics are written
to the query profile, otherwise stale info will be written.
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@527
PS3, Line 527: RuntimeProfile::Counter* profile_user_time_counter =
profile_->GetCounter("TotalThreadsUserTime");
> nit: long line
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@533
PS3, Line 533: RuntimeProfile::Counter* profile_system_time_counter =
profile_->GetCounter("TotalThreadsSysTime");
> nit: long line
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator-backend-state.cc@638
PS3, Line 638: "peak_mem_consumption",
resource_utilization_.peak_consumption, document->GetAllocator());
> nit:long line
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.h
File be/src/runtime/coordinator.h:
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.h@173
PS3, Line 173: peak_consumption
> nit: how about peak_mem_consumption
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.h@176
PS3, Line 176: backend_
> nit: we can probably get rid of this prefix now that all these counters are
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.h@185
PS3, Line 185: Aggregate CPU and bytes read metrics
> update comment: Aggregate CPU, scanned bytes and peak memory consumption me
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.cc@725
PS3, Line 725: info
> unused variable
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/runtime/coordinator.cc@732
PS3, Line 732: peak_memory = backend_resource_utilization.peak_consumption;
: backend_user_cpu =
backend_resource_utilization.backend_cpu_user;
: backend_sys_cpu =
backend_resource_utilization.backend_cpu_sys;
: backend_scanned_bytes =
backend_resource_utilization.backend_scanned_bytes;
> nit: can we directly use the values in the struct?
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/service/impala-server.cc@1011
PS3, Line 1011: queries_by_timestamp_.emplace(ExpirationEvent{
> We only need to queue one expiration event if both max_cpu_time_s and max_s
Done
http://gerrit.cloudera.org:8080/#/c/10415/3/be/src/service/impala-server.cc@1909
PS3, Line 1909: // 4. Check and cancel queries with Cpu and scan bytes
constraints if limit is exceeded
> nit:long line
Done
[Impala-ASF-CR] IMPALA-7071: make get fs path() idempotent
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10517 ) Change subject: IMPALA-7071: make get_fs_path() idempotent .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2622/ -- To view, visit http://gerrit.cloudera.org:8080/10517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c7cbb36119883718e60660db2ac7313f14d388 Gerrit-Change-Number: 10517 Gerrit-PatchSet: 4 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 21:04:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7071: make get fs path() idempotent
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10517 ) Change subject: IMPALA-7071: make get_fs_path() idempotent .. Patch Set 3: Sure -- To view, visit http://gerrit.cloudera.org:8080/10517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c7cbb36119883718e60660db2ac7313f14d388 Gerrit-Change-Number: 10517 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 21:04:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7071: make get fs path() idempotent
Dan Hecht has posted comments on this change. ( http://gerrit.cloudera.org:8080/10517 ) Change subject: IMPALA-7071: make get_fs_path() idempotent .. Patch Set 4: Code-Review+2 Carry +2 -- To view, visit http://gerrit.cloudera.org:8080/10517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c7cbb36119883718e60660db2ac7313f14d388 Gerrit-Change-Number: 10517 Gerrit-PatchSet: 4 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 21:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not required in comments
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10541 ) Change subject: IMPALA-2751: Matching quotes are not required in comments .. Patch Set 5: > Patch Set 5: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2618/ Failed because of IMPALA-7143, I'll rebase later. -- To view, visit http://gerrit.cloudera.org:8080/10541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c Gerrit-Change-Number: 10541 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 20:00:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-2751: Matching quotes are not required in comments
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10541 ) Change subject: IMPALA-2751: Matching quotes are not required in comments .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2618/ -- To view, visit http://gerrit.cloudera.org:8080/10541 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2feae34026a7e63f3d31489f757f093a73ca5d2c Gerrit-Change-Number: 10541 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:58:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Attila Jeges has uploaded a new patch set (#13). ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. IMPALA-3307: Add support for IANA time-zone db Impala currently uses two different libraries for timestamp manipulations: boost and glibc. Issues with boost: - Time-zone database is currently hard coded in timezone_db.cc. Impala admins cannot update it without upgrading Impala. - Time-zone database is flat, therefore can’t track year-to-year changes. - Time-zone database is not updated on a regular basis. Issues with glibc: - Uses /usr/share/zoneinfo/ database which could be out of sync on some of the nodes in the Impala cluster. - Uses the host system’s local time-zone. Different nodes in the Impala cluster might use a different local time-zone. - Conversion functions take a global lock, which causes severe performance degradation. In addition to the issues above, the fact that /usr/share/zoneinfo/ and the hard-coded boost time-zone database are both in use is a source of inconsistency in itself. This patch makes the following changes: - Instead of boost and glibc, impalad uses Google's CCTZ to implement time-zone conversions. - Introduces a new startup flag (--hdfs_zone_info_zip) to impalad to specify an HDFS/S3/ADLS path to a zip archive that contains the shared compiled IANA time-zone database. If the startup flag is set, impalad will use the specified time-zone database. Otherwise, impalad will use the default /usr/share/zoneinfo time-zone database. - impalad reads the entire time-zone database into an in-memory map on startup for fast lookups. - The name of the coordinator node’s local time-zone is saved to the query context when preparing query execution. This time-zone is used whenever the current time-zone is referred afterwards in an execution node. - Introduces a new startup flag (--hdfs_zone_abbrev_conf) to impalad to specify an HDFS/S3/ADLS path to a shared config file that contains definitions for non-standard time-zone abbreviations. Cherry-picks: not for 2.x. Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 --- M CMakeLists.txt M be/CMakeLists.txt M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/benchmarks/CMakeLists.txt A be/src/benchmarks/convert-timestamp-benchmark.cc M be/src/common/global-types.h M be/src/common/init.cc M be/src/exec/data-source-scan-node.cc M be/src/exec/data-source-scan-node.h M be/src/exec/hdfs-orc-scanner.cc M be/src/exec/parquet-column-readers.cc M be/src/exprs/CMakeLists.txt M be/src/exprs/aggregate-functions-ir.cc M be/src/exprs/cast-functions-ir.cc M be/src/exprs/decimal-operators-ir.cc M be/src/exprs/decimal-operators.h M be/src/exprs/expr-test.cc M be/src/exprs/literal.cc M be/src/exprs/timestamp-functions-ir.cc M be/src/exprs/timestamp-functions.cc A be/src/exprs/timezone_db-test.cc M be/src/exprs/timezone_db.cc M be/src/exprs/timezone_db.h M be/src/runtime/raw-value-test.cc M be/src/runtime/runtime-state.cc M be/src/runtime/runtime-state.h M be/src/runtime/timestamp-test.cc M be/src/runtime/timestamp-value.cc M be/src/runtime/timestamp-value.h M be/src/runtime/timestamp-value.inline.h M be/src/service/frontend.cc M be/src/service/impala-server.cc M be/src/service/impalad-main.cc M be/src/util/CMakeLists.txt M be/src/util/filesystem-util-test.cc M be/src/util/filesystem-util.cc M be/src/util/filesystem-util.h M be/src/util/hdfs-util-test.cc M be/src/util/hdfs-util.cc M be/src/util/hdfs-util.h M be/src/util/time-test.cc M be/src/util/time.cc M be/src/util/time.h A be/src/util/zip-util-test.cc A be/src/util/zip-util.cc A be/src/util/zip-util.h M bin/bootstrap_toolchain.py M bin/impala-config.sh M bin/rat_exclude_files.txt A cmake_modules/FindCctz.cmake M common/thrift/CMakeLists.txt M common/thrift/ImpalaInternalService.thrift A common/thrift/Zip.thrift M common/thrift/metrics.json A fe/src/main/java/org/apache/impala/util/ZipUtil.java M fe/src/test/java/org/apache/impala/testutil/TestUtils.java M testdata/bin/create-load-data.sh M testdata/data/timezoneverification.csv A testdata/tzdb/2017c-corrupt.zip A testdata/tzdb/2017c.zip A testdata/tzdb/abbrev.conf A testdata/tzdb_tiny/America/New_York A testdata/tzdb_tiny/Etc/GMT+4 A testdata/tzdb_tiny/US/Eastern A testdata/tzdb_tiny/UTC A testdata/tzdb_tiny/Zulu A testdata/tzdb_tiny/posix/UTC A testdata/tzdb_tiny/posixrules M testdata/workloads/functional-query/queries/QueryTest/exprs.test M tests/custom_cluster/test_hive_parquet_timestamp_conversion.py A tests/custom_cluster/test_shared_tzdb.py D tests/query_test/test_timezones.py 72 files changed, 3,098 insertions(+), 1,167 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/86/9986/13 -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master
[Impala-ASF-CR] IMPALA-3307: Add support for IANA time-zone db
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/9986 ) Change subject: IMPALA-3307: Add support for IANA time-zone db .. Patch Set 11: (2 comments) Added some tests for extracting files from a non-existent zip archive and from a corrupt zip archive. http://gerrit.cloudera.org:8080/#/c/9986/11/be/src/exprs/timezone_db.cc File be/src/exprs/timezone_db.cc: http://gerrit.cloudera.org:8080/#/c/9986/11/be/src/exprs/timezone_db.cc@198 PS11, Line 198: GetNextDirectoryEntry > This is subjective, but I do not like this interface too much. I would pref Done http://gerrit.cloudera.org:8080/#/c/9986/11/be/src/exprs/timezone_db.cc@213 PS11, Line 213: readdir_r > There was a discussion about readdir_r() vs readdir() in https://gerrit.clo Done -- To view, visit http://gerrit.cloudera.org:8080/9986 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I93c1fbffe81f067919706e30db0a34d0e58e7e77 Gerrit-Change-Number: 9986 Gerrit-PatchSet: 11 Gerrit-Owner: Attila Jeges Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Jun 2018 19:49:52 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Vincent Tran has posted comments on this change. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/10605/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/10605/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@1100 PS3, Line 1100: returns a single row Does this one need to be changed to be inline with 4. above as well? -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Jun 2018 19:37:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10488 ) Change subject: IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2621/ -- To view, visit http://gerrit.cloudera.org:8080/10488 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie456c87c34e7d75eb7482f499e90d50de740df40 Gerrit-Change-Number: 10488 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:27:10 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10488 ) Change subject: IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10488 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie456c87c34e7d75eb7482f499e90d50de740df40 Gerrit-Change-Number: 10488 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:27:00 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10488 ) Change subject: IMPALA-7062: fix unsafe RuntimeProfile::SortChildren() function .. Patch Set 3: I think the risk is limited here, I'll rebase and merge. -- To view, visit http://gerrit.cloudera.org:8080/10488 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie456c87c34e7d75eb7482f499e90d50de740df40 Gerrit-Change-Number: 10488 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:26:51 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7044: Prevent overflow when computing Parquet block size
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10483 ) Change subject: IMPALA-7044: Prevent overflow when computing Parquet block size .. Patch Set 6: (1 comment) Do you want to go ahead and merge this? Seems low risk. http://gerrit.cloudera.org:8080/#/c/10483/6/be/src/exec/hdfs-parquet-table-writer.cc File be/src/exec/hdfs-parquet-table-writer.cc: http://gerrit.cloudera.org:8080/#/c/10483/6/be/src/exec/hdfs-parquet-table-writer.cc@985 PS6, Line 985: int64_t static_cast -- To view, visit http://gerrit.cloudera.org:8080/10483 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6e63420e5a093c0bbc789201771708865b16e138 Gerrit-Change-Number: 10483 Gerrit-PatchSet: 6 Gerrit-Owner: Lars Volker Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:24:17 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7071: make get fs path() idempotent
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10517 ) Change subject: IMPALA-7071: make get_fs_path() idempotent .. Patch Set 3: Do you want to go ahead and merge this? Seems low risk. -- To view, visit http://gerrit.cloudera.org:8080/10517 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c7cbb36119883718e60660db2ac7313f14d388 Gerrit-Change-Number: 10517 Gerrit-PatchSet: 3 Gerrit-Owner: Dan Hecht Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: David Knupp Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:23:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5737: Tighten minicluster memory limit
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10277 ) Change subject: IMPALA-5737: Tighten minicluster memory limit .. Patch Set 8: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2620/ -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 8 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:22:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5737: Tighten minicluster memory limit
Tianyi Wang has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/10277 ) Change subject: IMPALA-5737: Tighten minicluster memory limit .. IMPALA-5737: Tighten minicluster memory limit This patch limits minicluster memory based on the memory available. Two memory profiles are introduced. If there is less than 32GB memory available, memory limit of the minicluster will be set to ~27GB. Otherwise it will be set to ~32GB. This will hopefully help with minicluster with more than 3 nodes. The 27GB profile works on m2.4xlarge which has 8 cores. It should be more than enough for 4-core machines. Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe --- M bin/impala-config.sh M bin/start-impala-cluster.py M testdata/bin/run-hbase.sh M testdata/bin/run-hive-server.sh M testdata/bin/run-sentry-service.sh M testdata/cluster/node_templates/common/etc/init.d/common.tmpl M testdata/cluster/node_templates/common/etc/init.d/hdfs-common M testdata/cluster/node_templates/common/etc/init.d/kudu-common 8 files changed, 91 insertions(+), 22 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/77/10277/8 -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 8 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5737: Tighten minicluster memory limit
Tianyi Wang has posted comments on this change. ( http://gerrit.cloudera.org:8080/10277 ) Change subject: IMPALA-5737: Tighten minicluster memory limit .. Patch Set 7: > Patch Set 7: Verified-1 > > Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2598/ The last GVO failure shows that a successful DELETE query doesn't take effect in KUDU. Perhaps Kudu needs more memory. I will rebase and run a precommit test. -- To view, visit http://gerrit.cloudera.org:8080/10277 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: If8240551e726c6da546a926a1ce3444f41ef87fe Gerrit-Change-Number: 10277 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:19:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7108: IllegalStateException hit during CardinalityCheckNode.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10605 ) Change subject: IMPALA-7108: IllegalStateException hit during CardinalityCheckNode. .. Patch Set 3: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/10605/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java: http://gerrit.cloudera.org:8080/#/c/10605/3/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@1082 PS3, Line 1082: returns a single row. I think this bit is still inaccurate. -- To view, visit http://gerrit.cloudera.org:8080/10605 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I82a7a3fe26db3e12131c030c4ad055a9c4955407 Gerrit-Change-Number: 10605 Gerrit-PatchSet: 3 Gerrit-Owner: Zoltan Borok-Nagy Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vincent Tran Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Jun 2018 19:15:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7143: disable affected describe formatted/extended tests
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10635 ) Change subject: IMPALA-7143: disable affected describe formatted/extended tests .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10635 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I47651e22881e9f561ce195b45e00c47d9f3c636a Gerrit-Change-Number: 10635 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:12:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Print logs around removeDirective()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10640 ) Change subject: IMPALA-3040: Print logs around removeDirective() .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2619/ -- To view, visit http://gerrit.cloudera.org:8080/10640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814 Gerrit-Change-Number: 10640 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:10:44 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6119: Fix issue with multiple partitions sharing same location
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/10543 ) Change subject: IMPALA-6119: Fix issue with multiple partitions sharing same location .. Patch Set 9: Verified-1 FYI, apparently test_alter_partition_location fails due to either my last change or some code what I got from the latest rebase. I'm working on it, expect a fix soon. -- To view, visit http://gerrit.cloudera.org:8080/10543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a54bc8224bcefe65b83de2df58bb84629f2aa4a Gerrit-Change-Number: 10543 Gerrit-PatchSet: 9 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Thu, 07 Jun 2018 19:08:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7146: log session to query mapping
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10638 ) Change subject: IMPALA-7146: log session to query mapping .. Patch Set 2: Oops verified the wrong thing... I guess leaving it is less bad than reverting it. -- To view, visit http://gerrit.cloudera.org:8080/10638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d66f41b829ab49b2e0670626bb79241dddff11a Gerrit-Change-Number: 10638 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:07:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7146: log session to query mapping
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10638 ) Change subject: IMPALA-7146: log session to query mapping .. IMPALA-7146: log session to query mapping Example output: I0607 11:12:25.542639 127653 impala-server.cc:962] Registered query query_id=894557dcebc637dd:743b75d1 session_id=c648cfc1ec86e3ae:7a071baf052f51ae Testing: Manually checked that both HS2 and Beeswax queries get the right log message. Change-Id: I6d66f41b829ab49b2e0670626bb79241dddff11a Reviewed-on: http://gerrit.cloudera.org:8080/10638 Reviewed-by: Philip Zeyliger Tested-by: Tim Armstrong --- M be/src/service/impala-server.cc 1 file changed, 2 insertions(+), 0 deletions(-) Approvals: Philip Zeyliger: Looks good to me, approved Tim Armstrong: Verified -- To view, visit http://gerrit.cloudera.org:8080/10638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6d66f41b829ab49b2e0670626bb79241dddff11a Gerrit-Change-Number: 10638 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7146: log session to query mapping
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10638 ) Change subject: IMPALA-7146: log session to query mapping .. Patch Set 1: Verified+1 $ ./bin/compare_branches.py 2018-06-07 12:06:20,190 MainThread WARNING: Commit e2e7c103a7ac87d848eed3138841ae5d2ec32863 contains multiple change ids; using first one. 2018-06-07 12:06:20,190 MainThread WARNING: Commit ea698cd497f63908b231e4b108c20d259e5bd8fb contains multiple change ids; using first one. 2018-06-07 12:06:20,191 MainThread WARNING: Commit 2c0926e2de22ecafafc460f2b31ca2423b8f7e98 (Revert "IMPALA-6759: align stress test m...) has no Change-Id. 2018-06-07 12:06:20,204 MainThread WARNING: Commit ffac1ab48c55084e13cc1dd517c25e37e48adc31 contains multiple change ids; using first one. 2018-06-07 12:06:20,205 MainThread WARNING: Commit d8815fb3802ae0ee20e0f30833b5185c914755e6 contains multiple change ids; using first one. 2018-06-07 12:06:20,206 MainThread WARNING: Commit ecf972a5b1d933d5d4f9f8f7658e8df900594b06 contains multiple change ids; using first one. Commits in asf-gerrit/master but not in asf-gerrit/2.x: Jira keys referenced (Note: not all commit messages will reference a jira key): -- To view, visit http://gerrit.cloudera.org:8080/10638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d66f41b829ab49b2e0670626bb79241dddff11a Gerrit-Change-Number: 10638 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:06:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Print logs around removeDirective()
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10640 ) Change subject: IMPALA-3040: Print logs around removeDirective() .. Patch Set 2: Code-Review+2 Makes sense. Let's merge this so we debug this issue. -- To view, visit http://gerrit.cloudera.org:8080/10640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814 Gerrit-Change-Number: 10640 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Thu, 07 Jun 2018 19:05:51 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Ignore "IMPALA-7143: disable affected describe formatted/extended tests"
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10639 ) Change subject: Ignore "IMPALA-7143: disable affected describe formatted/extended tests" .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10639 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: I1f0c1b79c7d5d2fec24eae09441a5a56554ce59f Gerrit-Change-Number: 10639 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 07 Jun 2018 19:04:27 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) Ignore "IMPALA-7143: disable affected describe formatted/extended tests"
Tim Armstrong has removed Lars Volker from this change. ( http://gerrit.cloudera.org:8080/10639 ) Change subject: Ignore "IMPALA-7143: disable affected describe formatted/extended tests" .. Removed reviewer Lars Volker. -- To view, visit http://gerrit.cloudera.org:8080/10639 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: deleteReviewer Gerrit-Change-Id: I1f0c1b79c7d5d2fec24eae09441a5a56554ce59f Gerrit-Change-Number: 10639 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7146: log session to query mapping
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10638 ) Change subject: IMPALA-7146: log session to query mapping .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10638 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d66f41b829ab49b2e0670626bb79241dddff11a Gerrit-Change-Number: 10638 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Thu, 07 Jun 2018 18:58:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-3040: Print logs around removeDirective()
Tianyi Wang has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10640 ) Change subject: IMPALA-3040: Print logs around removeDirective() .. IMPALA-3040: Print logs around removeDirective() I observed that some caching directives on partitions don't get removed. HDFS would throw an exception if removeDirective() operation fails on its side, so I suspect that removeDirective() isn't called by catalogd. This patch prints logs around removeDirective() if it's skipped. Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814 --- M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java 1 file changed, 22 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/10640/2 -- To view, visit http://gerrit.cloudera.org:8080/10640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814 Gerrit-Change-Number: 10640 Gerrit-PatchSet: 2 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-3040: Print logs around removeDirective()
Tianyi Wang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10640 Change subject: IMPALA-3040: Print logs around removeDirective() .. IMPALA-3040: Print logs around removeDirective() I observed that some caching directives on partitions doesn't get removed. HDFS would throw an exception if removeDirective operation fails on its side, so I suspect that removeDirective isn't called by catalogd. This patch prints logs around removeDirective if it's skipped. Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814 --- M fe/src/main/java/org/apache/impala/util/HdfsCachingUtil.java 1 file changed, 23 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/10640/1 -- To view, visit http://gerrit.cloudera.org:8080/10640 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I9d9f6eb1951c77e095d5029e8bbdc67e9b39b814 Gerrit-Change-Number: 10640 Gerrit-PatchSet: 1 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10510 )
Change subject: IMPALA-5552: Add support for authorized proxy groups
..
Patch Set 16:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/frontend.cc
File be/src/service/frontend.cc:
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/frontend.cc@61
PS14, Line 61: "Specifies the set of authorized proxy groups (users who can
delegate to other "
: "users belonging to the specified groups during
authorization) and who
> For other components that have done this, how have they documented this?
Borrowing some words from:
https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Superusers.html
Done.
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/frontend.cc@66
PS14, Line 66: "may be changed via
--authorized_proxy_group_config_delimiter), or '*' to indicate "
> What does "*" mean here? If you wanted to be able to delegate to anyone, yo
That's true, but then they're forced to use the user flag just for *. I think
it's nice to also have * in the group flag since users don't have to user the
user flag if they don't want to. As a reference, Oozie supports * in groups:
http://doc.mapr.com/display/MapR/User+Impersonation+for+Oozie
I'm open to removing * if you still think it doesn't make any sense for groups.
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.h
File be/src/service/impala-server.h:
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.h@1031
PS14, Line 1031: /// Map of short usernames of authorized proxy users to the
set of groups they are
> nit: "set of groups" and ("set of users" on line 1027). There's no reading
Done
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@285
PS14, Line 285: LOG(ERROR) << status.GetDetail();
> I believe the way we typically do this is TBackendGflags. I think you can u
Done
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@417
PS14, Line 417: boost::trim(config_str);
> The old impl didn't trim. I think that's fine, but wanted to check that's a
I'll update the comment. I believe user and group names don't allow spaces.
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@1382
PS14, Line 1382: users.find(do_as_user) != users.end()) {
> I think we're slowly migrating from boost::unordered_set to the c++11 equiv
I can change all boost::unordered_set into std::unordered_set, but it's
probably better to not do it in this CR.
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@1393
PS14, Line 1393: if (groups.find("*") != groups.end()) return
Status::OK();
> You're referring to proxy_group here without checking if it's groups.end()
Done
http://gerrit.cloudera.org:8080/#/c/10510/14/be/src/service/impala-server.cc@1402
PS14, Line 1402: if (!status.ok()) {
> If we're going to log it, let's include the short_user here as well.
Done
--
To view, visit http://gerrit.cloudera.org:8080/10510
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb
Gerrit-Change-Number: 10510
Gerrit-PatchSet: 16
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Greg Rahn
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Thu, 07 Jun 2018 18:38:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5552: Add support for authorized proxy groups
Fredy Wijaya has uploaded a new patch set (#16). ( http://gerrit.cloudera.org:8080/10510 ) Change subject: IMPALA-5552: Add support for authorized proxy groups .. IMPALA-5552: Add support for authorized proxy groups The patch adds support for mapping of users to a list of proxy groups. The following flags are added in impalad: - authorized_proxy_group_config - authorized_proxy_group_config_delimiter Example: --authorized_proxy_group_config=hue=group1,group2;user1=* This feature is not supported on Shell-based Hadoop groups mapping providers. The authorized_proxy_user/group_config parser will now strip leading and trailing whitespaces from the user/group names. Testing: - Added FE unit test to check for groups mapping provider - Added BE unit test for the parsing logic - Added a new test in test_authorization.py - Ran all end-to-end test_authorization.py Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb --- M be/src/service/CMakeLists.txt M be/src/service/frontend.cc M be/src/service/frontend.h A be/src/service/impala-server-test.cc M be/src/service/impala-server.cc M be/src/service/impala-server.h M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M common/thrift/Frontend.thrift M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java A fe/src/test/java/org/apache/impala/service/JniFrontendTest.java M tests/authorization/test_authorization.py 13 files changed, 384 insertions(+), 36 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/10/10510/16 -- To view, visit http://gerrit.cloudera.org:8080/10510 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6953f89c293b06b72f523b11802232133d9d6cbb Gerrit-Change-Number: 10510 Gerrit-PatchSet: 16 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Vuk Ercegovac
