[Impala-ASF-CR] IMPALA-6119: Fix issue with multiple partitions sharing same location
Gabor Kaszab has posted comments on this change. ( http://gerrit.cloudera.org:8080/10543 ) Change subject: IMPALA-6119: Fix issue with multiple partitions sharing same location .. Patch Set 20: Hey, Sorry for the delay, I had been busy with other stuff. Michael, no worries. Vuk, I think there is a misunderstanding here: I have already removed the HdfsTable member used for storing the mapping. In patch set 20 everything seems similar to the algorithm you described? Could you please take another look? Or am I missing something? -- To view, visit http://gerrit.cloudera.org:8080/10543 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2a54bc8224bcefe65b83de2df58bb84629f2aa4a Gerrit-Change-Number: 10543 Gerrit-PatchSet: 20 Gerrit-Owner: Gabor Kaszab Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Gabor Kaszab Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Sat, 14 Jul 2018 05:41:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6034: Add CPU and scanned bytes limits per query
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/10415 )
Change subject: IMPALA-6034: Add CPU and scanned bytes limits per query
..
Patch Set 9:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/10415/9//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10415/9//COMMIT_MSG@8
PS9, Line 8:
maybe mention the query_option names added in the patch. I know its mentioned
in the "testing" part but would still prefer that its mentioned explicitly in
the description.
http://gerrit.cloudera.org:8080/#/c/10415/9//COMMIT_MSG@10
PS9, Line 10: scanned bytes, limits are enforced via query options and applied
to the entire
nit: long line
http://gerrit.cloudera.org:8080/#/c/10415/9//COMMIT_MSG@14
PS9, Line 14: Query profile is updated to include query wide and per backend
metrics for Cpu
nit:long line
also include an example
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/runtime/coordinator-backend-state.cc@500
PS9, Line 500:
: // Update resource utilization and apply delta.
: ResourceUtilization prev_utilization = resource_utilization_;
: RuntimeProfile::Counter* user_time =
profile_->GetCounter("TotalThreadsUserTime");
: if (user_time != nullptr) resource_utilization_.cpu_user_ns =
user_time->value();
:
: RuntimeProfile::Counter* system_time =
profile_->GetCounter("TotalThreadsSysTime");
: if (system_time != nullptr) resource_utilization_.cpu_sys_ns =
system_time->value();
:
: int total_bytes = 0;
: for (RuntimeProfile::Counter* c: bytes_read_counters_)
total_bytes += c->value();
: resource_utilization_.bytes_read = total_bytes;
:
: RuntimeProfile::Counter* peak_mem =
:
profile_->GetCounter(FragmentInstanceState::PER_HOST_PEAK_MEM_COUNTER);
: if (peak_mem != nullptr)
resource_utilization_.peak_mem_consumption = peak_mem->value();
:
aggregate_utilization->Merge(resource_utilization_.Delta(prev_utilization));
I feel we should just defer this calculation to
BackendState::GetResourceUtilization(). Seems like we are only concerned with
the aggregate level ResourceUtilization if ComputeQuerySummary() is called
(only called at a terminal state) or if ComputeQueryResourceUtilization() is
called (which only happens if there is a resource limit on the query). This
deferred calculation will help get rid of BackendState::lock_ required in this
method and the logic around delta updates of ResourceUtilization. Moreover,
queries that dont have a resource limit wont have to pay for this
locking/updating cost everytime an update is recieved.
NOTE: if we envision resource management to always set a default limit on
resources then we can just stick with the current implementation as having
resources limits would be the norm rather than the exception.
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/runtime/coordinator-backend-state.cc@600
PS9, Line 600: value->AddMember("peak_mem_consumption",
resource_utilization_.peak_mem_consumption,
: document->GetAllocator());
maybe add other members of resource_utilization_ as well here
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/runtime/coordinator.h
File be/src/runtime/coordinator.h:
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/runtime/coordinator.h@171
PS9, Line 171: for this query
nit: since this struct can hold info for either finstance, backend and query
level, maybe clarify what we mean here.
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/service/impala-server.cc@2026
PS9, Line 2026: crs
love the variable name
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/service/impala-server.cc@2036
PS9, Line 2036: CheckResourceLimits
nit: how about we return a status from CheckResourceLimits(), and on a non-ok
status we call ExpireQuery(). This way its sounds intuitive what the name
CheckResourceLimits() is doing and its consistent with the how the rest of the
method behaves.
Dont have a strong preference so feel free to ignore this comment.
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/service/impala-server.cc@2125
PS9, Line 2125: TIME_NS
we can just use seconds here instead as the cpu_limit_s can only we defined in
seconds
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/service/query-options-test.cc
File be/src/service/query-options-test.cc:
http://gerrit.cloudera.org:8080/#/c/10415/9/be/src/service/query-options-test.cc@146
PS9, Line 146:
nit: not a huge deal but we can probably get this in
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. Patch Set 4: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 14 Jul 2018 01:03:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code from IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code changes in this patch, there is no code change for the shims. The shims for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default implementation. Testing: - Ran core and exhaustive tests Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Reviewed-on: http://gerrit.cloudera.org:8080/10940 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/create-test-configuration.sh M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh M fe/pom.xml D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/authorization/SentryAuthProvider.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/HdfsShim.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MetastoreShim.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MiniclusterProfile.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/util/SentryUtil.java D fe/src/compat-minicluster-profile-2/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java D fe/src/compat-minicluster-profile-3/java/org/apache/impala/compat/MiniclusterProfile.java R fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java R fe/src/main/java/org/apache/impala/authorization/ImpalaActionFactory.java R fe/src/main/java/org/apache/impala/authorization/ImpalaPrivilegeModel.java R fe/src/main/java/org/apache/impala/authorization/SentryAuthProvider.java R fe/src/main/java/org/apache/impala/compat/HdfsShim.java R fe/src/main/java/org/apache/impala/compat/MetastoreShim.java R fe/src/main/java/org/apache/impala/util/SentryUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java R fe/src/test/java/org/apache/impala/authorization/ImpalaActionFactoryTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java R fe/src/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java M impala-parent/pom.xml M testdata/bin/run-hbase.sh M testdata/bin/run-hive-server.sh M testdata/bin/run-mini-dfs.sh M testdata/cluster/node_templates/common/etc/init.d/common.tmpl M testdata/cluster/node_templates/common/etc/init.d/yarn-common M testdata/pom.xml M testdata/workloads/functional-query/queries/QueryTest/views-compatibility.test M tests/common/environ.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_partitioning.py 39 files changed, 237 insertions(+), 1,122 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum)
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10944 ) Change subject: IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum) .. IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum) In addition to previous fix for IMPALA-7288, this patch would prevent impala from crashing in case a code-path generates a malformed handcrafted function which it then tries to finalize. Ideally this would never happen since the code paths for generating handcrafted IRs would never generate a malformed function. Change-Id: Id09c6f59f677ba30145fb2081715f1a7d89fe20b Reviewed-on: http://gerrit.cloudera.org:8080/10944 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h 2 files changed, 3 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id09c6f59f677ba30145fb2081715f1a7d89fe20b Gerrit-Change-Number: 10944 Gerrit-PatchSet: 3 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10944 ) Change subject: IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum) .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id09c6f59f677ba30145fb2081715f1a7d89fe20b Gerrit-Change-Number: 10944 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Sat, 14 Jul 2018 00:20:02 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. Patch Set 9: Code-Review+1 I'll give Nghia a chance to look too. -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 23:25:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. Patch Set 9: Code-Review+1 Carry Mike's +1 -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 22:43:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails .. Patch Set 21: Code-Review+1 (6 comments) Carry Bharath's +1 http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java: http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@a100 PS18, Line 100: > That makes sense as it is much closer to the user written SQL than before. Done http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java File fe/src/main/java/org/apache/impala/analysis/UnionStmt.java: http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java@556 PS20, Line 556: > nit: merge Done http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java File fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java: http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java@81 PS20, Line 81: if (!rewritten && sqlString_ != null) return sqlString_; : : S > merge Done http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java: http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java@714 PS20, Line 714: WHERE a.int_col BETWEEN t.tinyint_col AND t.bigint_col) > This makes more sense Done http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java File fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java: http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@253 PS20, Line 253: > I wasn't aware of this till now :-) I was surprised this too :) http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@261 PS20, Line 261: : assertToSql(ctx, : "select * from (select * from functional.alltypes where id = (select 1 + 1)) a " + : "union " + : "select * from (select * from functional.alltypes where id = (select 1 + 1)) b", : "SELECT * FROM (SELECT * FROM functional.alltypes WHERE id = (SELECT 1 + 1)) a " + : "UNION " + : "SELECT * FROM (SELECT * FROM functional.alltypes W > can we factor these out into a method? Good idea. Done. -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 21 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 13 Jul 2018 22:42:30 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails
Fredy Wijaya has uploaded a new patch set (#21). ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails .. IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails toSql() method is used to print SQL string that is close to the original SQL string when errors arise or as the result of "SHOW CREATE". When debugging issues related to SQL rewrites, it can be very useful to be able to get the SQL string that is being rewritten. This patch adds a new method toSql(boolean rewritten) to get the rewritten SQL string. This patch also logs the original and rewritten SQL when SQL rewrite fails. Testing: - Added FE test for the rewritten SQL string - Ran all FE tests Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java 17 files changed, 198 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10571/21 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 21 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. Patch Set 9: (2 comments) http://gerrit.cloudera.org:8080/#/c/10939/8/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/10939/8/shell/impala_shell.py@72 PS8, Line 72: return ''.join(stack.run(sql, 'utf-8')).strip() > I noticed that sqlparse.format() uses an unicode string here: https://githu I don't think it makes a difference and I believe we use utf-8 everywhere in Impala shell. http://gerrit.cloudera.org:8080/#/c/10939/8/tests/shell/test_file_large.sql File tests/shell/test_file_large.sql: PS8: > Trying to boost your LoC count? :) LOL, I will auto-generate it instead. -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 22:19:02 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has uploaded a new patch set (#9). ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. IMPALA-7259: Improve Impala shell performance This patch fixes the slow performance in Impala shell, especially for large queries by replacing all calls to sqlparse.format(sql_string, strip_comments=True) with the custom implementation of strip comments that does not use grouping. The code to strip leading comments was also refactored to not use grouping. * Benchmark running a query with 12K columns * Before the patch: $ time impala-shell.sh -f large.sql --quiet real2m4.154s user2m0.536s sys 0m0.088s After the patch: $ time impala-shell.sh -f large.sql --quiet real0m3.885s user0m1.516s sys 0m0.048s Testing: - Added a new test to test the Impala shell performance - Ran all shell tests on Python 2.6 and Python 2.7 Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 --- M shell/impala_shell.py M tests/shell/test_shell_commandline.py 2 files changed, 70 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10939/9 -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 9 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7186: [DOCS] Documented the KUDU READ MODE query option
Hello Thomas Marshall, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10897 to look at the new patch set (#3). Change subject: IMPALA-7186: [DOCS] Documented the KUDU_READ_MODE query option .. IMPALA-7186: [DOCS] Documented the KUDU_READ_MODE query option Change-Id: I49b4ec29ae8cdbee8b3d38bdf2e678b4e9560952 --- M docs/impala.ditamap M docs/impala_keydefs.ditamap A docs/topics/impala_kudu_read_mode.xml 3 files changed, 92 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/10897/3 -- To view, visit http://gerrit.cloudera.org:8080/10897 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I49b4ec29ae8cdbee8b3d38bdf2e678b4e9560952 Gerrit-Change-Number: 10897 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Thomas Marshall
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 13 Jul 2018 21:45:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2816/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 13 Jul 2018 21:45:59 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. Patch Set 8: (2 comments) Thanks for fixing this! http://gerrit.cloudera.org:8080/#/c/10939/8/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/10939/8/shell/impala_shell.py@72 PS8, Line 72: return ''.join(stack.run(sql, 'utf-8')).strip() I noticed that sqlparse.format() uses an unicode string here: https://github.com/andialbrecht/sqlparse/blob/396f19d00fa20a8fc891ab9351bee64b334aac3a/sqlparse/__init__.py#L60 Does this make a difference? Or does '' get coerced to unicode anyway? http://gerrit.cloudera.org:8080/#/c/10939/8/tests/shell/test_file_large.sql File tests/shell/test_file_large.sql: PS8: Trying to boost your LoC count? :) Seriously though, this is ok, but it might be better to programmatically generate this in the test and write it to a temporary file. -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 21:20:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 13 Jul 2018 21:16:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10944 ) Change subject: IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum) .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id09c6f59f677ba30145fb2081715f1a7d89fe20b Gerrit-Change-Number: 10944 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 20:58:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum)
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10944 ) Change subject: IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum) .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2815/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id09c6f59f677ba30145fb2081715f1a7d89fe20b Gerrit-Change-Number: 10944 Gerrit-PatchSet: 2 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 20:58:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6810: runtime row filters.test: omit pool name in pattern
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10942 ) Change subject: IMPALA-6810: runtime_row_filters.test: omit pool name in pattern .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fe6beb169dc6bfefabde9dc7a4632c1a5e63fa7 Gerrit-Change-Number: 10942 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Brown Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 20:45:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6810: runtime row filters.test: omit pool name in pattern
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10942 ) Change subject: IMPALA-6810: runtime_row_filters.test: omit pool name in pattern .. IMPALA-6810: runtime_row_filters.test: omit pool name in pattern Some downstream tests run this with a fair-scheduler.xml set that, while not changing admission control behavior, does change the name of the pool. Omit the pool name to permit that downstream test to succeed. Testing: - local with change in minicluster - downstream in environment as well Change-Id: I3fe6beb169dc6bfefabde9dc7a4632c1a5e63fa7 Reviewed-on: http://gerrit.cloudera.org:8080/10942 Reviewed-by: Michael Brown Tested-by: Impala Public Jenkins --- M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test 1 file changed, 3 insertions(+), 4 deletions(-) Approvals: Michael Brown: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/10942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3fe6beb169dc6bfefabde9dc7a4632c1a5e63fa7 Gerrit-Change-Number: 10942 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Brown Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/10571 )
Change subject: IMPALA-7106: Log the original and rewritten SQL when SQL
rewrite fails
..
Patch Set 20: Code-Review+1
(6 comments)
The patch looks reasonable to me in the current form (pending nits).
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@a100
PS18, Line 100:
> There was an earlier discussion regarding this: https://gerrit.cloudera.org
That makes sense as it is much closer to the user written SQL than before.
http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java
File fe/src/main/java/org/apache/impala/analysis/UnionStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/main/java/org/apache/impala/analysis/UnionStmt.java@556
PS20, Line 556: if (t
nit: merge
http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java
File fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java@81
PS20, Line 81: if (!rewritten) {
: if (sqlString_ != null) return sqlString_;
: }
merge
http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java:
http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java@714
PS20, Line 714: WHERE a.int_col BETWEEN t.tinyint_col AND t.bigint_col)
This makes more sense
http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java:
http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@253
PS20, Line 253: values(1, '2', 3, 4.1)
I wasn't aware of this till now :-)
http://gerrit.cloudera.org:8080/#/c/10571/20/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@261
PS20, Line 261: stmt = (StatementBase) AnalyzesOk("select * from (" +
: "select * from functional.alltypes where id = (select 1
+ 1)) a", ctx);
: Assert.assertEquals(
: "SELECT * FROM (SELECT * FROM functional.alltypes WHERE
id = (SELECT 1 + 1)) a",
: stmt.toSql());
: Assert.assertEquals(
: "SELECT * FROM (SELECT * FROM functional.alltypes LEFT
SEMI JOIN " +
: "(SELECT 2) `$a$1` (`$c$1`) ON id = `$a$1`.`$c$1`)
can we factor these out into a method?
void foo(query, toSqlString, toSqlRewrittenString) {
AnalyzesOk(query)
AssertEquals(toSql(),toSqlString)
AssertEquals(toSql(true),toSqlRewrittenstring)
}
--
To view, visit http://gerrit.cloudera.org:8080/10571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53
Gerrit-Change-Number: 10571
Gerrit-PatchSet: 20
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 13 Jul 2018 20:39:59 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum)
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10944 ) Change subject: IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum) .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id09c6f59f677ba30145fb2081715f1a7d89fe20b Gerrit-Change-Number: 10944 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 20:34:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. IMPALA-7228: Add tpcds-unmodified to single-node-perf-run Description: tpcds-unmodified workload was added as a part of IMPALA-6819. This change allows tpcds-unmodified workload to be available for the single node perf run. Testing: Ran single node perf run using the following parameters and the test run was successful --iterations 2 --scale 2 --table_formats "parquet/none" \ --num_impalads 1 --workload "tpcds-unmodified" \ --load --query_names "TPCDS-Q17.*" --start_minicluster Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Reviewed-on: http://gerrit.cloudera.org:8080/10931 Reviewed-by: Impala Public Jenkins Reviewed-by: Jim Apple Tested-by: Impala Public Jenkins --- M bin/single_node_perf_run.py 1 file changed, 2 insertions(+), 1 deletion(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified Jim Apple: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 8 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. Patch Set 7: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 7 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Comment-Date: Fri, 13 Jul 2018 20:29:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum)
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10944 Change subject: IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum) .. IMPALA-7288: Fix Codegen Crash in FinalizeModule() (Addendum) In addition to previous fix for IMPALA-7288, this patch would prevent impala from crashing in case a code-path generates a malformed handcrafted function which it then tries to finalize. Ideally this would never happen since the code paths for generating handcrafted IRs would never generate a malformed function. Change-Id: Id09c6f59f677ba30145fb2081715f1a7d89fe20b --- M be/src/codegen/llvm-codegen.cc M be/src/codegen/llvm-codegen.h 2 files changed, 3 insertions(+), 6 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/44/10944/1 -- To view, visit http://gerrit.cloudera.org:8080/10944 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Id09c6f59f677ba30145fb2081715f1a7d89fe20b Gerrit-Change-Number: 10944 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10850 ) Change subject: IMPALA-6086: Use of permanent function should require SELECT privilege on DB .. Patch Set 6: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/10850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 Gerrit-Change-Number: 10850 Gerrit-PatchSet: 6 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 13 Jul 2018 19:39:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10850 ) Change subject: IMPALA-6086: Use of permanent function should require SELECT privilege on DB .. IMPALA-6086: Use of permanent function should require SELECT privilege on DB To use a permanent UDF should require at least SELECT privilege on the database. Functions that have constant arguments get constant-folded into string literals, losing their privilege requests in the process. This patch saves the privilege requests found during the first phase of query analysis, where all the objects and the privileges required to access them are identified. The requests are added back to the new analyzer created for re-analysis post expression rewrite. Testing: New FE test cases have been added to AuthorizationStmtTest. Manual tests were also done to identify the bug, as well as to test the fix. Ran exhaustive and covering tests. Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 Reviewed-on: http://gerrit.cloudera.org:8080/10850 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java 2 files changed, 50 insertions(+), 4 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 Gerrit-Change-Number: 10850 Gerrit-PatchSet: 7 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10571 )
Change subject: IMPALA-7106: Log the original and rewritten SQL when SQL
rewrite fails
..
Patch Set 20:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/10571/18//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10571/18//COMMIT_MSG@14
PS18, Line 14: ite fails.
:
> remove?
Done
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@453
PS18, Line 453: // The rewrites should have no user-visible effect.
Remember the original result
> How about moving the try-catch much closer to the analyze()?
Done
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@473
PS18, Line 473: throw e;
: }
:
> Do we still need this?
It think it's still quite useful wen TRACE is enabled.
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@479
PS18, Line 479:
> May be say "Error analyzing the rewritten query"? Rewrite is already succes
Done
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
File fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java@72
PS18, Line 72: if (!rewritten && sqlString_ != null) return sqlString_;
:
> merge?
Done
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java@76
PS18, Line 76: if (fromClause_.size() > 1 ||
targetTableRef_.hasExplicitAlias()) {
> If rewritten = true and sqlString_ == null, which means toSql(true) is call
By uninitialized states, did you mean targetTableRef, fromClause, and
wherePredicate? Those are supposedly initialized in the constructor. I believe
other statements follow similar patterns. Although I do believe we typically
call toSql() after the statement has been analyzed.
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@939
PS18, Line 939: // Return the SQL string before inline-view expression
substitution.
: if (!rewritten && sqlString_ != null) return sqlString_;
:
: S
> merge
Done
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StatementBase.java
File fe/src/main/java/org/apache/impala/analysis/StatementBase.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StatementBase.java@135
PS18, Line 135:
> If rewritten is true..returns the...if the stmt is rewritten..?
Done
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@a100
PS18, Line 100:
> Isn't the semantics that sqlString_ is populated in analyze(), if so, doesn
There was an earlier discussion regarding this:
https://gerrit.cloudera.org/c/10571/5/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java#117
and Tianyi suggested to remove this. Removing this fixes a different problem
in AnalyzeSubqueriesTest.java:
https://gerrit.cloudera.org/c/10571/18/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java#714
--
To view, visit http://gerrit.cloudera.org:8080/10571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53
Gerrit-Change-Number: 10571
Gerrit-PatchSet: 20
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 13 Jul 2018 19:09:21 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails
Fredy Wijaya has uploaded a new patch set (#20). ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails .. IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails toSql() method is used to print SQL string that is close to the original SQL string when errors arise or as the result of "SHOW CREATE". When debugging issues related to SQL rewrites, it can be very useful to be able to get the SQL string that is being rewritten. This patch adds a new method toSql(boolean rewritten) to get the rewritten SQL string. This patch also logs the original and rewritten SQL when SQL rewrite fails. Testing: - Added FE test for the rewritten SQL string - Ran all FE tests Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java 17 files changed, 206 insertions(+), 42 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10571/20 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 20 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7212: Deprecate --use krpc flag and remove old DataStream services
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/10835 ) Change subject: IMPALA-7212: Deprecate --use_krpc flag and remove old DataStream services .. Patch Set 3: (3 comments) http://gerrit.cloudera.org:8080/#/c/10835/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10835/3//COMMIT_MSG@7 PS3, Line 7: Deprecate Nit: I think it would be clearer to say "Remove" rather than "Deprecate". http://gerrit.cloudera.org:8080/#/c/10835/3//COMMIT_MSG@9 PS3, Line 9: deprecates Same here. http://gerrit.cloudera.org:8080/#/c/10835/3/be/src/common/global-flags.cc File be/src/common/global-flags.cc: http://gerrit.cloudera.org:8080/#/c/10835/3/be/src/common/global-flags.cc@260 PS3, Line 260: REMOVED_FLAG(disable_mem_pools); What are our rules about removing startup flags? Should use_krpc be here? (See IMPALA-3271 and IMPALA-5814) -- To view, visit http://gerrit.cloudera.org:8080/10835 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Icfed200751508478a3d728a917448f2dabfc67c3 Gerrit-Change-Number: 10835 Gerrit-PatchSet: 3 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 13 Jul 2018 18:48:25 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-3040: Remove cache directives if a partition is dropped externally
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/10792 ) Change subject: IMPALA-3040: Remove cache directives if a partition is dropped externally .. Patch Set 7: Did you get a chance to triage the GVO failure? Is it related to the patch change? -- To view, visit http://gerrit.cloudera.org:8080/10792 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id7701a499405e961456adea63f3592b43bd69170 Gerrit-Change-Number: 10792 Gerrit-PatchSet: 7 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Comment-Date: Fri, 13 Jul 2018 18:37:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails
Bharath Vissapragada has posted comments on this change. (
http://gerrit.cloudera.org:8080/10571 )
Change subject: IMPALA-7106: Log the original and rewritten SQL when SQL
rewrite fails
..
Patch Set 18:
(9 comments)
Oops I forgot to "Publish" the comments. I had them in drafts but then I saw
the latest comments and realized
http://gerrit.cloudera.org:8080/#/c/10571/18//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10571/18//COMMIT_MSG@14
PS18, Line 14: The LOG.trace
: statement that prints the rewritten SQL is also updated to use
toSql(true).
remove?
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@453
PS18, Line 453: try {
How about moving the try-catch much closer to the analyze()?
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@473
PS18, Line 473: if (LOG.isTraceEnabled()) {
: LOG.trace("rewrittenStmt: " +
analysisResult_.stmt_.toSql(true));
: }
Do we still need this?
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@479
PS18, Line 479: Error rewritting SQL statement
May be say "Error analyzing the rewritten query"? Rewrite is already successful
at this point.
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java
File fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java@72
PS18, Line 72: if (!rewritten) {
: if
merge?
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java@76
PS18, Line 76: StringBuilder b = new StringBuilder();
If rewritten = true and sqlString_ == null, which means toSql(true) is called
before ModifyStmt#analyze() runs, the following code could run into NPEs since
the toSql()s could be accessing uninitialized state. Should we handle that
case? It looks to me like this problem exists in the current version of toSql()
too (without this patch), so it is safe to ignore? Basically the precondition
for toSql() is that the Stmt should be analyzed. Is my understanding right?
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java
File fe/src/main/java/org/apache/impala/analysis/SelectStmt.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/SelectStmt.java@939
PS18, Line 939: if (!rewritten) {
: // Return the SQL string before inline-view expression
substitution.
: if (sqlString_ != null) return sqlString_;
: }
merge
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StatementBase.java
File fe/src/main/java/org/apache/impala/analysis/StatementBase.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StatementBase.java@135
PS18, Line 135:
If rewritten is true..returns the...if the stmt is rewritten..?
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java
File fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java@a100
PS18, Line 100:
Isn't the semantics that sqlString_ is populated in analyze(), if so, doesn't
it make sense to set this to null after rewrite and before reanalyze()?
--
To view, visit http://gerrit.cloudera.org:8080/10571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53
Gerrit-Change-Number: 10571
Gerrit-PatchSet: 18
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 13 Jul 2018 18:35:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails
Fredy Wijaya has uploaded a new patch set (#19). ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails .. IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails toSql() method is used to print SQL string that is close to the original SQL string when errors arise or as the result of "SHOW CREATE". When debugging issues related to SQL rewrites, it can be very useful to be able to get the SQL string that is being rewritten. This patch adds a new method toSql(boolean rewritten) to get the rewritten SQL string. This patch also logs the original and rewritten SQL when SQL rewrite fails. The LOG.trace statement that prints the rewritten SQL is also updated to use toSql(true). Testing: - Added FE test for the rewritten SQL string - Ran all FE tests Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/main/java/org/apache/impala/analysis/CreateTableAsSelectStmt.java M fe/src/main/java/org/apache/impala/analysis/DeleteStmt.java M fe/src/main/java/org/apache/impala/analysis/FromClause.java M fe/src/main/java/org/apache/impala/analysis/InlineViewRef.java M fe/src/main/java/org/apache/impala/analysis/InsertStmt.java M fe/src/main/java/org/apache/impala/analysis/ModifyStmt.java M fe/src/main/java/org/apache/impala/analysis/SelectStmt.java M fe/src/main/java/org/apache/impala/analysis/StatementBase.java M fe/src/main/java/org/apache/impala/analysis/StmtRewriter.java M fe/src/main/java/org/apache/impala/analysis/TableRef.java M fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java M fe/src/main/java/org/apache/impala/analysis/UnionStmt.java M fe/src/main/java/org/apache/impala/analysis/UpdateStmt.java M fe/src/main/java/org/apache/impala/analysis/WithClause.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java M fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java 17 files changed, 230 insertions(+), 62 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/71/10571/19 -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 19 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10571 ) Change subject: IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails .. Patch Set 19: (7 comments) http://gerrit.cloudera.org:8080/#/c/10571/18//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10571/18//COMMIT_MSG@10 PS18, Line 10: SQL string when errors arise or as the result of "SHOW CREATE". When > when errors arise or as the result of "show create" Done http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java: http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@474 PS18, Line 474: Rewritten SQL: > use the same label as L480: "Rewritten SQL: " Done http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/TableRef.java File fe/src/main/java/org/apache/impala/analysis/TableRef.java: http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/TableRef.java@575 PS18, Line 575: boolean rewritten > why's this not used? InlinViewRef is a subclass of TableRef and it's used by the overriden InlineViewRef::tableRefToSql(boolean rewritten) in InlineViewRef, which affects the TableRef::toSql(boolean rewritten) http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java: http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@191 PS18, Line 191: , : boolean rewritten > I think we should remove this and keep the original behavior. The intent is The original behavior is still the same when calling ctasStmt.toSql(), see the new ExpRewriterTest.java L300. http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java File fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java: http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java@714 PS18, Line 714: BETWEEN > why did this change? I thought the only externally visible change will be r Prior to this patch, the error message shows the rewritten SQL instead of the original one, which is incorrect. If you look at L711, the original SQL uses BETWEEN so the error message should technically return BETWEEN as well. http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java File fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java: http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@237 PS18, Line 237: > nit: remove these newlines following the banner Done http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@347 PS18, Line 347: : : : : > simpler: Assert.assertEquals(stmt.toSql(), stmt.toSql(true)) Done -- To view, visit http://gerrit.cloudera.org:8080/10571 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53 Gerrit-Change-Number: 10571 Gerrit-PatchSet: 19 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 13 Jul 2018 18:31:06 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. Patch Set 3: Code-Review+2 Carry Phil's + 2. -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 13 Jul 2018 18:04:03 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Fredy Wijaya has uploaded a new patch set (#3). ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code from IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code changes in this patch, there is no code change for the shims. The shims for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default implementation. Testing: - Ran core and exhaustive tests Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 --- M bin/create-test-configuration.sh M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh M fe/pom.xml D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/authorization/SentryAuthProvider.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/HdfsShim.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MetastoreShim.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MiniclusterProfile.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/util/SentryUtil.java D fe/src/compat-minicluster-profile-2/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java D fe/src/compat-minicluster-profile-3/java/org/apache/impala/compat/MiniclusterProfile.java R fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java R fe/src/main/java/org/apache/impala/authorization/ImpalaActionFactory.java R fe/src/main/java/org/apache/impala/authorization/ImpalaPrivilegeModel.java R fe/src/main/java/org/apache/impala/authorization/SentryAuthProvider.java R fe/src/main/java/org/apache/impala/compat/HdfsShim.java R fe/src/main/java/org/apache/impala/compat/MetastoreShim.java R fe/src/main/java/org/apache/impala/util/SentryUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java R fe/src/test/java/org/apache/impala/authorization/ImpalaActionFactoryTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java R fe/src/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java M impala-parent/pom.xml M testdata/bin/run-hbase.sh M testdata/bin/run-hive-server.sh M testdata/bin/run-mini-dfs.sh M testdata/cluster/node_templates/common/etc/init.d/common.tmpl M testdata/cluster/node_templates/common/etc/init.d/yarn-common M testdata/pom.xml M testdata/workloads/functional-query/queries/QueryTest/views-compatibility.test M tests/common/environ.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_partitioning.py 39 files changed, 237 insertions(+), 1,122 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/10940/3 -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 3 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. Patch Set 2: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/10940/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java: http://gerrit.cloudera.org:8080/#/c/10940/2/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@826 PS2, Line 826: // Historial note: Hadoop 2 accepts kerberos names missing a realm, but insists nit: historical -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 13 Jul 2018 18:01:58 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. Patch Set 2: Running GVO dry-run to make sure everything is fine. -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 13 Jul 2018 17:58:18 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2814/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 13 Jul 2018 17:57:36 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10940 )
Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
..
Patch Set 2:
(8 comments)
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/impala-config.sh@a727
PS1, Line 727:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
> I think this block should be kept. We presumably want to keep the Java != 1
Good catch. Done.
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh
File bin/jenkins/build-all-flag-combinations.sh:
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@a37
PS1, Line 37:
> This one is fine to remove, as "-so" is already covered on line39.
Done
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@a42
PS1, Line 42:
> Keep this one, but remove -profile2. This is the only coverage for the ubsa
Done
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@25
PS1, Line 25: # Usage: build-all-flag-combinations.sh [--dryrun]
> GVO will run this for you, but since you're changing it, we should run it a
Will test this with our dry-run GVO.
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@33
PS1, Line 33: CONFIGS=(
> Please update this comment. (Remove "special sigil...")
Done
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/pom.xml
File fe/pom.xml:
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/pom.xml@22
PS1, Line 22: http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
> In addition to the testing you've already done, could you run "mvn help:eff
Good idea. i ran a script to compare the before and after effective POMs and
they have the same dependencies.
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@826
PS1, Line 826: // Historial note: Hadoop 2 accepts kerberos names
missing a realm, but insists
: // on having a terminating '@' even when the default
realm is intended. Hadoop 3
: // now has more normal name conventions, where to
specify the default realm,
: // everything after and including the '@' character is
omitted.
: new User(USER.getName() + "/abc.host.com")
> This comment is a little bit odd without the context of the if in old line
Done
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java:
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@411
PS1, Line 411: // TODO: This logic can be removed.
: // Different versions of Hive have slightly different
error messages;
: // we normalize here as follows:
: // 'No FileSystem for Scheme "x"' -> 'No FileSystem for
scheme: x'
: if (errorString.contains("No FileSystem for scheme ")) {
: errorString = errorString.replace("\"", "");
: errorString = errorString.replace("No FileSystem for
scheme ",
:
> I think it's possible to remove this in a future change. No need to do it a
Will put a TODO.
--
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 2
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Fri, 13 Jul 2018 17:54:58 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Fredy Wijaya has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/10940 ) Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code from IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code changes in this patch, there is no code change for the shims. The shims for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default implementation. Testing: - Ran core and exhaustive tests Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 --- M bin/create-test-configuration.sh M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh M fe/pom.xml D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/authorization/SentryAuthProvider.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/HdfsShim.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MetastoreShim.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MiniclusterProfile.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/util/SentryUtil.java D fe/src/compat-minicluster-profile-2/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java D fe/src/compat-minicluster-profile-3/java/org/apache/impala/compat/MiniclusterProfile.java R fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java R fe/src/main/java/org/apache/impala/authorization/ImpalaActionFactory.java R fe/src/main/java/org/apache/impala/authorization/ImpalaPrivilegeModel.java R fe/src/main/java/org/apache/impala/authorization/SentryAuthProvider.java R fe/src/main/java/org/apache/impala/compat/HdfsShim.java R fe/src/main/java/org/apache/impala/compat/MetastoreShim.java R fe/src/main/java/org/apache/impala/util/SentryUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java R fe/src/test/java/org/apache/impala/authorization/ImpalaActionFactoryTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java R fe/src/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java M impala-parent/pom.xml M testdata/bin/run-hbase.sh M testdata/bin/run-hive-server.sh M testdata/bin/run-mini-dfs.sh M testdata/cluster/node_templates/common/etc/init.d/common.tmpl M testdata/cluster/node_templates/common/etc/init.d/yarn-common M testdata/pom.xml M testdata/workloads/functional-query/queries/QueryTest/views-compatibility.test M tests/common/environ.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_partitioning.py 39 files changed, 237 insertions(+), 1,122 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/10940/2 -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 2 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has uploaded a new patch set (#8). ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. IMPALA-7259: Improve Impala shell performance This patch fixes the slow performance in Impala shell, especially for large queries by replacing all calls to sqlparse.format(sql_string, strip_comments=True) with the custom implementation of strip comments that does not use grouping. The code to strip leading comments was also refactored to not use grouping. * Benchmark running a query with 12K columns * Before the patch: $ time impala-shell.sh -f large.sql --quiet real2m4.154s user2m0.536s sys 0m0.088s After the patch: $ time impala-shell.sh -f large.sql --quiet real0m3.885s user0m1.516s sys 0m0.048s Testing: - Added a new test to test the Impala shell performance - Ran all shell tests on Python 2.6 and Python 2.7 Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 --- M shell/impala_shell.py A tests/shell/test_file_large.sql M tests/shell/test_shell_commandline.py 3 files changed, 16,051 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10939/8 -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 8 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7201. Support DDL with LocalCatalog enabled
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10806 ) Change subject: IMPALA-7201. Support DDL with LocalCatalog enabled .. IMPALA-7201. Support DDL with LocalCatalog enabled This fixes a couple issues with DDL commands when LocalCatalog is enabled: - updateCatalogCache() gets called after any DDL. Instead of throwing an exception, we can just no-op this by returning some fake result. - In order to support 'drop database' we need to properly implement the various function-related calls such that they don't throw exceptions. This changes them to be stubbed out as having no functions. - Fixes for 'alter view' and 'drop view' so that the underlying target table gets loaded by the catalogd before attempting the operation. Without this, in the LocalCatalog case, the catalogd would only have an IncompleteTable and these operations would fail with "unexpected table type" errors. With this patch I was able to run 'run-tests.py -k views' and 3/4 passed. The one that failed depends on HBase tables, not yet implemented. Change-Id: Ic39c97a5f5ad145e03b96d1a470dc2dfa6ec71a5 Reviewed-on: http://gerrit.cloudera.org:8080/10806 Reviewed-by: Todd Lipcon Tested-by: Todd Lipcon --- M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java 3 files changed, 38 insertions(+), 10 deletions(-) Approvals: Todd Lipcon: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/10806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ic39c97a5f5ad145e03b96d1a470dc2dfa6ec71a5 Gerrit-Change-Number: 10806 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7140 (part 7): small fixes to enable most queries on HDFS tables
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10798 ) Change subject: IMPALA-7140 (part 7): small fixes to enable most queries on HDFS tables .. IMPALA-7140 (part 7): small fixes to enable most queries on HDFS tables This is a grab-bag of small fixes necessary to get most queries on HDFS tables passing with the correct plans: * Change the loading of tables to check for other table types before checking for FS tables, since without specific checks against various properties, other tables may look like FS tables and try to instantiate LocalFsTable incorrectly. * Return -1 for extrapolated row count -- the previous 0 value was convincing the planner that it had a valid value. * Fix up the handling of BuiltinsDb so that we don't depend on ImpaladCatalog to have been loaded in order to instantiate it. * Properly handle the case where all partitions are pruned by a predicate. With this change, about half of the tests in PlannerTest pass. The tests that don't pass all rely on views, HBase tables, etc. Change-Id: I6f603e62b7a013c148c0905ebdec2f4303f9c4e5 Reviewed-on: http://gerrit.cloudera.org:8080/10798 Tested-by: Impala Public Jenkins Reviewed-by: Todd Lipcon --- M fe/src/main/java/org/apache/impala/analysis/CastExpr.java M fe/src/main/java/org/apache/impala/analysis/Expr.java M fe/src/main/java/org/apache/impala/analysis/ExtractFromExpr.java M fe/src/main/java/org/apache/impala/analysis/FunctionCallExpr.java M fe/src/main/java/org/apache/impala/analysis/FunctionName.java M fe/src/main/java/org/apache/impala/catalog/BuiltinsDb.java M fe/src/main/java/org/apache/impala/catalog/ImpaladCatalog.java M fe/src/main/java/org/apache/impala/catalog/ScalarFunction.java M fe/src/main/java/org/apache/impala/catalog/local/LocalCatalog.java M fe/src/main/java/org/apache/impala/catalog/local/LocalFsTable.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java M fe/src/main/jflex/sql-scanner.flex M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 14 files changed, 65 insertions(+), 42 deletions(-) Approvals: Impala Public Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/10798 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I6f603e62b7a013c148c0905ebdec2f4303f9c4e5 Gerrit-Change-Number: 10798 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7201. Support DDL with LocalCatalog enabled
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10806 ) Change subject: IMPALA-7201. Support DDL with LocalCatalog enabled .. Patch Set 5: Verified+1 Code-Review+2 Forwarding +2 and +1 after simple comment-only fix -- To view, visit http://gerrit.cloudera.org:8080/10806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic39c97a5f5ad145e03b96d1a470dc2dfa6ec71a5 Gerrit-Change-Number: 10806 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 13 Jul 2018 17:39:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7140 (part 8): support views in LocalCatalog
Todd Lipcon has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10805 ) Change subject: IMPALA-7140 (part 8): support views in LocalCatalog .. IMPALA-7140 (part 8): support views in LocalCatalog This adds basic support for loading views in LocalCatalog. Tested with a small unit test and also verified from the shell that I can select from a view. Change-Id: Ib3516b9ceff6dce12ded68d93afde09728627e08 Reviewed-on: http://gerrit.cloudera.org:8080/10805 Tested-by: Impala Public Jenkins Reviewed-by: Todd Lipcon --- M fe/src/main/java/org/apache/impala/catalog/View.java M fe/src/main/java/org/apache/impala/catalog/local/LocalTable.java A fe/src/main/java/org/apache/impala/catalog/local/LocalView.java M fe/src/test/java/org/apache/impala/catalog/local/LocalCatalogTest.java 4 files changed, 116 insertions(+), 33 deletions(-) Approvals: Impala Public Jenkins: Verified Todd Lipcon: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/10805 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib3516b9ceff6dce12ded68d93afde09728627e08 Gerrit-Change-Number: 10805 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7201. Support DDL with LocalCatalog enabled
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/10806 ) Change subject: IMPALA-7201. Support DDL with LocalCatalog enabled .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/10806/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/10806/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1395 PS4, Line 1395: // be loaded because the planning phase on the impalad side triggered the loading. > nit: be Done -- To view, visit http://gerrit.cloudera.org:8080/10806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic39c97a5f5ad145e03b96d1a470dc2dfa6ec71a5 Gerrit-Change-Number: 10806 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 13 Jul 2018 17:38:55 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7201. Support DDL with LocalCatalog enabled
Hello Tianyi Wang, Vuk Ercegovac, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10806 to look at the new patch set (#5). Change subject: IMPALA-7201. Support DDL with LocalCatalog enabled .. IMPALA-7201. Support DDL with LocalCatalog enabled This fixes a couple issues with DDL commands when LocalCatalog is enabled: - updateCatalogCache() gets called after any DDL. Instead of throwing an exception, we can just no-op this by returning some fake result. - In order to support 'drop database' we need to properly implement the various function-related calls such that they don't throw exceptions. This changes them to be stubbed out as having no functions. - Fixes for 'alter view' and 'drop view' so that the underlying target table gets loaded by the catalogd before attempting the operation. Without this, in the LocalCatalog case, the catalogd would only have an IncompleteTable and these operations would fail with "unexpected table type" errors. With this patch I was able to run 'run-tests.py -k views' and 3/4 passed. The one that failed depends on HBase tables, not yet implemented. Change-Id: Ic39c97a5f5ad145e03b96d1a470dc2dfa6ec71a5 --- M fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java M fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java M fe/src/main/java/org/apache/impala/service/FeCatalogManager.java 3 files changed, 38 insertions(+), 10 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/10806/5 -- To view, visit http://gerrit.cloudera.org:8080/10806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic39c97a5f5ad145e03b96d1a470dc2dfa6ec71a5 Gerrit-Change-Number: 10806 Gerrit-PatchSet: 5 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR] IMPALA-7201. Support DDL with LocalCatalog enabled
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10806 ) Change subject: IMPALA-7201. Support DDL with LocalCatalog enabled .. Patch Set 4: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/10806/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/10806/4/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@1395 PS4, Line 1395: // loaded because the planning phase on the impalad side triggered the loading. nit: be -- To view, visit http://gerrit.cloudera.org:8080/10806 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic39c97a5f5ad145e03b96d1a470dc2dfa6ec71a5 Gerrit-Change-Number: 10806 Gerrit-PatchSet: 4 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 13 Jul 2018 17:30:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6810: runtime row filters.test: omit pool name in pattern
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10942 ) Change subject: IMPALA-6810: runtime_row_filters.test: omit pool name in pattern .. Patch Set 2: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2813/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fe6beb169dc6bfefabde9dc7a4632c1a5e63fa7 Gerrit-Change-Number: 10942 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Brown Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 17:27:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6810: runtime row filters.test: omit pool name in pattern
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/10942 ) Change subject: IMPALA-6810: runtime_row_filters.test: omit pool name in pattern .. Patch Set 2: Code-Review+2 rebase -- To view, visit http://gerrit.cloudera.org:8080/10942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fe6beb169dc6bfefabde9dc7a4632c1a5e63fa7 Gerrit-Change-Number: 10942 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Brown Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 17:27:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6810: runtime row filters.test: omit pool name in pattern
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10942 ) Change subject: IMPALA-6810: runtime_row_filters.test: omit pool name in pattern .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3fe6beb169dc6bfefabde9dc7a4632c1a5e63fa7 Gerrit-Change-Number: 10942 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Brown Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 17:19:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. Patch Set 7: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2812/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 7 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Comment-Date: Fri, 13 Jul 2018 17:17:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. Patch Set 7: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 7 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Comment-Date: Fri, 13 Jul 2018 17:17:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Nithya Janarthanan has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. IMPALA-7228: Add tpcds-unmodified to single-node-perf-run Description: tpcds-unmodified workload was added as a part of IMPALA-6819. This change allows tpcds-unmodified workload to be available for the single node perf run. Testing: Ran single node perf run using the following parameters and the test run was successful --iterations 2 --scale 2 --table_formats "parquet/none" \ --num_impalads 1 --workload "tpcds-unmodified" \ --load --query_names "TPCDS-Q17.*" --start_minicluster Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 --- M bin/single_node_perf_run.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10931/7 -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 7 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2811/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 6 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Comment-Date: Fri, 13 Jul 2018 17:12:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 6 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Comment-Date: Fri, 13 Jul 2018 17:12:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 5 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Comment-Date: Fri, 13 Jul 2018 17:11:40 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1624: Allow toggling and unsetting some command-line options inside impala-shell
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Jinchul Kim, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10900 to look at the new patch set (#4). Change subject: IMPALA-1624: Allow toggling and unsetting some command-line options inside impala-shell .. IMPALA-1624: Allow toggling and unsetting some command-line options inside impala-shell This change provides a way to modify command-line options like -B, --output_file and --delimiter inside impala-shell without quitting the shell and then restarting again. Also fixed IMPALA-7286: comment "unset" does not work for shell options Testing: Added tests for all new options in test_shell_interactive.py Change-Id: Id8d4487c24f24806223bfd5c54336914e3afd763 --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 72 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/10900/4 -- To view, visit http://gerrit.cloudera.org:8080/10900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id8d4487c24f24806223bfd5c54336914e3afd763 Gerrit-Change-Number: 10900 Gerrit-PatchSet: 4 Gerrit-Owner: Le Minh Nghia Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Le Minh Nghia Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Nithya Janarthanan has posted comments on this change. ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/10931/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10931/4//COMMIT_MSG@10 PS4, Line 10: tpcds-unmodified workload was added as a part of IMPALA-6819 > nits: period at end of both sentences. Done -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 4 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Nithya Janarthanan Gerrit-Comment-Date: Fri, 13 Jul 2018 17:08:42 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7228: Add tpcds-unmodified to single-node-perf-run
Nithya Janarthanan has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10931 ) Change subject: IMPALA-7228: Add tpcds-unmodified to single-node-perf-run .. IMPALA-7228: Add tpcds-unmodified to single-node-perf-run Description: tpcds-unmodified workload was added as a part of IMPALA-6819. This change allows tpcds-unmodified workload to be available for the single node perf run. Testing: Ran single node perf run using the following parameters and the test run was successful --iterations 2 --scale 2 --table_formats "parquet/none" \ --num_impalads 1 --workload "tpcds-unmodified" \ --load --query_names "TPCDS-Q17.*" --start_minicluster Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 --- M bin/single_node_perf_run.py 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/31/10931/5 -- To view, visit http://gerrit.cloudera.org:8080/10931 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I511661c586cd55e3240ccbea9c499b9c3fc98440 Gerrit-Change-Number: 10931 Gerrit-PatchSet: 5 Gerrit-Owner: Nithya Janarthanan Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Michael Brown
[Impala-ASF-CR] IMPALA-6810: runtime row filters.test: omit pool name in pattern
Michael Brown has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10942 Change subject: IMPALA-6810: runtime_row_filters.test: omit pool name in pattern .. IMPALA-6810: runtime_row_filters.test: omit pool name in pattern Some downstream tests run this with a fair-scheduler.xml set that, while not changing admission control behavior, does change the name of the pool. Omit the pool name to permit that downstream test to succeed. Testing: - local with change in minicluster - downstream in environment as well Change-Id: I3fe6beb169dc6bfefabde9dc7a4632c1a5e63fa7 --- M testdata/workloads/functional-query/queries/QueryTest/runtime_row_filters.test 1 file changed, 3 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/42/10942/1 -- To view, visit http://gerrit.cloudera.org:8080/10942 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3fe6beb169dc6bfefabde9dc7a4632c1a5e63fa7 Gerrit-Change-Number: 10942 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Brown
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Michael Brown has posted comments on this change. ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. Patch Set 7: Code-Review+1 (1 comment) for the test http://gerrit.cloudera.org:8080/#/c/10939/7/tests/shell/test_shell_commandline.py File tests/shell/test_shell_commandline.py: http://gerrit.cloudera.org:8080/#/c/10939/7/tests/shell/test_shell_commandline.py@31 PS7, Line 31: from time import sleep : from time import time Sorry, you can do: from time import sleep, time -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 13 Jul 2018 17:03:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7106: Log the original and rewritten SQL when SQL rewrite fails
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/10571 )
Change subject: IMPALA-7106: Log the original and rewritten SQL when SQL
rewrite fails
..
Patch Set 18:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/10571/18//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10571/18//COMMIT_MSG@10
PS18, Line 10: SQL string, which is something that users want to see. When
debugging
when errors arise or as the result of "show create"
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java
File fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java@474
PS18, Line 474: rewrittenStmt:
use the same label as L480: "Rewritten SQL: "
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/TableRef.java
File fe/src/main/java/org/apache/impala/analysis/TableRef.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/TableRef.java@575
PS18, Line 575: boolean rewritten
why's this not used?
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java
File fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/main/java/org/apache/impala/analysis/ToSqlUtils.java@191
PS18, Line 191: ,
: boolean rewritten
I think we should remove this and keep the original behavior. The intent is to
show the original sql, not the rewritten sql.
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java
File fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/AnalyzeSubqueriesTest.java@714
PS18, Line 714: BETWEEN
why did this change? I thought the only externally visible change will be
reflected in the logs.
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java
File fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java:
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@237
PS18, Line 237:
nit: remove these newlines following the banner
http://gerrit.cloudera.org:8080/#/c/10571/18/fe/src/test/java/org/apache/impala/analysis/ExprRewriterTest.java@347
PS18, Line 347: Assert.assertEquals("WITH t AS (SELECT 1 + 1) " +
: "SELECT id FROM functional.alltypes UNION SELECT id
FROM functional.alltypesagg",
: stmt.toSql());
: Assert.assertEquals("WITH t AS (SELECT 1 + 1) " +
: "
simpler: Assert.assertEquals(stmt.toSql(), stmt.toSql(true))
--
To view, visit http://gerrit.cloudera.org:8080/10571
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iab58b0cc865135d261dd4a7f72be130f2e7bde53
Gerrit-Change-Number: 10571
Gerrit-PatchSet: 18
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 13 Jul 2018 16:56:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10850 ) Change subject: IMPALA-6086: Use of permanent function should require SELECT privilege on DB .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2810/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 Gerrit-Change-Number: 10850 Gerrit-PatchSet: 6 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 13 Jul 2018 16:17:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10850 ) Change subject: IMPALA-6086: Use of permanent function should require SELECT privilege on DB .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 Gerrit-Change-Number: 10850 Gerrit-PatchSet: 6 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 13 Jul 2018 16:17:56 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10850 ) Change subject: IMPALA-6086: Use of permanent function should require SELECT privilege on DB .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 Gerrit-Change-Number: 10850 Gerrit-PatchSet: 5 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 13 Jul 2018 16:17:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Philip Zeyliger has posted comments on this change. (
http://gerrit.cloudera.org:8080/10940 )
Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2
..
Patch Set 1:
(8 comments)
Thanks. This looked pretty straight-forward to me. I've only got minor comments
below.
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/impala-config.sh
File bin/impala-config.sh:
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/impala-config.sh@a727
PS1, Line 727:
:
:
:
:
:
:
:
:
:
:
:
:
:
:
I think this block should be kept. We presumably want to keep the Java != 1.7
assertion here.
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh
File bin/jenkins/build-all-flag-combinations.sh:
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@a37
PS1, Line 37:
This one is fine to remove, as "-so" is already covered on line39.
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@a42
PS1, Line 42:
Keep this one, but remove -profile2. This is the only coverage for the ubsan
build.
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@25
PS1, Line 25: # Usage: build-all-flag-combinations.sh [--dryrun]
GVO will run this for you, but since you're changing it, we should run it as
part of our testing (or look explicitly at GVO).
http://gerrit.cloudera.org:8080/#/c/10940/1/bin/jenkins/build-all-flag-combinations.sh@33
PS1, Line 33: # "minicluster profile" where appropriate.
Please update this comment. (Remove "special sigil...")
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/pom.xml
File fe/pom.xml:
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/pom.xml@22
PS1, Line 22: http://maven.apache.org/POM/4.0.0;
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;>
In addition to the testing you've already done, could you run "mvn
help:effective-pom"
(https://maven.apache.org/plugins/maven-help-plugin/effective-pom-mojo.html)
with the relevant profile and compare before and after? Equivalently, "mvn
dependency:tree"?
It would have been easy to drop something accidentally here.
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java:
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java@826
PS1, Line 826: // Hadoop 2 accepts kerberos names missing a realm, but
insists
: // on having a terminating '@' even when the default
realm
: // is intended. Hadoop 3 now has more normal name
convetions,
: // where to specify the default realm, everything after
and
: // including the '@' character is omitted.
This comment is a little bit odd without the context of the if in old line 833.
Perhaps add "Historical note: " to the comment or something...
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java
File fe/src/test/java/org/apache/impala/common/FrontendTestBase.java:
http://gerrit.cloudera.org:8080/#/c/10940/1/fe/src/test/java/org/apache/impala/common/FrontendTestBase.java@411
PS1, Line 411: // Different versions of Hive have slightly different
error messages;
: // we normalize here as follows:
: // 'No FileSystem for Scheme "x"' -> 'No FileSystem for
scheme: x'
: if (errorString.contains("No FileSystem for scheme ")) {
: errorString = errorString.replace("\"", "");
: errorString = errorString.replace("No FileSystem for
scheme ",
: "No FileSystem for scheme: ");
: }
I think it's possible to remove this in a future change. No need to do it as
part of this change.
--
To view, visit http://gerrit.cloudera.org:8080/10940
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08
Gerrit-Change-Number: 10940
Gerrit-PatchSet: 1
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Philip Zeyliger
Gerrit-Comment-Date: Fri, 13 Jul 2018 16:16:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Zoram Thanga has posted comments on this change. (
http://gerrit.cloudera.org:8080/10850 )
Change subject: IMPALA-6086: Use of permanent function should require SELECT
privilege on DB
..
Patch Set 5: Code-Review+1
(1 comment)
Fixed the indentation. Thanks for the review.
http://gerrit.cloudera.org:8080/#/c/10850/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java
File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java:
http://gerrit.cloudera.org:8080/#/c/10850/4/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@2183
PS4, Line 2183: Type retType, String uriPath, String symbolName) {
> nit: incorrect indentation
Done
--
To view, visit http://gerrit.cloudera.org:8080/10850
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57
Gerrit-Change-Number: 10850
Gerrit-PatchSet: 5
Gerrit-Owner: Zoram Thanga
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Fri, 13 Jul 2018 16:15:47 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6086: Use of permanent function should require SELECT privilege on DB
Hello Fredy Wijaya, Vuk Ercegovac, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10850 to look at the new patch set (#5). Change subject: IMPALA-6086: Use of permanent function should require SELECT privilege on DB .. IMPALA-6086: Use of permanent function should require SELECT privilege on DB To use a permanent UDF should require at least SELECT privilege on the database. Functions that have constant arguments get constant-folded into string literals, losing their privilege requests in the process. This patch saves the privilege requests found during the first phase of query analysis, where all the objects and the privileges required to access them are identified. The requests are added back to the new analyzer created for re-analysis post expression rewrite. Testing: New FE test cases have been added to AuthorizationStmtTest. Manual tests were also done to identify the bug, as well as to test the fix. Ran exhaustive and covering tests. Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 --- M fe/src/main/java/org/apache/impala/analysis/AnalysisContext.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java 2 files changed, 50 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/50/10850/5 -- To view, visit http://gerrit.cloudera.org:8080/10850 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iee70f15e4c04f7daaed9cac2400ec626e1fb0e57 Gerrit-Change-Number: 10850 Gerrit-PatchSet: 5 Gerrit-Owner: Zoram Thanga Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has posted comments on this change. (
http://gerrit.cloudera.org:8080/10939 )
Change subject: IMPALA-7259: Improve Impala shell performance
..
Patch Set 7:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/10939/6/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/10939/6/tests/shell/test_shell_commandline.py@664
PS6, Line 664: f test_large_sql(self
> It's preferred style to keep imports at the top of modules. You can add tim
Done
http://gerrit.cloudera.org:8080/#/c/10939/6/tests/shell/test_shell_commandline.py@672
PS6, Line 672: assert actual_time_s <= time_limit_s, (
: "It took {0} seconds to execute the query. Time limit is
{1} seconds.".format(
: actual_time_s, time_limit_s))
> assert actual_time_s <= time_limit_s, (
Ooops, I must've been too sleepy last night :) Done.
--
To view, visit http://gerrit.cloudera.org:8080/10939
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81
Gerrit-Change-Number: 10939
Gerrit-PatchSet: 7
Gerrit-Owner: Fredy Wijaya
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Michael Brown
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 13 Jul 2018 16:14:42 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has uploaded a new patch set (#7). ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. IMPALA-7259: Improve Impala shell performance This patch fixes the slow performance in Impala shell, especially for large queries by replacing all calls to sqlparse.format(sql_string, strip_comments=True) with the custom implementation of strip comments that does not use grouping. The code to strip leading comments was also refactored to not use grouping. * Benchmark running a query with 12K columns * Before the patch: $ time impala-shell.sh -f large.sql --quiet real2m4.154s user2m0.536s sys 0m0.088s After the patch: $ time impala-shell.sh -f large.sql --quiet real0m3.885s user0m1.516s sys 0m0.048s Testing: - Added a new test to test the Impala shell performance - Ran all shell tests on Python 2.6 and Python 2.7 Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 --- M shell/impala_shell.py A tests/shell/test_file_large.sql M tests/shell/test_shell_commandline.py 3 files changed, 16,051 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10939/7 -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 7 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Michael Brown Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-1624: Allow toggling and unsetting some command-line options inside impala-shell
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/10900 ) Change subject: IMPALA-1624: Allow toggling and unsetting some command-line options inside impala-shell .. Patch Set 3: (2 comments) http://gerrit.cloudera.org:8080/#/c/10900/3//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/10900/3//COMMIT_MSG@12 PS3, Line 12: Also fixed IMPALA-7286 Please be a bit more talkative about this :) at least add the title of the Jira like "IMPALA-7286 Command "unset" does not work for shell options" http://gerrit.cloudera.org:8080/#/c/10900/3/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/10900/3/shell/impala_shell.py@657 PS3, Line 657: print 'Unsetting shell option %s' % token Please move this to the calling function to be consistent with the other printed messages. -- To view, visit http://gerrit.cloudera.org:8080/10900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id8d4487c24f24806223bfd5c54336914e3afd763 Gerrit-Change-Number: 10900 Gerrit-PatchSet: 3 Gerrit-Owner: Le Minh Nghia Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Le Minh Nghia Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 13 Jul 2018 16:08:05 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1624: Allow toggling and unsetting some command-line options inside impala-shell
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Jinchul Kim, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10900 to look at the new patch set (#3). Change subject: IMPALA-1624: Allow toggling and unsetting some command-line options inside impala-shell .. IMPALA-1624: Allow toggling and unsetting some command-line options inside impala-shell This change provides a way to modify command-line options like -B, --output_file and --delimiter inside impala-shell without quitting the shell and then restarting again. Also fixed IMPALA-7286 Testing: Added tests for all new options in test_shell_interactive.py Change-Id: Id8d4487c24f24806223bfd5c54336914e3afd763 --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 72 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/10900/3 -- To view, visit http://gerrit.cloudera.org:8080/10900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id8d4487c24f24806223bfd5c54336914e3afd763 Gerrit-Change-Number: 10900 Gerrit-PatchSet: 3 Gerrit-Owner: Le Minh Nghia Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Le Minh Nghia Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Hello Lars Volker, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10893 to look at the new patch set (#3). Change subject: IMPALA-6677: [DOCS] Document the next_day function .. IMPALA-6677: [DOCS] Document the next_day function Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 --- M docs/topics/impala_datetime_functions.xml 1 file changed, 61 insertions(+), 15 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/10893/3 -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] IMPALA-6677: [DOCS] Document the next day function
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10893 ) Change subject: IMPALA-6677: [DOCS] Document the next_day function .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/10893/2/docs/topics/impala_datetime_functions.xml File docs/topics/impala_datetime_functions.xml: http://gerrit.cloudera.org:8080/#/c/10893/2/docs/topics/impala_datetime_functions.xml@1941 PS2, Line 1941: Saturday December 25 > "first Saturday after December..."? Done -- To view, visit http://gerrit.cloudera.org:8080/10893 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I2dacc86ff69a1016b1863d9db66dd29fd832b715 Gerrit-Change-Number: 10893 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Fri, 13 Jul 2018 15:55:37 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1624: Allow toggling some command-line options inside impala-shell
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/10900 )
Change subject: IMPALA-1624: Allow toggling some command-line options inside
impala-shell
..
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/10900/2/shell/impala_shell.py
File shell/impala_shell.py:
http://gerrit.cloudera.org:8080/#/c/10900/2/shell/impala_shell.py@131
PS2, Line 131: # Valid true value for some shell options
nit: we typically finish comments with '.' in Impala
I would also rephrase the sentence to something like "Strings that are
interpreted as True for some boolean options."
http://gerrit.cloudera.org:8080/#/c/10900/2/shell/impala_shell.py@177
PS2, Line 177: \
nit: please add a space before the last \ - I think that we do this everywhere
in Impala.
http://gerrit.cloudera.org:8080/#/c/10900/2/shell/impala_shell.py@743
PS2, Line 743: elif not self._handle_unset_shell_options(option):
Please print a message when the unset succeeds.
http://gerrit.cloudera.org:8080/#/c/10900/1/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/10900/1/tests/shell/test_shell_interactive.py@125
PS1, Line 125: p2 = ImpalaShell()
> Done
Thanks for doing IMPALA-7286! Please also mention this change in the commit
message.
http://gerrit.cloudera.org:8080/#/c/10900/2/tests/shell/test_shell_interactive.py
File tests/shell/test_shell_interactive.py:
http://gerrit.cloudera.org:8080/#/c/10900/2/tests/shell/test_shell_interactive.py@94
PS2, Line 94: p.send_cmd("set delimiter=,")
Can we keep the default delimiter in this test to cover more scenarios by the
tests?
--
To view, visit http://gerrit.cloudera.org:8080/10900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id8d4487c24f24806223bfd5c54336914e3afd763
Gerrit-Change-Number: 10900
Gerrit-PatchSet: 2
Gerrit-Owner: Le Minh Nghia
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Jinchul Kim
Gerrit-Reviewer: Le Minh Nghia
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 13 Jul 2018 15:46:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1624: Allow toggling some command-line options inside impala-shell
Le Minh Nghia has posted comments on this change. ( http://gerrit.cloudera.org:8080/10900 ) Change subject: IMPALA-1624: Allow toggling some command-line options inside impala-shell .. Patch Set 2: (7 comments) http://gerrit.cloudera.org:8080/#/c/10900/1/shell/impala_shell.py File shell/impala_shell.py: http://gerrit.cloudera.org:8080/#/c/10900/1/shell/impala_shell.py@135 PS1, Line 135: alaShell.TRUE_STRINGS, "pri > This could be moved to a separate variable like TRUE_STRINGS or the whole l Done http://gerrit.cloudera.org:8080/#/c/10900/1/shell/impala_shell.py@137 PS1, Line 137: pal > Will this handle other special characters like tabs and new lines? Yes, it will. http://gerrit.cloudera.org:8080/#/c/10900/1/shell/impala_shell.py@138 PS1, Line 138: RUE_ > I would prefer an empty string instead on 'None' to be consistent with the Done http://gerrit.cloudera.org:8080/#/c/10900/1/shell/impala_shell.py@1435 PS1, Line 1435: > I have noticed that this tip is wrong. Please correct the default to \t. It Done http://gerrit.cloudera.org:8080/#/c/10900/1/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/10900/1/tests/shell/test_shell_interactive.py@98 PS1, Line 98: assert "21,VIETNAM,2" in result.stdout > Please also add a positive test (a row that should be included in the resul Done http://gerrit.cloudera.org:8080/#/c/10900/1/tests/shell/test_shell_interactive.py@119 PS1, Line 119: assert "VIETNAM" not in result.stdout > It should be also verified that the results were written to the file. Done http://gerrit.cloudera.org:8080/#/c/10900/1/tests/shell/test_shell_interactive.py@125 PS1, Line 125: p2 = ImpalaShell() > The primary way to set back options to their default should be calling "uns Done -- To view, visit http://gerrit.cloudera.org:8080/10900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id8d4487c24f24806223bfd5c54336914e3afd763 Gerrit-Change-Number: 10900 Gerrit-PatchSet: 2 Gerrit-Owner: Le Minh Nghia Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Le Minh Nghia Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 13 Jul 2018 15:20:18 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1624: Allow toggling some command-line options inside impala-shell
Hello Zoltan Borok-Nagy, Csaba Ringhofer, Jinchul Kim, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10900 to look at the new patch set (#2). Change subject: IMPALA-1624: Allow toggling some command-line options inside impala-shell .. IMPALA-1624: Allow toggling some command-line options inside impala-shell This change provides a way to modify command-line options like -B, --output_file and --delimiter inside impala-shell without quitting the shell and then restarting again. Testing: Added tests for all new options in test_shell_interactive.py Change-Id: Id8d4487c24f24806223bfd5c54336914e3afd763 --- M shell/impala_shell.py M tests/shell/test_shell_interactive.py 2 files changed, 72 insertions(+), 5 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/10900/2 -- To view, visit http://gerrit.cloudera.org:8080/10900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id8d4487c24f24806223bfd5c54336914e3afd763 Gerrit-Change-Number: 10900 Gerrit-PatchSet: 2 Gerrit-Owner: Le Minh Nghia Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Jinchul Kim Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has uploaded a new patch set (#6). ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. IMPALA-7259: Improve Impala shell performance This patch fixes the slow performance in Impala shell, especially for large queries by replacing all calls to sqlparse.format(sql_string, strip_comments=True) with the custom implementation of strip comments that does not use grouping. The code to strip leading comments was also refactored to not use grouping. * Benchmark running a query with 12K columns * Before the patch: $ time impala-shell.sh -f large.sql --quiet real2m4.154s user2m0.536s sys 0m0.088s After the patch: $ time impala-shell.sh -f large.sql --quiet real0m3.885s user0m1.516s sys 0m0.048s Testing: - Added a new test to test the Impala shell performance - Ran all shell tests on Python 2.6 and Python 2.7 Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 --- M shell/impala_shell.py A tests/shell/test_file_large.sql M tests/shell/test_shell_commandline.py 3 files changed, 16,051 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10939/6 -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7295: Remove IMPALA MINICLUSTER PROFILE=2
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10940 Change subject: IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 .. IMPALA-7295: Remove IMPALA_MINICLUSTER_PROFILE=2 This patch removes the use of IMPALA_MINICLUSTER_PROFILE. The code that uses IMPALA_MINICLUSTER_PROFILE=2 is removed and it defaults to code from IMPALA_MINICLUSTER_PROFILE=3. In order to reduce having too many code changes in this patch, there is no code change for the shims. The shims for IMPALA_MINICLUSTER_PROFILE=3 automatically become the default implementation. Testing: - Ran core and exhaustive tests Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 --- M bin/create-test-configuration.sh M bin/impala-config.sh M bin/jenkins/build-all-flag-combinations.sh M fe/pom.xml D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetCatalogsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetColumnsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetFunctionsReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetInfoReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetSchemasReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/hive/service/rpc/thrift/TGetTablesReq.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/authorization/SentryAuthProvider.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/HdfsShim.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MetastoreShim.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/compat/MiniclusterProfile.java D fe/src/compat-minicluster-profile-2/java/org/apache/impala/util/SentryUtil.java D fe/src/compat-minicluster-profile-2/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java D fe/src/compat-minicluster-profile-3/java/org/apache/impala/compat/MiniclusterProfile.java R fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java R fe/src/main/java/org/apache/impala/authorization/ImpalaActionFactory.java R fe/src/main/java/org/apache/impala/authorization/ImpalaPrivilegeModel.java R fe/src/main/java/org/apache/impala/authorization/SentryAuthProvider.java R fe/src/main/java/org/apache/impala/compat/HdfsShim.java R fe/src/main/java/org/apache/impala/compat/MetastoreShim.java R fe/src/main/java/org/apache/impala/util/SentryUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationTest.java R fe/src/test/java/org/apache/impala/authorization/ImpalaActionFactoryTest.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java R fe/src/test/java/org/apache/impala/datagenerator/HBaseTestDataRegionAssignment.java M impala-parent/pom.xml M testdata/bin/run-hbase.sh M testdata/bin/run-hive-server.sh M testdata/bin/run-mini-dfs.sh M testdata/cluster/node_templates/common/etc/init.d/common.tmpl M testdata/cluster/node_templates/common/etc/init.d/yarn-common M testdata/pom.xml M testdata/workloads/functional-query/queries/QueryTest/views-compatibility.test M tests/common/environ.py M tests/metadata/test_views_compatibility.py M tests/query_test/test_partitioning.py 39 files changed, 224 insertions(+), 1,123 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/40/10940/1 -- To view, visit http://gerrit.cloudera.org:8080/10940 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iba4a81165b3d2012dc04d4115454372c41e39f08 Gerrit-Change-Number: 10940 Gerrit-PatchSet: 1 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/10939 ) Change subject: IMPALA-7259: Improve Impala shell performance .. IMPALA-7259: Improve Impala shell performance This patch fixes the slow performance in Impala shell by replacing all calls to sqlparse.format(sql, strip_comments=True) with the custom implementation of strip comments that does not use grouping. The code to strip leading comments was also refactored to not use grouping. * Benchmark running a query with 12K columns * Before the patch: $ time impala-shell.sh -f large.sql --quiet real2m4.154s user2m0.536s sys 0m0.088s After the patch: $ time impala-shell.sh -f large.sql --quiet real0m3.885s user0m1.516s sys 0m0.048s Testing: - Added a new test to test the Impala shell performance - Ran all shell tests Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 --- M shell/impala_shell.py A tests/shell/test_file_large.sql M tests/shell/test_shell_commandline.py 3 files changed, 16,051 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10939/5 -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya
[Impala-ASF-CR] IMPALA-7259: Improve Impala shell performance
Fredy Wijaya has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10939 Change subject: IMPALA-7259: Improve Impala shell performance .. IMPALA-7259: Improve Impala shell performance This patch fixes the slow performance in Impala shell by replacing all calls to sqlparse.format(sql, strip_comments=True) with the custom implementation of strip comments that does not use grouping. The code to strip leading comments was also refactored to not use grouping. * Benchmark running a query with 12K columns * Before the patch: $ time impala-shell.sh -f large.sql --quiet real2m4.154s user2m0.536s sys 0m0.088s After the patch: $ time impala-shell.sh -f large.sql --quiet real0m3.885s user0m1.516s sys 0m0.048s Testing: - Added a new test to test the Impala shell performance - Ran all shell tests Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 --- M shell/impala_shell.py M tests/shell/test_shell_commandline.py 2 files changed, 46 insertions(+), 14 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/39/10939/4 -- To view, visit http://gerrit.cloudera.org:8080/10939 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idac9f3caed7c44846a8c922dbe5ca3bf3b095b81 Gerrit-Change-Number: 10939 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya
