[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Bikramjeet Vig has posted comments on this change. (
http://gerrit.cloudera.org:8080/10813 )
Change subject: IMPALA-7163: Implement a state machine for the QueryState class
..
Patch Set 13:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/10813/13//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/10813/13//COMMIT_MSG@28
PS13, Line 28: The fragment instances update the query wide query status if an
error is hit
nit: a few long lines here
http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.h
File be/src/runtime/fragment-instance-state.h:
http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.h@79
PS13, Line 79: hould WaitForPrepare
nit: update comment
http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc
File be/src/runtime/fragment-instance-state.cc:
http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc@86
PS13, Line 86: if (!status.ok()) {
: goto done;
: }
nit: one line
http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc@98
PS13, Line 98: if (!status.ok()) {
: goto done;
: }
nit: one line
http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/query-state.cc
File be/src/runtime/query-state.cc:
http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/query-state.cc@217
PS13, Line 217:
The method comment mentions that : "A state transition happens if the current
state is a non-terminal state"
maybe add if(IsTerminalState(old_state)) return query_status_;
or add a dcheck for that
otherwise we can potentially set an error/cancel state after
BackendExecState::FINISHED
http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/query-state.cc@390
PS13, Line 390: fis_map_.emplace(fis->instance_id(), fis);
any specific reason why you moved this before you actually start up the thread?
If you keep it here, then maybe update the comment above it
http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/query-state.cc@435
PS13, Line 435: ReportExecStatusAux(true, thread_create_status, nullptr, true);
looks like we can now return a done exec status before the started fragments
complete prepare phase. Is that expected? or should we wait for prepare here.
the commit message says that:
"The PREPARING
state blocks regardless of whether a fragment instance hit an error or not,
until all the fragment instances have completed successfully or unsuccessfully"
we should probably call wait for prepare before calling UpdateBackendExecState()
--
To view, visit http://gerrit.cloudera.org:8080/10813
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe
Gerrit-Change-Number: 10813
Gerrit-PatchSet: 13
Gerrit-Owner: Sailesh Mukil
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Comment-Date: Sat, 04 Aug 2018 03:45:28 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7387: Set correct MIME type for JSON webpages
Anonymous Coward #168 has posted comments on this change. ( http://gerrit.cloudera.org:8080/0 ) Change subject: IMPALA-7387: Set correct MIME type for JSON webpages .. Patch Set 9: (1 comment) Patch set 11 http://gerrit.cloudera.org:8080/#/c/0/9/be/src/util/webserver.cc File be/src/util/webserver.cc: http://gerrit.cloudera.org:8080/#/c/0/9/be/src/util/webserver.cc@151 PS9, Line 151: \ > I don't think that is a standard. Relevant discussion here [1] which refers please : https://blog.csdn.net/gufachongyang02/article/details/21627539 -- To view, visit http://gerrit.cloudera.org:8080/0 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ad94343151730851a4ad01a1f4b9326a36a37ea Gerrit-Change-Number: 0 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 04 Aug 2018 02:40:01 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6789: disable impersonation in hive in minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonation in hive in minicluster .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/188/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 02:32:28 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonation in hive in minicluster
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9 to look at the new patch set (#3). Change subject: IMPALA-6789: disable impersonation in hive in minicluster .. IMPALA-6789: disable impersonation in hive in minicluster Due to bug of HADOOP-7050, users with dots in their username can't launch HiveServer2 in minicluster. To work arround this, we can set hive.server2.enable.doAs to false to disable impersonation in hive. Impala's authorization only depends on Sentry, so we can disable this without breaking any tests. This patch also quotes the group name in AuthorizationStmtTest#testShow to avoid syntax errors when group name contains dots. Test: * Build succeed with username quanlong.huang locally Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/resources/mysql-hive-site.xml.template M fe/src/test/resources/postgresql-hive-site.xml.template 3 files changed, 16 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/9/3 -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 2: (3 comments) http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG@7 PS2, Line 7: impersonate > nit: impersonation Done http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG@11 PS2, Line 11: impersonate > same Done http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@963 PS1, Line 963: `%s` > Ah ok. Is this the only test that is broken due to this? Wondering if there Yes. Currently, it's the only place that needs fixing. I build successfully in our internal Jenkins. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 01:58:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/7 ) Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2920/ -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 04 Aug 2018 01:43:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 2: Code-Review+1 (3 comments) Patch lgtm. I'll let Phil take a pass since he helped you root cause this. http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG@7 PS2, Line 7: impersonate nit: impersonation http://gerrit.cloudera.org:8080/#/c/9/2//COMMIT_MSG@11 PS2, Line 11: impersonate same http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@963 PS1, Line 963: `%s` > Sorry that I only mentioned this in the commit message. My username is quan Ah ok. Is this the only test that is broken due to this? Wondering if there are other places that need some fixing. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 01:34:32 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/187/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 01:30:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7396: unblock testing with --thread creation fault injection
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11123 ) Change subject: IMPALA-7396: unblock testing with --thread_creation_fault_injection .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/186/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Idc9da645c449a266e3a54d0e35b18a2823019cb7 Gerrit-Change-Number: 11123 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 04 Aug 2018 01:00:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Hello Bharath Vissapragada, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/9 to look at the new patch set (#2). Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. IMPALA-6789: disable impersonate in hive in minicluster Due to bug of HADOOP-7050, users with dots in their username can't launch HiveServer2 in minicluster. To work arround this, we can set hive.server2.enable.doAs to false to disable impersonate in hive. Impala's authorization only depends on Sentry, so we can disable this without breaking any tests. This patch also quotes the group name in AuthorizationStmtTest#testShow to avoid syntax errors when group name contains dots. Test: * Build succeed with username quanlong.huang locally Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/resources/mysql-hive-site.xml.template M fe/src/test/resources/postgresql-hive-site.xml.template 3 files changed, 17 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/9/2 -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 2 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@963 PS1, Line 963: `%s` > Why this? Do you have some usernames that conflict with keywords or somethi Sorry that I only mentioned this in the commit message. My username is quanlong.huang which will have syntax error if the name is not quoted. The DOT in the name is the cause. http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template File fe/src/test/resources/postgresql-hive-site.xml.template: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template@145 PS1, Line 145: : hive.server2.enable.impersonation : Enable user impersonation for HiveServer2 : true : > This looks the same. Is it deprecated in favor of doAs? If so, this should Yes, according to https://github.com/cloudera/hive/commit/1cac04b0e739019b9da99fe916e2fe5497c4abb8 , this is the old name of hive.server2.enable.doAs http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template@198 PS1, Line 198: : hive.server2.enable.doAs : false : > Can you add a comment as to why we are explicitly disabling this. Done -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Sat, 04 Aug 2018 00:58:28 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7396: unblock testing with --thread creation fault injection
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11123 Change subject: IMPALA-7396: unblock testing with --thread_creation_fault_injection .. IMPALA-7396: unblock testing with --thread_creation_fault_injection These are minimal fixes to allow testing new patches with the fault injection. ClientRequestState: fixing this seems somewhat involved, so disabled the fault injection for now to unblock further testing. HdfsScanNode: thread_state_.DecrementNumActive() was called on the error path before the count had actually been incremented. Testing: Ran some of the end-to-end tests for 20-30 minutes and confirmed that no Impalads crashed. start-impala-cluster.py \ --impalad_args=--thread_creation_fault_injection=true impala-py.test tests/query_test/*.py -n2 --verbose Change-Id: Idc9da645c449a266e3a54d0e35b18a2823019cb7 --- M be/src/exec/hdfs-scan-node.cc M be/src/service/client-request-state.cc 2 files changed, 3 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/23/11123/1 -- To view, visit http://gerrit.cloudera.org:8080/11123 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Idc9da645c449a266e3a54d0e35b18a2823019cb7 Gerrit-Change-Number: 11123 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR] IMPALA-7387: Set correct MIME type for JSON webpages
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/0 ) Change subject: IMPALA-7387: Set correct MIME type for JSON webpages .. Patch Set 10: 21:54:06 [INFO] BUILD FAILURE 21:54:06 [ERROR] Failed to execute goal on project impala-frontend: Could not resolve dependencies for project org.apache.impala:impala-frontend:jar:0.1-SNAPSHOT: Could not transfer artifact org.apache.sentry:sentry-provider-db:jar:2.0.0-cdh6.x-20180731.151011-502571 from/to impala.cdh.repo (https://native-toolchain.s3.amazonaws.com/build/cdh_components/506967/maven): Access denied to: https://native-toolchain.s3.amazonaws.com/build/cdh_components/506967/maven/org/apache/sentry/sentry-provider-db/2.0.0-cdh6.x-SNAPSHOT/sentry-provider-db-2.0.0-cdh6.x-20180731.151011-502571.jar , ReasonPhrase:Forbidden. -> [Help 1] 21:54:06 [ERROR] 21:54:06 [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. 21:54:06 [ERROR] Re-run Maven using the -X switch to enable full debug logging. 21:54:06 [ERROR] -- To view, visit http://gerrit.cloudera.org:8080/0 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ad94343151730851a4ad01a1f4b9326a36a37ea Gerrit-Change-Number: 0 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Sat, 04 Aug 2018 00:17:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 1: (3 comments) http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java File fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java@963 PS1, Line 963: `%s` Why this? Do you have some usernames that conflict with keywords or something? http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template File fe/src/test/resources/postgresql-hive-site.xml.template: http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template@145 PS1, Line 145: : hive.server2.enable.impersonation : Enable user impersonation for HiveServer2 : true : This looks the same. Is it deprecated in favor of doAs? If so, this should probably be deleted. http://gerrit.cloudera.org:8080/#/c/9/1/fe/src/test/resources/postgresql-hive-site.xml.template@198 PS1, Line 198: : hive.server2.enable.doAs : false : Can you add a comment as to why we are explicitly disabling this. Also place it in the Kerberos section? -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Sat, 04 Aug 2018 00:15:09 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 8: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/185/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 8 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Sat, 04 Aug 2018 00:03:55 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 6: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2918/ -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 23:52:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/9 ) Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/183/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 03 Aug 2018 23:46:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10847 ) Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. Patch Set 7: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/184/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 7 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga Gerrit-Comment-Date: Fri, 03 Aug 2018 23:46:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11120 ) Change subject: IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement .. IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Reviewed-on: http://gerrit.cloudera.org:8080/11120 Tested-by: Impala Public Jenkins Reviewed-by: Tim Armstrong --- M docs/topics/impala_truncate_table.xml 1 file changed, 51 insertions(+), 82 deletions(-) Approvals: Impala Public Jenkins: Verified Tim Armstrong: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/11120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Gerrit-Change-Number: 11120 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Hello Lars Volker, Zoram Thanga, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10847 to look at the new patch set (#8). Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. IMPALA-6271: Impala daemon should log a message when it's being shut down Currently Impalad does not log any message when SIGTERM is sent to impalad to terminate or to do a graceful shut down. This change logs a message when SIGTERM is received by impalad/catalogd/statestored. This logging will assist in debugging the issues seen in the field where impalad was not gracefully shut down (some other signal was generated that led to impalad/catalogd/statestored crash). Testing: --- a) Used kill to send signals to impalad/catalogd/statestored `kill -s SIGTERM ` and see the log message is being logged in impalad/catalogd/statestored.INFO. b) Ran test_breakpad.py to check that existing breakpad functionalities are not affected. c) Ran exhaustive tests without failure. d) Added new test in test_breakpad.py to handle SIGTERM for impalad/statestored/catalogd. Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 --- M be/src/common/init.cc M be/src/util/minidump.cc M tests/custom_cluster/test_breakpad.py 3 files changed, 50 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/10847/8 -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 8 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/182/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 23:31:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Hello Lars Volker, Zoram Thanga, Todd Lipcon, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10847 to look at the new patch set (#7). Change subject: IMPALA-6271: Impala daemon should log a message when it's being shut down .. IMPALA-6271: Impala daemon should log a message when it's being shut down Currently Impalad does not log any message when SIGTERM is sent to impalad to terminate or to do a graceful shut down. This change logs a message when SIGTERM is received by impalad/catalogd/statestored. This logging will assist in debugging the issues seen in the field where impalad was not gracefully shut down (some other signal was generated that led to impalad/catalogd/statestored crash). Testing: --- a) Used kill to send signals to impalad/catalogd/statestored `kill -s SIGTERM ` and see the log message is being logged in impalad/catalogd/statestored.INFO. b) Ran test_breakpad.py to check that existing breakpad functionalities are not affected. c) Ran exhaustive tests without failure. d) Added new test in test_breakpad.py to handle SIGTERM for impalad/statestored/catalogd. Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 --- M be/src/common/init.cc M be/src/statestore/statestored-main.cc M be/src/util/minidump.cc M tests/custom_cluster/test_breakpad.py 4 files changed, 50 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/47/10847/7 -- To view, visit http://gerrit.cloudera.org:8080/10847 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29 Gerrit-Change-Number: 10847 Gerrit-PatchSet: 7 Gerrit-Owner: Pranay Singh Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Pranay Singh Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Zoram Thanga
[Impala-ASF-CR] IMPALA-6271: Impala daemon should log a message when it's being shut down
Pranay Singh has posted comments on this change. (
http://gerrit.cloudera.org:8080/10847 )
Change subject: IMPALA-6271: Impala daemon should log a message when it's being
shut down
..
Patch Set 6:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc
File be/src/catalog/catalogd-main.cc:
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@62
PS6, Line 62: HandlerTerm
> "HandleSigTerm"?
Done
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@63
PS6, Line 63: LOG(INFO) << "Caught SIGTERM daemon going down" << endl;
> nit: please format this more grammatically. For example "Caught SIGTERM. Da
Changed the log message and removed the endl , looks like the messages are
being flushed perhaps by atexit hook as you say. Since the testcase check for
the message "Caught SIGTERM" which seems to be working fine.
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@64
PS6, Line 64: exit(0);
> Does this change the behavior? I.e., would we have exited with 0 before, to
This exit will ensure a normal process termination, if I don't use it the
process will not be terminated.
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@120
PS6, Line 120: // We want to log a message when catalogd is
> nit: wrap comments at 90 chars, here and elsewhere.
Done
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/catalog/catalogd-main.cc@125
PS6, Line 125: sigaction(SIGTERM, , nullptr);
> why not set this earlier? seems like there could be a relatively long start
Done
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/service/impalad-main.cc
File be/src/service/impalad-main.cc:
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/service/impalad-main.cc@60
PS6, Line 60: // doing an exit.
> This code is now copy-pasted to several files. Can you dedup it?
Removed the code block and moved it to the common handler
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/service/impalad-main.cc@104
PS6, Line 104: // We want to log a message when impalad is
> same here
Done
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/util/minidump.cc
File be/src/util/minidump.cc:
http://gerrit.cloudera.org:8080/#/c/10847/6/be/src/util/minidump.cc@101
PS6, Line 101: LOG(INFO) << "Signal received SIGUSR1" << endl;
> The other messages are "Caught SIGTERM daemon going down". Can you make the
Done
http://gerrit.cloudera.org:8080/#/c/10847/6/tests/custom_cluster/test_breakpad.py
File tests/custom_cluster/test_breakpad.py:
http://gerrit.cloudera.org:8080/#/c/10847/6/tests/custom_cluster/test_breakpad.py@239
PS6, Line 239: self.assert_impalad_log_contains('INFO', 'Caught SIGTERM
daemon going down',
> Is there a way to also check the exit code of the processes?
Checked the code for exit of the process, could not find the function where the
exit of the process is checked.
--
To view, visit http://gerrit.cloudera.org:8080/10847
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Id20da9e30440b7348557beccb8a0da14775fcc29
Gerrit-Change-Number: 10847
Gerrit-PatchSet: 6
Gerrit-Owner: Pranay Singh
Gerrit-Reviewer: Lars Volker
Gerrit-Reviewer: Pranay Singh
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Zoram Thanga
Gerrit-Comment-Date: Fri, 03 Aug 2018 23:29:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11120 ) Change subject: IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/27/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Gerrit-Change-Number: 11120 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 23:22:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/181/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 23:19:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11120 ) Change subject: IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/27/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Gerrit-Change-Number: 11120 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Fri, 03 Aug 2018 23:14:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/11120 Change subject: IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement .. IMPALA-7395: [DOCS] The TABLE keyword is optional in the TRUNCATE statement Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 --- M docs/topics/impala_truncate_table.xml 1 file changed, 51 insertions(+), 82 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/11120/1 -- To view, visit http://gerrit.cloudera.org:8080/11120 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Iec64cd225449cea07b353201a84ac2fb8bc07a00 Gerrit-Change-Number: 11120 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] IMPALA-6789: disable impersonate in hive in minicluster
Quanlong Huang has uploaded this change for review. ( http://gerrit.cloudera.org:8080/9 Change subject: IMPALA-6789: disable impersonate in hive in minicluster .. IMPALA-6789: disable impersonate in hive in minicluster Due to bug of HADOOP-7050, users with dots in their username can't launch HiveServer2 in minicluster. To work arround this, we can set hive.server2.enable.doAs to false to disable impersonate in hive. Impala's authorization only depends on Sentry, so we can disable this without breaking any tests. This patch also quotes the group name in AuthorizationStmtTest#testShow to avoid syntax errors when group name contains dots. Test: * Build succeed with username quanlong.huang locally Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a --- M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java M fe/src/test/resources/mysql-hive-site.xml.template M fe/src/test/resources/postgresql-hive-site.xml.template 3 files changed, 10 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/19/9/1 -- To view, visit http://gerrit.cloudera.org:8080/9 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I39d8945e0fe90baf7e9e4b26eebab08d2058a14a Gerrit-Change-Number: 9 Gerrit-PatchSet: 1 Gerrit-Owner: Quanlong Huang
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Hello Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/8 to look at the new patch set (#2). Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. IMPALA-5031: Fix undefined behavior: ptr overflow In expr.add, the standard says: When an expression that has integral type is added to or subtracted from a pointer, the result has the type of the pointer operand. ... If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined. In the end-to-end tests this is triggered, and the interesting part of the backtrace is: include/c++/4.9.2/bits/stl_iterator.h:782:45: runtime error: pointer index expression with base 0x overflowed to 0xffe8 #0 __normal_iterator>::operator+(long) const stl_iterator.h:782:45 #1 MemPool::AcquireData(MemPool*, bool) runtime/mem-pool.cc:190:62 #2 RowBatch::TransferResourceOwnership(RowBatch*) runtime/row-batch.cc:444:26 #3 RowBatch::AcquireState(RowBatch*) runtime/row-batch.cc:505:8 #4 HdfsScanNode::GetNextInternal(RuntimeState*, RowBatch*, bool*) exec/hdfs-scan-node.cc:105:16 #5 HdfsScanNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/hdfs-scan-node.cc:81:19 #6 StreamingAggregationNode::GetRowsStreaming(RuntimeState*, RowBatch*) exec/streaming-aggregation-node.cc:116:51 #7 StreamingAggregationNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/streaming-aggregation-node.cc:92:41 Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 --- M be/src/runtime/mem-pool.cc 1 file changed, 4 insertions(+), 2 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8/2 -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 2 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7354: planner test resource estimates for more workloads
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11087 ) Change subject: IMPALA-7354: planner test resource estimates for more workloads .. Patch Set 5: Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java@273 PS5, Line 273: printBytes > My thought was that the reservation is a precise number (if the query gets not too strong an opinion on this one. just noted that in the commit message, part of this change was motivated by user perception so wanted to see more about the difference between these too. I'm fine with keeping it this way. -- To view, visit http://gerrit.cloudera.org:8080/11087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9f836699200ea87fb03bf36abad0e23949ac26 Gerrit-Change-Number: 11087 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 22:51:14 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/8/1/be/src/runtime/mem-pool.cc File be/src/runtime/mem-pool.cc: http://gerrit.cloudera.org:8080/#/c/8/1/be/src/runtime/mem-pool.cc@190 PS1, Line 190: vector::iterator insert_chunk = chunks_.begin() + (current_chunk_idx_ + 1); I had to stare at this for a minute. Is the bug in the case when current_chunk_idx_ is -1? I always found the current_chunk_idx_ logic here confusing... The new code seems fine, might be worth dropping a comment to explain why the parens are significant. -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 22:50:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8 ) Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 22:50:29 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7354: planner test resource estimates for more workloads
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11087 ) Change subject: IMPALA-7354: planner test resource estimates for more workloads .. Patch Set 5: (1 comment) http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java File fe/src/main/java/org/apache/impala/planner/Planner.java: http://gerrit.cloudera.org:8080/#/c/11087/5/fe/src/main/java/org/apache/impala/planner/Planner.java@273 PS5, Line 273: printBytes > just wondering why this one is not changed as well? My thought was that the reservation is a precise number (if the query gets any less than this, it will fail) but the estimate is an imprecise number. I was thinking that the extra precision lets planner tests catch smaller errors in the calculations. On the other hand, sub-megabyte precision isn't informative for users and we still expose and verify it at higher explain levels at plan and fragment levels: https://github.com/apache/impala/blob/master/testdata/workloads/functional-planner/queries/PlannerTest/resource-requirements.test#L8 So I could go either way. Given that it jumped out at you, maybe users will find the inconsistency confusing. What do you think? -- To view, visit http://gerrit.cloudera.org:8080/11087 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9f836699200ea87fb03bf36abad0e23949ac26 Gerrit-Change-Number: 11087 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 22:46:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-5031: Fix undefined behavior: ptr overflow
Jim Apple has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8 Change subject: IMPALA-5031: Fix undefined behavior: ptr overflow .. IMPALA-5031: Fix undefined behavior: ptr overflow In expr.add, the standard says: When an expression that has integral type is added to or subtracted from a pointer, the result has the type of the pointer operand. ... If both the pointer operand and the result point to elements of the same array object, or one past the last element of the array object, the evaluation shall not produce an overflow; otherwise, the behavior is undefined. In the end-to-end tests this is triggered, and the interesting part of the backtrace is: include/c++/4.9.2/bits/stl_iterator.h:782:45: runtime error: pointer index expression with base 0x overflowed to 0xffe8 #0 __normal_iterator>::operator+(long) const stl_iterator.h:782:45 #1 MemPool::AcquireData(MemPool*, bool) runtime/mem-pool.cc:190:62 #2 RowBatch::TransferResourceOwnership(RowBatch*) runtime/row-batch.cc:444:26 #3 RowBatch::AcquireState(RowBatch*) runtime/row-batch.cc:505:8 #4 HdfsScanNode::GetNextInternal(RuntimeState*, RowBatch*, bool*) exec/hdfs-scan-node.cc:105:16 #5 HdfsScanNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/hdfs-scan-node.cc:81:19 #6 StreamingAggregationNode::GetRowsStreaming(RuntimeState*, RowBatch*) exec/streaming-aggregation-node.cc:116:51 #7 StreamingAggregationNode::GetNext(RuntimeState*, RowBatch*, bool*) exec/streaming-aggregation-node.cc:92:41 Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 --- M be/src/runtime/mem-pool.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/18/8/1 -- To view, visit http://gerrit.cloudera.org:8080/8 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3d28a80763adb62572b3dd81ea732d18d957d248 Gerrit-Change-Number: 8 Gerrit-PatchSet: 1 Gerrit-Owner: Jim Apple
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/7 ) Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/180/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 22:35:25 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/7 ) Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2920/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 22:30:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/7 ) Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 22:06:57 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7394: Don't print stack trace in ExpireSessions()
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/7 Change subject: IMPALA-7394: Don't print stack trace in ExpireSessions() .. IMPALA-7394: Don't print stack trace in ExpireSessions() When sessions expire, ExpireSessions() may create excessive amount of log spew due to printing of the stack traces esp when there are a massive amount of sessions expiring at the same time. This stack crawling also unnecessarily extends the critical section under the session_state_map_lock_, increasing the contention for that lock. This change converts the use of Status() to Status::Expected() to remove the unnecessary stack crawl. Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa --- M be/src/service/impala-server.cc 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/17/7/1 -- To view, visit http://gerrit.cloudera.org:8080/7 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I29cb143af96e2eef45479a03365d8f49a2ee2dfa Gerrit-Change-Number: 7 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] IMPALA-7390: Pin rpc-mgr-kerberized-test to localhost.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/3 ) Change subject: IMPALA-7390: Pin rpc-mgr-kerberized-test to localhost. .. Patch Set 1: (1 comment) http://gerrit.cloudera.org:8080/#/c/3/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/3/1//COMMIT_MSG@11 PS1, Line 11: failing due to DNS resolution reasons > Does it mean in general, to run Impala with docker, one has to specify the test-with-docker creates an /etc/hosts file that looks like: 127.0.0.1 localhost That's happening because of a workaround: # Update /etc/hosts to remove the entry for the unique docker hostname, # and instead point it to 127.0.0.1. Otherwise, HttpFS returns Location: # redirects to said hostname, but the relevant datanode isn't listening # on the wildcard address. sed -e /$(hostname)/d /etc/hosts -e /127.0.0.1/s,localhost,"localhost $(hostname)," \ > /tmp/hosts Kerberos decides the hostname is dockercontainerhostname, but the IP is 127.0.0.1, and then when it looks up the reverse, it finds localhost. Interestingly, if I swap the order in /etc/hosts, this test passes, but a different test fails. (thrift-server-test, if I recall correctly.) All that to say, this is quite specific to the way test-with-docker is working (and the way we configure our minicluster). A user genuinely using docker probably would be doing different hostname anyway. The workaround above exists because tests that use webhdfs (e.g., metadata.test_ddl.TestDdlStatements.test_drop_table_with_purge) end up getting the wrong addresses. I'm going to take a little bit more of a look about why WebHDFS is different in this docker environment than not, but I don't want to go down too deep a rabbit hole there. I'll report back if I figure it out, and otherwise I'll just update the comments here a bit. -- To view, visit http://gerrit.cloudera.org:8080/3 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91003cbc86177feb7f99563f61297f7da7fabab4 Gerrit-Change-Number: 3 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 03 Aug 2018 21:14:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7387: Set correct MIME type for JSON webpages
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/0 ) Change subject: IMPALA-7387: Set correct MIME type for JSON webpages .. Patch Set 10: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2919/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/0 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ad94343151730851a4ad01a1f4b9326a36a37ea Gerrit-Change-Number: 0 Gerrit-PatchSet: 10 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 21:01:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7387: Set correct MIME type for JSON webpages
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/0 ) Change subject: IMPALA-7387: Set correct MIME type for JSON webpages .. Patch Set 9: (1 comment) http://gerrit.cloudera.org:8080/#/c/0/9/be/src/util/webserver.cc File be/src/util/webserver.cc: http://gerrit.cloudera.org:8080/#/c/0/9/be/src/util/webserver.cc@151 PS9, Line 151: \ > Compatible Chinese should add charset=utf-8 I don't think that is a standard. Relevant discussion here [1] which refers to [2] that says "Note: No "charset" parameter is defined for this registration. Adding one really has no effect on compliant recipients." Also, I tried running a sample query in chinese charset and I can see it fine in my firefox browser without UTF-8 charset in the HTTP header. Here is a screenshot [3] [1] https://github.com/request/request/issues/383 [2] http://www.iana.org/assignments/media-types/application/json [3] https://pasteboard.co/HxvPg1X.png -- To view, visit http://gerrit.cloudera.org:8080/0 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7ad94343151730851a4ad01a1f4b9326a36a37ea Gerrit-Change-Number: 0 Gerrit-PatchSet: 9 Gerrit-Owner: Bharath Vissapragada Gerrit-Reviewer: Anonymous Coward #168 Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 21:00:35 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11005 ) Change subject: [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/179/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11005 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I669db217f86db5ff2802335e7b1ae8027ea7161c Gerrit-Change-Number: 11005 Gerrit-PatchSet: 7 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 20:48:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 4: clang-tidy failed doing maven compilation related to Sentry. [ERROR] location: class org.apache.impala.util.SentryPolicyService.SentryServiceClient [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/util/SentryUtil.java:[25,52] package org.apache.sentry.provider.db.service.thrift does not exist [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/util/SentryUtil.java:[26,52] package org.apache.sentry.provider.db.service.thrift does not exist [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/util/SentryUtil.java:[50,37] cannot find symbol [ERROR] symbol: class SentryPolicyServiceClient [ERROR] location: class org.apache.impala.util.SentryUtil [ERROR] /home/ubuntu/Impala/fe/src/main/java/org/apache/impala/util/SentryUtil.java:[50,14] cannot find symbol [ERROR] symbol: class TSentryRole [ERROR] location: class org.apache.impala.util.SentryUtil [ERROR] -> [Help 1] [ERROR] This can't possibly have anything to do with that, so I'm re-running. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 20:40:27 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 5: Code-Review+2 Carry. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 5 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 20:40:37 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 6: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 20:33:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 6: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2918/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 20:33:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 5: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2917/ -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 20:28:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [WIP] IMPALA-6153: Execute UpdateFilter() only for executing queries
Pooja Nilangekar has posted comments on this change. (
http://gerrit.cloudera.org:8080/11005 )
Change subject: [WIP] IMPALA-6153: Execute UpdateFilter() only for executing
queries
..
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.h
File be/src/runtime/coordinator.h:
http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.h@301
PS6, Line 301: /// Synchronizes updates to the filter_routing_table_.
: SpinLock filter_update_lock_;
:
: /// Protects filter_routing_table_.
: /// Usage pattern:
: /// 1. To update filter_routing_table_: Acquire shared access
on filter_lock_ and
: ///upgrade to exclusive access by subsequently acquiring
filter_update_lock_.
: /// 2. To read, initialize/destroy filter_routing_table:
Directly acquire exclusive
: ///access on filter_lock_.
: boost::shared_mutex filter_lock_;
:
> maybe write a consolidated comment to better explain the need for two locks
Done
http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.h@334
PS6, Line 334:
: /// Called when the query is done executing due to reaching
EOS or client
: /// cancellation. If 'exec_state_' != EXECUTING, does nothing.
> nit: "Caller must have exclusive access to filter_lock_"
Done
http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/11005/6/be/src/runtime/coordinator.cc@265
PS6, Line 265: lock_guard lock(filter_lock_);
: for (const FragmentExecParams& fragment_params:
schedule_.fragment_exec_params()) {
: int num_instances = fragment_params.instance_exec_par
> fyi: this method is called in exec before any fragments are started. Also,
Make sense. I have removed the comment. I have kept the lock_guard for the sake
of keeping a consistent code flow across all accesses. Acquiring the lock
should not be expensive if there is no contention on it. Or would you recommend
entirely removing it form here?
--
To view, visit http://gerrit.cloudera.org:8080/11005
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I669db217f86db5ff2802335e7b1ae8027ea7161c
Gerrit-Change-Number: 11005
Gerrit-PatchSet: 7
Gerrit-Owner: Pooja Nilangekar
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Pooja Nilangekar
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 03 Aug 2018 20:24:15 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 4: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/2916/ -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 20:23:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. Patch Set 5: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 19:47:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. IMPALA-7377: Update Sentry for the object ownership feature Update CDH_BUILD_NUMBER to 506967, especially for the new Sentry update with the object owership feature (IMPALA-7075). This patch updates the Sentry package names and Sentry Maven dependencies. Testing: - Ran core tests Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Reviewed-on: http://gerrit.cloudera.org:8080/11094 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M bin/impala-config.sh M fe/pom.xml M fe/src/main/java/org/apache/impala/util/SentryPolicyService.java M fe/src/main/java/org/apache/impala/util/SentryProxy.java M fe/src/main/java/org/apache/impala/util/SentryUtil.java M fe/src/test/java/org/apache/impala/analysis/AuthorizationStmtTest.java 6 files changed, 38 insertions(+), 20 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 6 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/178/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 5 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 03 Aug 2018 19:23:24 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/10813 ) Change subject: IMPALA-7163: Implement a state machine for the QueryState class .. Patch Set 12: (3 comments) http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc File be/src/runtime/fragment-instance-state.cc: http://gerrit.cloudera.org:8080/#/c/10813/13/be/src/runtime/fragment-instance-state.cc@107 PS13, Line 107: yS == Apparently WAITING_FOR_CODEGEN < WAITING_FOR_PREPARE http://gerrit.cloudera.org:8080/#/c/10813/12/be/src/runtime/query-state.h File be/src/runtime/query-state.h: http://gerrit.cloudera.org:8080/#/c/10813/12/be/src/runtime/query-state.h@244 PS12, Line 244: Status WaitForFinish(); > We call it towards the end of StartFInstances(). We update the query state Oh right. Missed that. http://gerrit.cloudera.org:8080/#/c/10813/12/tests/failure/test_failpoints.py File tests/failure/test_failpoints.py: http://gerrit.cloudera.org:8080/#/c/10813/12/tests/failure/test_failpoints.py@162 PS12, Line 162: i = 0 > When I remove this, it gives me this error: It should work with for i in range(50): -- To view, visit http://gerrit.cloudera.org:8080/10813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Gerrit-Change-Number: 10813 Gerrit-PatchSet: 12 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 03 Aug 2018 19:10:40 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Pooja Nilangekar has posted comments on this change. ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. Patch Set 5: (12 comments) http://gerrit.cloudera.org:8080/#/c/11052/4//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/11052/4//COMMIT_MSG@9 PS4, Line 9: After this patch, the statestore keeps track of the last successful : heartbeat timestamp which is exposed as a subscriber metric on the : statestore debug page. Also adds a monitoring thread that : periodically checks the last heartbeat timestamp for all the : subscribers and l > After this patch, the statestore keeps track of the last successful heartbe Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h File be/src/statestore/statestore.h: http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@418 PS4, Line 418: ms); > milliseconds or ms Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@442 PS4, Line 442: /// The timestamp of the last successful heartbeat in milliseconds. A timestamp much : /// older than the heartbeat frequency implies an unr > The timestamp of the last successful heartbeat in milliseconds. Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@444 PS4, Line 444: 4_t last_heartbe > heartbeat frequency Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@532 PS4, Line 532: :unique_ptr heartbeat_monitoring_thread_; > Thread that monitors the heartbeats of all subscribers. Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@684 PS4, Line 684: rtbeat_monitoring_frequency_ms mi > the heartbeats of all subscribers every Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@686 PS4, Line 686: > that Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.h@686 PS4, Line 686: : [[noreturn]] void MonitorSub > subscriber's Id. Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.cc File be/src/statestore/statestore.cc: http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.cc@424 PS4, Line 424: DCHECK(metrics != NULL); : metrics_ = metrics; : num_subscribers_metric_ = metrics->AddGauge(STATESTORE_LIVE_SUBSCRIBERS, 0); > move this to Init() and return status Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.cc@1003 PS4, Line 1003: s, > $0 Done http://gerrit.cloudera.org:8080/#/c/11052/4/be/src/statestore/statestore.cc@1004 PS4, Line 1004: > nit: add a space after comma Done http://gerrit.cloudera.org:8080/#/c/11052/4/www/statestore_subscribers.tmpl File www/statestore_subscribers.tmpl: http://gerrit.cloudera.org:8080/#/c/11052/4/www/statestore_subscribers.tmpl@31 PS4, Line 31: Last heartbeat timestamp > nit: add " (ms)" The timestamp is printed in the following format: -mm-dd hh:mm:ss.ms -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 5 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 03 Aug 2018 18:47:54 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6644: Add last heartbeat timestamp into Statestore metric
Pooja Nilangekar has uploaded a new patch set (#5). ( http://gerrit.cloudera.org:8080/11052 ) Change subject: IMPALA-6644: Add last heartbeat timestamp into Statestore metric .. IMPALA-6644: Add last heartbeat timestamp into Statestore metric After this patch, the statestore keeps track of the last successful heartbeat timestamp which is exposed as a subscriber metric on the statestore debug page. Also adds a monitoring thread that periodically checks the last heartbeat timestamp for all the subscribers and logs the IDs of those that have not been updated since the last periodic check. Testing: Manually inspected the Web UI and statestore logs to verify that the last heartbeat timestamp for each subscriber is updated periodically. Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 --- M be/src/statestore/statestore.cc M be/src/statestore/statestore.h M www/statestore_subscribers.tmpl 3 files changed, 67 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/52/11052/5 -- To view, visit http://gerrit.cloudera.org:8080/11052 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I754adccc4569e8219d5d01500cccdfc8782953f7 Gerrit-Change-Number: 11052 Gerrit-PatchSet: 5 Gerrit-Owner: Pooja Nilangekar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Sailesh Mukil Gerrit-Reviewer: Todd Lipcon
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11064 ) Change subject: IMPALA-7362: Add query option to set timezone .. Patch Set 9: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/11064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Gerrit-Change-Number: 11064 Gerrit-PatchSet: 9 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:45:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/11064 ) Change subject: IMPALA-7362: Add query option to set timezone .. IMPALA-7362: Add query option to set timezone This change adds a new query option "timezone" which defines the timezone used for utc<->local conversions. The main goal is to simplify testing, but I think that some users may also find it useful so it is added as a "general" query option. Examples: set timezone=UTC; set timezone="Europe/Budapest" The timezones are validated, but as query options are not sent to the coordinator immediately, the error checking will only happen when running a query. Leading/trailing " and 'characters are stripped because the / character cannot be entered unquoted in some contexts. Currently the timezone has effect in the following cases: -function now() -conversions between unix time and timestamp if flag use_local_tz_for_unix_timestamp_conversions is true -reading parquet timestamps written by Hive if flag convert_legacy_hive_parquet_utc_timestamps is true In the near future Parquet timestamps's isAdjustedToUTC property will be supported, which will decide whether to do utc->local conversion on a per file+column basis. This conversion will be also affected. Testing: - Extended test_local_tz_conversion.py to actually test utc<->local conversion. Until now the effect of flag use_local_tz_for_unix_timestamp_conversions was practically untested. - Added a shell test to check that the default of the query option is the system's timezone. - Added a shell test to check timezone validation. Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Reviewed-on: http://gerrit.cloudera.org:8080/11064 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M be/src/service/impala-server.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift A testdata/workloads/functional-query/queries/QueryTest/local-timestamp-functions.test M tests/custom_cluster/test_local_tz_conversion.py M tests/shell/test_shell_commandline.py M tests/shell/test_shell_interactive.py 9 files changed, 159 insertions(+), 7 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/11064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Gerrit-Change-Number: 11064 Gerrit-PatchSet: 10 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-4690: [DOCS] More content for CONV()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11075 ) Change subject: IMPALA-4690: [DOCS] More content for CONV() .. Patch Set 4: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/26/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 Gerrit-Change-Number: 11075 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Fri, 03 Aug 2018 17:27:19 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11080 ) Change subject: [DOCS] Dedicated coordinators and executors explained .. Patch Set 4: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/25/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:20:43 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2917/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 17:18:35 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 17:18:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/10995 ) Change subject: IMPALA-7325: Incorrect SHOW CREATE VIEW with built-in functions .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/10995 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia164c55fd9459cf5f11eb72561e9cd4ffe1d5367 Gerrit-Change-Number: 10995 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Adam Holley Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 17:15:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4690: [DOCS] More content for CONV()
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/11075 ) Change subject: IMPALA-4690: [DOCS] More content for CONV() .. Patch Set 4: > I think you may have forgotten to upload PS4. Oops, sorry! -- To view, visit http://gerrit.cloudera.org:8080/11075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 Gerrit-Change-Number: 11075 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Fri, 03 Aug 2018 17:07:46 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4690: [DOCS] More content for CONV()
Hello Jim Apple, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11075 to look at the new patch set (#4). Change subject: IMPALA-4690: [DOCS] More content for CONV() .. IMPALA-4690: [DOCS] More content for CONV() Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 --- M docs/topics/impala_math_functions.xml 1 file changed, 54 insertions(+), 9 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/75/11075/4 -- To view, visit http://gerrit.cloudera.org:8080/11075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 Gerrit-Change-Number: 11075 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple
[native-toolchain-CR] IMPALA-7364: Add RapidJson 1.1.0
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11092 ) Change subject: IMPALA-7364: Add RapidJson 1.1.0 .. Patch Set 3: It should be uploaded. LMK if you have any issues with it. -- To view, visit http://gerrit.cloudera.org:8080/11092 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ie98462fde43c97101e95d0761be1594bc2636859 Gerrit-Change-Number: 11092 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Thomas Marshall Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:04:14 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/11080 ) Change subject: [DOCS] Dedicated coordinators and executors explained .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11080/3/docs/impala_keydefs.ditamap File docs/impala_keydefs.ditamap: http://gerrit.cloudera.org:8080/#/c/11080/3/docs/impala_keydefs.ditamap@10960 PS3, Line 10960: impala_dedicated_coordinatori.xml > A typo here which caused the merge to fail. Done -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:03:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11080 ) Change subject: [DOCS] Dedicated coordinators and executors explained .. Patch Set 4: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/25/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 17:02:20 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Hello Greg Rahn, Fredy Wijaya, Alan Choi, Philip Zeyliger, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11080 to look at the new patch set (#4). Change subject: [DOCS] Dedicated coordinators and executors explained .. [DOCS] Dedicated coordinators and executors explained Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 --- M docs/impala.ditamap M docs/impala_keydefs.ditamap A docs/topics/impala_dedicated_coordinator.xml M docs/topics/impala_scalability.xml 4 files changed, 631 insertions(+), 115 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/80/11080/4 -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 17:00:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 4: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2916/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 4 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 17:00:34 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-6034: Add scanned bytes limits per query
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/11081 ) Change subject: IMPALA-6034: Add scanned bytes limits per query .. Patch Set 1: Zoltan, would be good if you could review this one given you've been working on the I/O layer. I think Bikram wanted some more eyes on it. -- To view, visit http://gerrit.cloudera.org:8080/11081 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3e85f80b70b3fce47e637e9322ed0316ee84f6a9 Gerrit-Change-Number: 11081 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Mostafa Mokhtar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 16:59:39 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 3: Code-Review+2 Thanks for fixing this! -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:57:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/177/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:56:34 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Dedicated coordinators and executors explained
Fredy Wijaya has posted comments on this change. ( http://gerrit.cloudera.org:8080/11080 ) Change subject: [DOCS] Dedicated coordinators and executors explained .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/11080/3/docs/impala_keydefs.ditamap File docs/impala_keydefs.ditamap: http://gerrit.cloudera.org:8080/#/c/11080/3/docs/impala_keydefs.ditamap@10960 PS3, Line 10960: impala_dedicated_coordinatori.xml A typo here which caused the merge to fail. [gen-list] [DOTX008E][ERROR] File 'file:/home/ubuntu/Impala/docs/topics/impala_dedicated_coordinatori.xml' does not exist or cannot be loaded. -- To view, visit http://gerrit.cloudera.org:8080/11080 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ic0038eb40951ddf619ea6470a7363122635391d8 Gerrit-Change-Number: 11080 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alan Choi Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Greg Rahn Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 16:49:19 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7390: Pin rpc-mgr-kerberized-test to localhost.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/3 ) Change subject: IMPALA-7390: Pin rpc-mgr-kerberized-test to localhost. .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/3/1//COMMIT_MSG Commit Message: http://gerrit.cloudera.org:8080/#/c/3/1//COMMIT_MSG@11 PS1, Line 11: failing due to DNS resolution reasons Does it mean in general, to run Impala with docker, one has to specify the hostname ? Is there anything we can do at this line (https://github.com/apache/impala/blob/master/be/src/common/init.cc#L191) to fix the issue ? http://gerrit.cloudera.org:8080/#/c/3/1/be/src/rpc/rpc-mgr-kerberized-test.cc File be/src/rpc/rpc-mgr-kerberized-test.cc: http://gerrit.cloudera.org:8080/#/c/3/1/be/src/rpc/rpc-mgr-kerberized-test.cc@86 PS1, Line 86: FLAGS_hostname = "localhost"; nit: may help to add a comment on why InitCommonRuntime() doesn't set this up correctly. -- To view, visit http://gerrit.cloudera.org:8080/3 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I91003cbc86177feb7f99563f61297f7da7fabab4 Gerrit-Change-Number: 3 Gerrit-PatchSet: 1 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Philip Zeyliger Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 03 Aug 2018 16:48:11 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4690: [DOCS] More content for CONV()
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/11075 ) Change subject: IMPALA-4690: [DOCS] More content for CONV() .. Patch Set 3: I think you may have forgotten to upload PS4. -- To view, visit http://gerrit.cloudera.org:8080/11075 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4e424fd5a009ff5aa2d35a403e08fcd33c75fec5 Gerrit-Change-Number: 11075 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Comment-Date: Fri, 03 Aug 2018 16:46:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7203. Support UDFs in LocalCatalog
Vuk Ercegovac has posted comments on this change. (
http://gerrit.cloudera.org:8080/11053 )
Change subject: IMPALA-7203. Support UDFs in LocalCatalog
..
Patch Set 3:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java
File fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java:
http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@75
PS3, Line 75: If the value is 'null', indicates that the function exists but
:* has not been loaded yet.
seems to have been replaced by the bit in FunctionOverloads that specifies if
loading is needed. perhaps stale?
http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/LocalDb.java@190
PS3, Line 190: private void loadFunctionNames() {
lemme see if I've got this correct. each query has its own LocalCatalog,
therefore its own LocalDb. the provider has a lifetime that spans all
LocalCatalogs. As a result, any query that uses any udf will wind up fetching
all function names. a caching provider can help with this.
http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java
File fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java:
http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@64
PS3, Line 64: Load
Retrieve? Unclear if this has a side-effect.
http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@69
PS3, Line 69: from the HMS
remove since its specific for the provider implementation?
http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/catalog/local/MetaProvider.java@71
PS3, Line 71: org.apache.hadoop.hive.metastore.api.
remove the fully qualified name for consistency with the other api's here.
http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/util/FunctionUtils.java
File fe/src/main/java/org/apache/impala/util/FunctionUtils.java:
http://gerrit.cloudera.org:8080/#/c/11053/3/fe/src/main/java/org/apache/impala/util/FunctionUtils.java@84
PS3, Line 84: // TODO(todd): cache these jars based on the mtime and file ID of
the
: // remote JAR? Can we share a cache with the backend?
> agreed, we should re-use FeSupport CacheJar and related methods. however, a
clarification the code that was refactored seems to be used on the catalogd
end. the localdb otoh lives in the impalad. in that case, there's a lib-cache
available so we should use it so that we avoid copying jars around multiple
times to the same host when functions are used in the frontend and the backend.
this redundancy will only be there when an impalad is both a coordinator and an
executor. while this should be less likely, the combo is still supported.
--
To view, visit http://gerrit.cloudera.org:8080/11053
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6130d07b9c641525382a618a9f8da048c7ae75ed
Gerrit-Change-Number: 11053
Gerrit-PatchSet: 3
Gerrit-Owner: Todd Lipcon
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tianyi Wang
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: Vuk Ercegovac
Gerrit-Comment-Date: Fri, 03 Aug 2018 16:41:51 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 3: Code-Review+1 Thanks! I'd like someone who's more of an expert on docker to give this a final +2. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:40:48 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. Patch Set 5: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:22:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. Patch Set 5: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2915/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 5 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:22:06 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11106/2/docker/test-with-docker.py File docker/test-with-docker.py: http://gerrit.cloudera.org:8080/#/c/11106/2/docker/test-with-docker.py@506 PS2, Line 506: # Ubuntu16.04 machines use an absolute symlink here, : # e.g., /etc/localtime -> /usr/share/zoneinfo/America/Los_Angeles : # but CentOS7 uses a relative one, e.g., /etc/localtime -> ../usr/share/zoneinfo/UTC, : # so we use "in" rather than "startswith" > Thaks for the explanation. Yep, you're right. I was still thinking I was using readlink here. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:20:50 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Hello Laszlo Gaal, Attila Jeges, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11106 to look at the new patch set (#3). Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. IMPALA-7385: Fix test-with-docker errors having to do with time zones. ExprTest.TimestampFunctions, query_test.test_scanners.TestOrc.test_type_conversions, and query_test.test_queries.TestHdfsQueries.test_hdfs_scan_node were all failing when using test-with-docker with mismatched dates. As it turns out, there is code that calls readlink(/etc/localtime) and parses the output to identify the current timezone name. This is described in localtime(5) on Ubuntu16: It should be an absolute or relative symbolic link pointing to /usr/share/zoneinfo/, followed by a timezone identifier such as "Europe/Berlin" or "Etc/UTC". ... Because the timezone identifier is extracted from the symlink target name of /etc/localtime, this file may not be a normal file or hardlink." To honor this requirement, and to make the tests pass, I re-jiggered how I pass the time zone information from the host into the container. The previously failing tests now pass. Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f --- M docker/entrypoint.sh M docker/test-with-docker.py 2 files changed, 17 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/06/11106/3 -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 3 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger
[Impala-ASF-CR] IMPALA-7377: Update Sentry for the object ownership feature
Philip Zeyliger has posted comments on this change. ( http://gerrit.cloudera.org:8080/11094 ) Change subject: IMPALA-7377: Update Sentry for the object ownership feature .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I28671d03cf7785334c333055d8f02c8af5645496 Gerrit-Change-Number: 11094 Gerrit-PatchSet: 4 Gerrit-Owner: Fredy Wijaya Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 16:19:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7096: restore scanner thread memory heuristics
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11103 ) Change subject: IMPALA-7096: restore scanner thread memory heuristics .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/176/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/11103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib9907fa8c4d2b0b85f67f4f160899c1c258ad82b Gerrit-Change-Number: 11103 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 16:17:11 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7096: restore scanner thread memory heuristics
Hello Bikramjeet Vig, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11103 to look at the new patch set (#5). Change subject: IMPALA-7096: restore scanner thread memory heuristics .. IMPALA-7096: restore scanner thread memory heuristics This restores some of the heuristics removed in IMPALA-4835 that can help scans from hitting OOM conditions. The heuristics are implemented at the query level rather than in each scan node in isolation. Introduce a ScannerMemLimiter class that belongs to the QueryState that tracks the amount of memory estimated to be consumed for all scanner threads running for the query on the current backend. Also check soft memory limits to see if scanner threads should be started or the current scanner thread should stop. The long-term plan is to switch to the MT scan node implementations. When that happens this code can be removed. In the meantime this code is imperfect but will help avoid OOM in many scenarios. Testing: Added regression tests for HDFS and Kudu where we previously could run out of memory with a low mem_limit. Change-Id: Ib9907fa8c4d2b0b85f67f4f160899c1c258ad82b --- M be/src/exec/hdfs-scan-node.cc M be/src/exec/hdfs-scan-node.h M be/src/exec/kudu-scan-node-base.h M be/src/exec/kudu-scan-node.cc M be/src/exec/kudu-scan-node.h M be/src/exec/scan-node.cc M be/src/exec/scan-node.h M be/src/runtime/CMakeLists.txt M be/src/runtime/query-state.cc M be/src/runtime/query-state.h A be/src/runtime/scanner-mem-limiter.cc A be/src/runtime/scanner-mem-limiter.h A testdata/workloads/functional-query/queries/QueryTest/hdfs-scanner-thread-mem-scaling.test A testdata/workloads/functional-query/queries/QueryTest/kudu-scan-mem-usage.test M tests/query_test/test_mem_usage_scaling.py 15 files changed, 439 insertions(+), 49 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/11103/5 -- To view, visit http://gerrit.cloudera.org:8080/11103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ib9907fa8c4d2b0b85f67f4f160899c1c258ad82b Gerrit-Change-Number: 11103 Gerrit-PatchSet: 5 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-7258: Support querying HBase tables in LocalCatalog
Vuk Ercegovac has posted comments on this change. ( http://gerrit.cloudera.org:8080/11079 ) Change subject: IMPALA-7258: Support querying HBase tables in LocalCatalog .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11079 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I8cc94bc38861443de5a375b7e63d29215e0ca899 Gerrit-Change-Number: 11079 Gerrit-PatchSet: 4 Gerrit-Owner: Tianyi Wang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tianyi Wang Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Fri, 03 Aug 2018 15:15:20 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/11057 )
Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and
TIMESTAMP_MICROS from Parquet
..
Patch Set 2:
(4 comments)
"It seems that parquet.thrift isn't just updated to contain the timestamp
logical type, but a couple of unrelated things are added as well."
Yes, I have updated it to a newer version from
https://github.com/apache/parquet-format/blob/master/src/main/thrift/parquet.thrift
which contains several unrelated changes. I have modified the commit message
to reflect this more clearly.
http://gerrit.cloudera.org:8080/#/c/11057/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11057/1//COMMIT_MSG@19
PS1, Line 19:
> nit: 'a solution'
Done
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-readers.cc@1657
PS1, Line 1657: Create
> Here you use the word "Create", while for TYPE_DECIMAL we use the word "Get
Thanks for noticing the inconsistency - I prefer "Create".
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-readers.cc@1690
PS1, Line 1690: null
> nit: Please use nullptr.
Done
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-stats.inline.h
File be/src/exec/parquet-column-stats.inline.h:
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-stats.inline.h@158
PS1, Line 158: data + size, size, result) == -1) {
: return false;
> nit: too much indentation
Done
--
To view, visit http://gerrit.cloudera.org:8080/11057
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3
Gerrit-Change-Number: 11057
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 03 Aug 2018 14:57:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11057 ) Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/11057/2/tests/query_test/test_scanners.py File tests/query_test/test_scanners.py: http://gerrit.cloudera.org:8080/#/c/11057/2/tests/query_test/test_scanners.py@751 PS2, Line 751: " flake8: E128 continuation line under-indented for visual indent http://gerrit.cloudera.org:8080/#/c/11057/2/tests/query_test/test_scanners.py@762 PS2, Line 762: " flake8: E128 continuation line under-indented for visual indent -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy Gerrit-Comment-Date: Fri, 03 Aug 2018 14:53:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Hello Zoltan Borok-Nagy, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/11057 to look at the new patch set (#2). Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet .. WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and TIMESTAMP_MICROS from Parquet Changes: - parquet.thrift is updated to a newer version which contains the timestamp logical type. - INT64 columns with converted types TIMESTAMP_MILLIS and TIMESTAMP_MICROS can be read as TIMESTAMP. - If the logical type is timestamp, then the type will contain the information whether the UTC->local conversion is necessary. This feature is only supported for the new timestamp types, so INT96 timestamps must still use flag convert_legacy_hive_parquet_utc_timestamps. TODOs: - the tests only work if local time is CET - a solution would be to create a custom cluster test which sets the local time zone, but I would prefer to solve this without restarting the cluster - it would be better to test with more parquet files (for example test both plain and dictionary encoding). - CREATE TABLE LIKE PARQUET should be also tested with the new types Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 --- M be/src/exec/parquet-column-readers.cc M be/src/exec/parquet-column-readers.h M be/src/exec/parquet-column-stats.inline.h M be/src/exec/parquet-common.h M be/src/exec/parquet-metadata-utils.cc M common/thrift/parquet.thrift M fe/src/main/java/org/apache/impala/analysis/ParquetHelper.java M testdata/data/README A testdata/data/int64_timestamp_plain.parquet A testdata/workloads/functional-query/queries/QueryTest/parquet-int64-timestamps.test M tests/query_test/test_scanners.py 11 files changed, 399 insertions(+), 45 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/57/11057/2 -- To view, visit http://gerrit.cloudera.org:8080/11057 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3 Gerrit-Change-Number: 11057 Gerrit-PatchSet: 2 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Zoltan Borok-Nagy
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11064 ) Change subject: IMPALA-7362: Add query option to set timezone .. Patch Set 9: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Gerrit-Change-Number: 11064 Gerrit-PatchSet: 9 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 14:31:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/11064 ) Change subject: IMPALA-7362: Add query option to set timezone .. Patch Set 9: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/2914/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/11064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Gerrit-Change-Number: 11064 Gerrit-PatchSet: 9 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 14:31:42 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7385: Fix test-with-docker errors having to do with time zones.
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/11106 ) Change subject: IMPALA-7385: Fix test-with-docker errors having to do with time zones. .. Patch Set 2: (1 comment) http://gerrit.cloudera.org:8080/#/c/11106/2/docker/test-with-docker.py File docker/test-with-docker.py: http://gerrit.cloudera.org:8080/#/c/11106/2/docker/test-with-docker.py@506 PS2, Line 506: # Ubuntu16.04 machines use an absolute symlink here, : # e.g., /etc/localtime -> /usr/share/zoneinfo/America/Los_Angeles : # but CentOS7 uses a relative one, e.g., /etc/localtime -> ../usr/share/zoneinfo/UTC, : # so we use "in" rather than "startswith" Thaks for the explanation. I tested os.path.realpath() quickly in a REPL and it always returns an absolute canonical path, even if the symlink contained a relative path. Python docs: "os.path.realpath(filename): Return the canonical path of the specified filename, eliminating any symbolic links encountered in the path." so I think, 'startswith' should always work. -- To view, visit http://gerrit.cloudera.org:8080/11106 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia9facfd9741806e7dbb868d8d06d9296bf86e52f Gerrit-Change-Number: 11106 Gerrit-PatchSet: 2 Gerrit-Owner: Philip Zeyliger Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Philip Zeyliger Gerrit-Comment-Date: Fri, 03 Aug 2018 12:55:57 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10813 ) Change subject: IMPALA-7163: Implement a state machine for the QueryState class .. Patch Set 13: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/174/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/10813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Gerrit-Change-Number: 10813 Gerrit-PatchSet: 13 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil Gerrit-Comment-Date: Fri, 03 Aug 2018 12:54:07 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Attila Jeges has posted comments on this change. ( http://gerrit.cloudera.org:8080/11064 ) Change subject: IMPALA-7362: Add query option to set timezone .. Patch Set 8: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/11064 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272 Gerrit-Change-Number: 11064 Gerrit-PatchSet: 8 Gerrit-Owner: Csaba Ringhofer Gerrit-Reviewer: Attila Jeges Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Fri, 03 Aug 2018 12:53:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Attila Jeges has posted comments on this change. (
http://gerrit.cloudera.org:8080/11064 )
Change subject: IMPALA-7362: Add query option to set timezone
..
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/11064/7/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/11064/7/tests/shell/test_shell_commandline.py@54
PS7, Line 54: values[0].strip("[]")
> Options without brackets are normal if the option does not have the default
Thanks for the explanation.
--
To view, visit http://gerrit.cloudera.org:8080/11064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272
Gerrit-Change-Number: 11064
Gerrit-PatchSet: 7
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 03 Aug 2018 12:53:31 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] WIP IMPALA-5050: Add support to read TIMESTAMP MILLIS and TIMESTAMP MICROS from Parquet
Zoltan Borok-Nagy has posted comments on this change. (
http://gerrit.cloudera.org:8080/11057 )
Change subject: WIP IMPALA-5050: Add support to read TIMESTAMP_MILLIS and
TIMESTAMP_MICROS from Parquet
..
Patch Set 1:
(4 comments)
It seems that parquet.thrift isn't just updated to contain the timestamp
logical type, but a couple of unrelated things are added as well.
The code looks, only found some nit comments. Will do another pass next week.
http://gerrit.cloudera.org:8080/#/c/11057/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/11057/1//COMMIT_MSG@19
PS1, Line 19: on
nit: 'a solution'
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-readers.cc
File be/src/exec/parquet-column-readers.cc:
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-readers.cc@1657
PS1, Line 1657: Create
Here you use the word "Create", while for TYPE_DECIMAL we use the word "Get".
I don't have a preference to either words, but these should be consistent I
think.
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-readers.cc@1690
PS1, Line 1690: NULL
nit: Please use nullptr.
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-stats.inline.h
File be/src/exec/parquet-column-stats.inline.h:
http://gerrit.cloudera.org:8080/#/c/11057/1/be/src/exec/parquet-column-stats.inline.h@158
PS1, Line 158:data + size, size, result) == -1) {
: return false;
nit: too much indentation
--
To view, visit http://gerrit.cloudera.org:8080/11057
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4c7c01fffa31b3d2ca3480adf6ff851137dadac3
Gerrit-Change-Number: 11057
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Zoltan Borok-Nagy
Gerrit-Comment-Date: Fri, 03 Aug 2018 12:31:48 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Hello Michael Ho, Joe McDonnell, Bikramjeet Vig, Dan Hecht, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/10813 to look at the new patch set (#13). Change subject: IMPALA-7163: Implement a state machine for the QueryState class .. IMPALA-7163: Implement a state machine for the QueryState class This patch adds a state machine for the QueryState class. The motivation behind this patch is to make the query lifecycle from the point of view of an executor much easier to reason about and this patch is key for a follow on patch for IMPALA-2990 where the status reporting will be per-query rather than per-fragment-instance. Currently, the state machine provides no other purpose, and it will mostly be used for IMPALA-2990. We introduce 5 possible states for the QueryState which include 3 terminal states (FINISHED, CANCELLED and ERROR) and 2 non-terminal states (PREPARING, EXECUTING). The transition from one state to the next is always handled by a single thread which is also the QueryState thread. This thread will additionally bear the purpose of sending periodic updates after IMPALA-4063, which is the primary reason behind having only this thread modify the state of the query. Counting barriers are introduced to keep a count of how many fragment instances have finished Preparing and Executing. These barriers also block until all the fragment instances have finished a respective state. The fragment instances update the query wide query status if an error is hit and unblocks the barrier if it is in the EXECUTING state. The PREPARING state blocks regardless of whether a fragment instance hit an error or not, until all the fragment instances have completed successfully or unsuccessfully, to maintain the invariant that fragment instances cannot be cancelled until the entire QueryState has finished PREPARING. The status reporting protocol has not been changed and remains exactly as it was. Testing: - Added 3 failure points in the query lifecycle using debug actions and added tests to validate the same (extension of IMPALA-7376). - Ran 'core' and 'exhaustive' tests. Future related work: 1) IMPALA-2990: Make status reporting per-query. 2) Try to logically align the FIS states with the QueryState states. 3) Consider mirroring the QueryState state machine to CoordinatorBackendState Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe --- M be/src/runtime/coordinator.cc M be/src/runtime/fragment-instance-state.cc M be/src/runtime/fragment-instance-state.h M be/src/runtime/query-state.cc M be/src/runtime/query-state.h M tests/failure/test_failpoints.py 6 files changed, 283 insertions(+), 67 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/13/10813/13 -- To view, visit http://gerrit.cloudera.org:8080/10813 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe Gerrit-Change-Number: 10813 Gerrit-PatchSet: 13 Gerrit-Owner: Sailesh Mukil Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Dan Hecht Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sailesh Mukil
[Impala-ASF-CR] IMPALA-7163: Implement a state machine for the QueryState class
Sailesh Mukil has posted comments on this change. (
http://gerrit.cloudera.org:8080/10813 )
Change subject: IMPALA-7163: Implement a state machine for the QueryState class
..
Patch Set 12:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/10813/12/be/src/runtime/fragment-instance-state.cc
File be/src/runtime/fragment-instance-state.cc:
http://gerrit.cloudera.org:8080/#/c/10813/12/be/src/runtime/fragment-instance-state.cc@106
PS12, Line 106: if (!status.ok() && current_state_.Load() <=
TFInstanceExecState::WAITING_FOR_PREPARE) {
: // Tell the managing 'QueryState' that we hit an error
during Prepare().
: query_state_->ErrorDuringPrepare(status, instance_id());
: } else if (!status.ok()) {
: // Tell the managing 'QueryState' that we hit an error
during execution.
: query_state_->ErrorDuringExecute(status, instance_id());
: }
> Seems easier to read if written like the following:
Done
http://gerrit.cloudera.org:8080/#/c/10813/12/be/src/runtime/query-state.h
File be/src/runtime/query-state.h:
http://gerrit.cloudera.org:8080/#/c/10813/12/be/src/runtime/query-state.h@244
PS12, Line 244: Status WaitForFinish();
> Is this actually still needed in this patch ? It doesn't seem to be called
We call it towards the end of StartFInstances(). We update the query state only
after this unblocks.
http://gerrit.cloudera.org:8080/#/c/10813/12/tests/failure/test_failpoints.py
File tests/failure/test_failpoints.py:
http://gerrit.cloudera.org:8080/#/c/10813/12/tests/failure/test_failpoints.py@162
PS12, Line 162: i = 0
> This line isn't needed if you use for i in range(50) below.
When I remove this, it gives me this error:
"NameError: global name 'i' is not defined"
--
To view, visit http://gerrit.cloudera.org:8080/10813
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Iec5670a7db83ecae4656d7bb2ea372d3767ba7fe
Gerrit-Change-Number: 10813
Gerrit-PatchSet: 12
Gerrit-Owner: Sailesh Mukil
Gerrit-Reviewer: Bikramjeet Vig
Gerrit-Reviewer: Dan Hecht
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sailesh Mukil
Gerrit-Comment-Date: Fri, 03 Aug 2018 12:19:39 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-6335. Allow most shell tests to run in parallel
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/11045 ) Change subject: IMPALA-6335. Allow most shell tests to run in parallel .. Patch Set 6: (1 comment) http://gerrit.cloudera.org:8080/#/c/11045/6/tests/shell/test_shell_interactive.py File tests/shell/test_shell_interactive.py: http://gerrit.cloudera.org:8080/#/c/11045/6/tests/shell/test_shell_interactive.py@126 PS6, Line 126: tempfile. Hint for fixing build: "tempfile." should be removed. -- To view, visit http://gerrit.cloudera.org:8080/11045 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1da5739276e63a50590dfcb2b050703f8e35fec7 Gerrit-Change-Number: 11045 Gerrit-PatchSet: 6 Gerrit-Owner: Todd Lipcon Gerrit-Reviewer: Csaba Ringhofer Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Pooja Nilangekar Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Fri, 03 Aug 2018 12:16:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-7362: Add query option to set timezone
Csaba Ringhofer has posted comments on this change. (
http://gerrit.cloudera.org:8080/11064 )
Change subject: IMPALA-7362: Add query option to set timezone
..
Patch Set 8:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/11064/7/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/11064/7/common/thrift/ImpalaService.thrift@332
PS7, Line 332: overridde
> nit: overridden
Done
http://gerrit.cloudera.org:8080/#/c/11064/7/tests/shell/test_shell_commandline.py
File tests/shell/test_shell_commandline.py:
http://gerrit.cloudera.org:8080/#/c/11064/7/tests/shell/test_shell_commandline.py@54
PS7, Line 54: values[0].strip("[]")
> I think, values[0][1:-1] might be more correct here. In extreme cases, valu
Options without brackets are normal if the option does not have the default
value. I have changed the regexp to reflect this + added a unit test for
test_find_query_option.
--
To view, visit http://gerrit.cloudera.org:8080/11064
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I73de86eff096e1c581d3b56a0d9330d686f77272
Gerrit-Change-Number: 11064
Gerrit-PatchSet: 8
Gerrit-Owner: Csaba Ringhofer
Gerrit-Reviewer: Attila Jeges
Gerrit-Reviewer: Csaba Ringhofer
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Fri, 03 Aug 2018 11:55:40 +
Gerrit-HasComments: Yes
