[Impala-ASF-CR] Give each config change in bootstrap system.sh its own line
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12900 ) Change subject: Give each config change in bootstrap_system.sh its own line .. Patch Set 4: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2608/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1d3ae6c0b816113b7bf690adff4f1cd905388776 Gerrit-Change-Number: 12900 Gerrit-PatchSet: 4 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 02 Apr 2019 03:13:17 + Gerrit-HasComments: No
[Impala-ASF-CR] Give each config change in bootstrap system.sh its own line
Impala Public Jenkins has posted comments on this change. (
http://gerrit.cloudera.org:8080/12900 )
Change subject: Give each config change in bootstrap_system.sh its own line
..
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/12900/4/bin/bootstrap_system.sh
File bin/bootstrap_system.sh:
http://gerrit.cloudera.org:8080/#/c/12900/4/bin/bootstrap_system.sh@303
PS4, Line 303: { echo "" | cat - ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys; }
&& chmod 0600 ~/.ssh/authorized_keys
line too long (101 > 90)
http://gerrit.cloudera.org:8080/#/c/12900/4/bin/bootstrap_system.sh@304
PS4, Line 304: echo -e "\nNoHostAuthenticationForLocalhost yes" >>
~/.ssh/config && chmod 0600 ~/.ssh/config
line too long (93 > 90)
--
To view, visit http://gerrit.cloudera.org:8080/12900
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1d3ae6c0b816113b7bf690adff4f1cd905388776
Gerrit-Change-Number: 12900
Gerrit-PatchSet: 4
Gerrit-Owner: Jim Apple
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Jim Apple
Gerrit-Reviewer: Joe McDonnell
Gerrit-Reviewer: Lars Volker
Gerrit-Comment-Date: Tue, 02 Apr 2019 02:28:30 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Give each config change in bootstrap system.sh its own line
Hello Lars Volker, Joe McDonnell, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12900 to look at the new patch set (#4). Change subject: Give each config change in bootstrap_system.sh its own line .. Give each config change in bootstrap_system.sh its own line If a config file like ~/.ssh/config does not end in a newline character, than the change to that file in bootstrap_system.sh would get appended to the last line, causeing parsing of that line to fail. Change-Id: I1d3ae6c0b816113b7bf690adff4f1cd905388776 --- M bin/bootstrap_system.sh 1 file changed, 8 insertions(+), 7 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/00/12900/4 -- To view, visit http://gerrit.cloudera.org:8080/12900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I1d3ae6c0b816113b7bf690adff4f1cd905388776 Gerrit-Change-Number: 12900 Gerrit-PatchSet: 4 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker
[Impala-ASF-CR] Give each config change in bootstrap system.sh its own line
Jim Apple has posted comments on this change. ( http://gerrit.cloudera.org:8080/12900 ) Change subject: Give each config change in bootstrap_system.sh its own line .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/12900/3/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/12900/3/bin/bootstrap_system.sh@303 PS3, Line 303: cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 0600 ~/.ssh/authorized_keys > Is this file affected by the same issue? Good point. Updated here and elsewhere. -- To view, visit http://gerrit.cloudera.org:8080/12900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1d3ae6c0b816113b7bf690adff4f1cd905388776 Gerrit-Change-Number: 12900 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Jim Apple Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Tue, 02 Apr 2019 02:28:16 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8143: Enhance DoRpcWithRetry().
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12672 )
Change subject: IMPALA-8143: Enhance DoRpcWithRetry().
..
Patch Set 7:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc
File be/src/rpc/rpc-mgr-test.cc:
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr-test.cc@333
PS5, Line 333: // Call DoRpcWithRetry with no retry on our busy service.
: const int retries = 3;
: PingRequestPB request3;
: PingResponsePB response3;
: impala::Status rpc_status = RpcMgr::DoRpcWithRetry(proxy,
::Ping,
: request3, , query_ctx, "ping failed", retries,
100 * MILLIS_PER_SEC);
: EXPECT_ERROR(rpc_status, TErrorCode::GENERAL);
: EXPECT_STR_CONTAINS(rpc_status.msg().msg(),
: "dropped due to backpressure. The service queue contains 1
items out of a maximum "
: "of 1; memory consumption is ");
: ASSERT_EQ(overflow_metric->GetValue(), retries);
:
: // Call DoRpcWithRetry, but this time we do retey on a busy
service, and this waiting
: // allows the outstanding async calls to complete, so that
then this call will succeed.
: PingRequestPB request4;
: PingResponsePB response4;
: impala::Status rpc_status_backoff =
: RpcMgr::DoRpcWithRetry(proxy, ::Ping,
request4, ,
: query_ctx, "ping failed", 10, 100 * MILLIS_PER_SEC, 2
* MILLIS_PER_SEC);
: ASSERT_OK(rpc_status_backoff);
: ASSERT_GT(overflow_metric->GetValue(
> I reviewed the test and tried to make to more bulletproof. I think any test
How about the following idea:
- the RPC handler can share some sort of condition variable or barrier with the
main test. The test can then indirectly control when the RPC handler will
complete by signaling those condition variables.
In this way, as long as the main thread doesn't signal the CV, the first RPC
will never finish and the second RPC will also block forever. This makes the
test not timing dependent and less prone to failure due to random timing issue.
What do you think ?
http://gerrit.cloudera.org:8080/#/c/12672/7/be/src/rpc/rpc-mgr.h
File be/src/rpc/rpc-mgr.h:
http://gerrit.cloudera.org:8080/#/c/12672/7/be/src/rpc/rpc-mgr.h@178
PS7, Line 178: // A no-op method that is used as part of overloading
DoRpcWithRetry().
: static void logging_noop() {}
:
: // The type of the log method passed to DoRpcWithRetry().
: typedef boost::function RpcLogFn;
Not needed.
http://gerrit.cloudera.org:8080/#/c/12672/7/be/src/rpc/rpc-mgr.h@186
PS7, Line 186: timeout
timeout_ms
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr.h
File be/src/rpc/rpc-mgr.h:
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr.h@178
PS5, Line 178: // A no-op method that is used as part of overloading
DoRpcWithRetry().
: static void logging_noop() {}
:
: // The type of the log method passed to DoRpcWithRetry().
: typedef boost::function RpcLogFn;
> Removed
They still seem to be in PS7
http://gerrit.cloudera.org:8080/#/c/12672/5/be/src/rpc/rpc-mgr.h@201
PS5, Line 201:
: private:
: /// One pool per registered service. scoped_refptr<> is
dictated by the Kudu interface.
: std::vector> service_pools_;
:
> OK I'll leave this alone and Thomas will clean up the mess? :-)
Sounds good to me.
--
To view, visit http://gerrit.cloudera.org:8080/12672
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ia9693151c35e02235665b3c285a48c585973d390
Gerrit-Change-Number: 12672
Gerrit-PatchSet: 7
Gerrit-Owner: Andrew Sherman
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Thomas Marshall
Gerrit-Comment-Date: Tue, 02 Apr 2019 02:08:54 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] Use -- to denote startup flags for consistency
Alex Rodoni has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12908 ) Change subject: [DOCS] Use -- to denote startup flags for consistency .. [DOCS] Use -- to denote startup flags for consistency Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e Reviewed-on: http://gerrit.cloudera.org:8080/12908 Tested-by: Impala Public Jenkins Reviewed-by: Alex Rodoni --- M docs/shared/impala_common.xml M docs/topics/impala_admission_config.xml M docs/topics/impala_auditing.xml M docs/topics/impala_config_options.xml M docs/topics/impala_custom_timezones.xml M docs/topics/impala_datetime_functions.xml M docs/topics/impala_dedicated_coordinator.xml M docs/topics/impala_delegation.xml M docs/topics/impala_idle_session_timeout.xml M docs/topics/impala_isilon.xml M docs/topics/impala_jdbc.xml M docs/topics/impala_kerberos.xml M docs/topics/impala_known_issues.xml M docs/topics/impala_kudu_read_mode.xml M docs/topics/impala_ldap.xml M docs/topics/impala_logging.xml M docs/topics/impala_mem_limit.xml M docs/topics/impala_metadata.xml M docs/topics/impala_timestamp.xml M docs/topics/impala_upgrading.xml 20 files changed, 3,671 insertions(+), 2,618 deletions(-) Approvals: Impala Public Jenkins: Verified Alex Rodoni: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/12908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e Gerrit-Change-Number: 12908 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] [DOCS] Use -- to denote startup flags for consistency
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/12908 ) Change subject: [DOCS] Use -- to denote startup flags for consistency .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e Gerrit-Change-Number: 12908 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Apr 2019 01:26:22 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Use -- to denote startup flags for consistency
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12908 ) Change subject: [DOCS] Use -- to denote startup flags for consistency .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/297/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e Gerrit-Change-Number: 12908 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Apr 2019 01:21:35 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Use -- to denote startup flags for consistency
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12908 ) Change subject: [DOCS] Use -- to denote startup flags for consistency .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/296/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/12908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e Gerrit-Change-Number: 12908 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Apr 2019 01:20:57 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Use -- to denote startup flags for consistency
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12908 ) Change subject: [DOCS] Use -- to denote startup flags for consistency .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/297/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e Gerrit-Change-Number: 12908 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Apr 2019 01:14:45 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Use -- to denote startup flags for consistency
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12908 to look at the new patch set (#2). Change subject: [DOCS] Use -- to denote startup flags for consistency .. [DOCS] Use -- to denote startup flags for consistency Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e --- M docs/shared/impala_common.xml M docs/topics/impala_admission_config.xml M docs/topics/impala_auditing.xml M docs/topics/impala_config_options.xml M docs/topics/impala_custom_timezones.xml M docs/topics/impala_datetime_functions.xml M docs/topics/impala_dedicated_coordinator.xml M docs/topics/impala_delegation.xml M docs/topics/impala_idle_session_timeout.xml M docs/topics/impala_isilon.xml M docs/topics/impala_jdbc.xml M docs/topics/impala_kerberos.xml M docs/topics/impala_known_issues.xml M docs/topics/impala_kudu_read_mode.xml M docs/topics/impala_ldap.xml M docs/topics/impala_logging.xml M docs/topics/impala_mem_limit.xml M docs/topics/impala_metadata.xml M docs/topics/impala_timestamp.xml M docs/topics/impala_upgrading.xml 20 files changed, 3,671 insertions(+), 2,618 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/12908/2 -- To view, visit http://gerrit.cloudera.org:8080/12908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e Gerrit-Change-Number: 12908 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-2990: timeout unresponsive queries in coordinator
Michael Ho has posted comments on this change. (
http://gerrit.cloudera.org:8080/12299 )
Change subject: IMPALA-2990: timeout unresponsive queries in coordinator
..
Patch Set 7:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/runtime/coordinator-backend-state.cc
File be/src/runtime/coordinator-backend-state.cc:
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/runtime/coordinator-backend-state.cc@174
PS7, Line 174: const auto trigger = MakeScopeExitTrigger(
: [this]() { last_report_time_ms_ =
GenerateReportTimestamp(); });
Does it implicitly rely on the LIFO order for local variables' destruction in
C++ so "notifier" won't be destroyed before "trigger" ? Otherwise, we could
have notified 'exec_complete_barrier' without setting 'last_report_time_ms'.
I may be paranoid but can you please add a comment about this so future changes
won't accidentally swap their orders ? I suppose we can skip the use of
NotifyBarrierOnExit here and just notify 'exec_complete_barrier' in 'trigger'
after setting 'last_report_tims_ms_'.
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/runtime/coordinator.cc@549
PS7, Line 549: DCHECK(
DCHECK_LE
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/runtime/coordinator.cc@664
PS7, Line 664: DCHECK(
DCHECK_LE
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@398
PS4, Line 398:
> k. We could deprecate it in a separate patch I suppose. Fine with this as i
As Thomas said, this config option was useful when users faced instability with
large Impala cluster due to IMPALA-2990 (which this patch is fixing). We can
consider deprecating this flag once we are convinced that this patch makes
enough improvement that the instability with large cluster is mostly mitigated.
The periodic reporting matters for long running queries. The default reporting
interval of 5 seconds could be longer than most of the short running query's
duration. For long running queries, it probably doesn't really matter to get a
report every 5 seconds vs say 30s or 60s. If we ever deprecate this flag,
please consider bumping the the default reporting interval to something larger.
http://gerrit.cloudera.org:8080/#/c/12299/4/be/src/service/impala-server.cc@2288
PS4, Line 2288: return Status::OK();
: }
:
: void ImpalaServer::ExpireQuery(ClientRequestState* crs, const
Status& status) {
: DCHECK(!status.ok());
: cancellation_thread_pool_->Offer(
> k, would be interested what Michael thinks about that idea
The executor's reporting path currently can tolerate up to
FLAGS_status_report_max_retries number of failures before giving up and
declaring the coordinator as inaccessible. It's intentionally not a time-based
approach and instead relies on the periodic reporting to handle the retry.
Stepping back a bit, my understanding for why we want to approximate the
maximum retry time is that the coordinator may not want to prematurely cancel a
query due to a missing report from an executor if it knows that the executor
*may* still be retrying to send a report. However, as pointed out elsewhere by
Todd, if an Impala can experience random pauses or random scheduling delay for
the reporting thread, no amount of wait time will guarantee that an executor is
not in the middle of retrying to send a report when the query is cancelled. I
suppose that's not an end-goal we try to achieve.
This prompts me to think whether it's any different if we just do the following:
- expose a coordinator's maximum tolerable lag time as a knob set to arbitrary
number
- expose an executor's maximum tolerable retry time as a knob set to some
arbitrary number
- deprecate FLAGS_status_report_max_retries
An alternative would be to:
- expose an executor's maximum tolerable retry time as a knob set to some
arbitrary number
- deprecate FLAGS_status_report_max_retries
- set coordinator's maximum tolerable lag time as (100 + x)% of the maximum
tolerable retry time where x *may be* tunable in case the default value causes
any trouble.
Todd / Thomas, what do you think ?
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
http://gerrit.cloudera.org:8080/#/c/12299/7/be/src/service/impala-server.cc@218
PS7, Line 218: DEFINE_int32(status_report_max_retries, 3,
> Is our timeout too aggressive here? If one of the backends goes into some k
I agree that the coordinator should be more lenient in handling lost reports to
avoid false positive. On the other hand, no amount of timeout can save us from
random pauses in Impalad so we should work on
[Impala-ASF-CR] [DOCS] Use -- to denote startup flags for consistency
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12908 ) Change subject: [DOCS] Use -- to denote startup flags for consistency .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/296/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/12908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e Gerrit-Change-Number: 12908 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 02 Apr 2019 00:58:50 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Use -- to denote startup flags for consistency
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12908 Change subject: [DOCS] Use -- to denote startup flags for consistency .. [DOCS] Use -- to denote startup flags for consistency Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e --- M docs/shared/impala_common.xml M docs/topics/impala_admission_config.xml M docs/topics/impala_auditing.xml M docs/topics/impala_config_options.xml M docs/topics/impala_custom_timezones.xml M docs/topics/impala_datetime_functions.xml M docs/topics/impala_dedicated_coordinator.xml M docs/topics/impala_delegation.xml M docs/topics/impala_idle_session_timeout.xml M docs/topics/impala_isilon.xml M docs/topics/impala_jdbc.xml M docs/topics/impala_kerberos.xml M docs/topics/impala_known_issues.xml M docs/topics/impala_kudu_read_mode.xml M docs/topics/impala_ldap.xml M docs/topics/impala_logging.xml M docs/topics/impala_mem_limit.xml M docs/topics/impala_metadata.xml M docs/topics/impala_timestamp.xml M docs/topics/impala_upgrading.xml 20 files changed, 413 insertions(+), 453 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/08/12908/1 -- To view, visit http://gerrit.cloudera.org:8080/12908 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ia665e3f230f218d8bbf998dfd1ae21338c21b36e Gerrit-Change-Number: 12908 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] Fix stress test tracking of past metrics values
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12903 ) Change subject: Fix stress test tracking of past metrics values .. Fix stress test tracking of past metrics values The stress test keeps track of various metrics about queries that it has already run, such as the number of completed queries and the number of errors. The stress test also forks off several processes to handle running of queries. The intention is that the metrics should be shared across all of the processes. However, a recent patch (IMPALA-6662) changed these metrics to be stored in a dict and in the process caused them to no longer be shared. This patch solves the problem by storing the value in the dict as 'Value' objects, which are shared. Synchronization of access to these values is handled by taking '_query_runners_lock'. Testing: - Ran the stress test for 1000 queries on tpch 500 on a 10 node cluster. Previously, this would reach a point where there were 0 'Active' queries and hang. Change-Id: I03b5e24b41fff1d33f9bb64b14a6d0b7a23a56e2 Reviewed-on: http://gerrit.cloudera.org:8080/12903 Reviewed-by: Tim Armstrong Tested-by: Impala Public Jenkins --- M tests/stress/concurrent_select.py 1 file changed, 6 insertions(+), 4 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/12903 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I03b5e24b41fff1d33f9bb64b14a6d0b7a23a56e2 Gerrit-Change-Number: 12903 Gerrit-PatchSet: 2 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Fix stress test tracking of past metrics values
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12903 ) Change subject: Fix stress test tracking of past metrics values .. Patch Set 1: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12903 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I03b5e24b41fff1d33f9bb64b14a6d0b7a23a56e2 Gerrit-Change-Number: 12903 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 22:40:14 + Gerrit-HasComments: No
[Impala-ASF-CR] Improve error handling for validation of unified backend executable
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12895 ) Change subject: Improve error handling for validation of unified backend executable .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia84cd074be79bc9d99b0621a1ae216f3debf5833 Gerrit-Change-Number: 12895 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 01 Apr 2019 21:32:41 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7971: Add support for insert events in event processor.
Bharath Krishna has posted comments on this change. ( http://gerrit.cloudera.org:8080/12889 ) Change subject: IMPALA-7971: Add support for insert events in event processor. .. Patch Set 4: (1 comment) http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java File fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java: http://gerrit.cloudera.org:8080/#/c/12889/2/fe/src/main/java/org/apache/impala/service/CatalogOpExecutor.java@3778 PS2, Line 3778: rqst.setPartitionVals(partVals); > Should this condition be reversed, because it will be false by default? Done -- To view, visit http://gerrit.cloudera.org:8080/12889 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I7c48c5ca4bde18d532c582980aebbc25f1bf1c52 Gerrit-Change-Number: 12889 Gerrit-PatchSet: 4 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Krishna Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Paul Rogers Gerrit-Reviewer: Vihang Karajgaonkar Gerrit-Comment-Date: Mon, 01 Apr 2019 20:46:41 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Revert "Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8"
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12905 ) Change subject: Revert "Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8" .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2607/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I501721c428b63d035ad618ac7da97acfb828537c Gerrit-Change-Number: 12905 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 01 Apr 2019 18:35:41 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix stress test tracking of past metrics values
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12903 ) Change subject: Fix stress test tracking of past metrics values .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12903 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I03b5e24b41fff1d33f9bb64b14a6d0b7a23a56e2 Gerrit-Change-Number: 12903 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 17:54:02 + Gerrit-HasComments: No
[Impala-ASF-CR] Fix stress test tracking of past metrics values
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12903 ) Change subject: Fix stress test tracking of past metrics values .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/3970/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12903 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I03b5e24b41fff1d33f9bb64b14a6d0b7a23a56e2 Gerrit-Change-Number: 12903 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 18:02:23 + Gerrit-HasComments: No
[Impala-ASF-CR] Revert "Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8"
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12905 Change subject: Revert "Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8" .. Revert "Bump toolchain version to 8430101a-b65f-4e4d-921e-767771649ae4-713afc77a8" It appears that there's something wrong with the Kudu binaries that causes all Impala binaries to crash when built against them. This was not noticed in testing this patch as we generally use the CDH produced Kudu binaries, but it breaks Impala on noo-CDH supported platforms. Revert while we figure out what's going on. Change-Id: I501721c428b63d035ad618ac7da97acfb828537c --- M bin/impala-config.sh 1 file changed, 3 insertions(+), 3 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/12905/1 -- To view, visit http://gerrit.cloudera.org:8080/12905 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I501721c428b63d035ad618ac7da97acfb828537c Gerrit-Change-Number: 12905 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[native-toolchain-CR] Include python-dev in all build images
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12882 ) Change subject: Include python-dev in all build images .. Include python-dev in all build images Thrift doesn't fail the build if it fails to find the python development headers, it instead silently builds without support for the TBinaryProtocolAccelerated class. This commit adds python-dev to the missing docker images and adds a couple of assertions to ensure we fail the build whenever this happens. Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 Reviewed-on: http://gerrit.cloudera.org:8080/12882 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M docker/all/assert-dependencies-present.py M docker/debian7.df M docker/debian8.df M docker/ubuntu1204.df M docker/ubuntu1404.df M docker/ubuntu1604.df M docker/ubuntu1804.df M source/thrift/build.sh 8 files changed, 22 insertions(+), 1 deletion(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 Gerrit-Change-Number: 12882 Gerrit-PatchSet: 5 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] Fix stress test tracking of past metrics values
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12903 ) Change subject: Fix stress test tracking of past metrics values .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2606/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12903 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I03b5e24b41fff1d33f9bb64b14a6d0b7a23a56e2 Gerrit-Change-Number: 12903 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 17:16:50 + Gerrit-HasComments: No
[Impala-ASF-CR] Ensure ssh config change is on its own line
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/12900 ) Change subject: Ensure ssh config change is on its own line .. Patch Set 3: (1 comment) http://gerrit.cloudera.org:8080/#/c/12900/3/bin/bootstrap_system.sh File bin/bootstrap_system.sh: http://gerrit.cloudera.org:8080/#/c/12900/3/bin/bootstrap_system.sh@303 PS3, Line 303: cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys && chmod 0600 ~/.ssh/authorized_keys Is this file affected by the same issue? -- To view, visit http://gerrit.cloudera.org:8080/12900 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I1d3ae6c0b816113b7bf690adff4f1cd905388776 Gerrit-Change-Number: 12900 Gerrit-PatchSet: 3 Gerrit-Owner: Jim Apple Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Joe McDonnell Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 01 Apr 2019 17:12:49 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Austin Nobis has posted comments on this change. (
http://gerrit.cloudera.org:8080/12901 )
Change subject: IMPALA-8309: add user authorization_provider flag
..
Patch Set 2:
(13 comments)
You also need to add E2E tests in the python testing.
http://gerrit.cloudera.org:8080/#/c/12901/2/be/src/service/frontend.cc
File be/src/service/frontend.cc:
http://gerrit.cloudera.org:8080/#/c/12901/2/be/src/service/frontend.cc@41
PS2, Line 41: "sentry",
nit: move default to line above like other DEFINE
http://gerrit.cloudera.org:8080/#/c/12901/2/be/src/service/frontend.cc@45
PS2, Line 45: "",
nit: move default to line above like other DEFINE.
http://gerrit.cloudera.org:8080/#/c/12901/2/be/src/service/frontend.cc@46
PS2, Line 46: "Specifies the class name that implements the authorization
provider. "
: "This will override the authorization_provider flag if both
are specified."
nit: Wrap at the character limit not at the end of a sentence.
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java
File
fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java:
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java@31
PS2, Line 31: factoryClass
Should this be `factoryClassName`?
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java
File
fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java:
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java@67
PS2, Line 67: public String getProviderName() { return "none"; }
Can we make "none" a public static final String?
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java
File
fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java:
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java@45
PS2, Line 45: public String getProviderName() { return "ranger"; }
Can we make "ranger" a public static final String?
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java
File
fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java:
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java@130
PS2, Line 130: public String getProviderName() { return "sentry"; }
Can we make "sentry" a public static final String?
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/BackendConfig.java
File fe/src/main/java/org/apache/impala/service/BackendConfig.java:
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/BackendConfig.java@141
PS2, Line 141: public @Nullable String getAuthorizationFactoryClassOrNull() {
> Not sure if there's a standard for handling optional flags... at least I di
nit: I'm not opposed to adding the @Nullable annotation and adding `orNull` in
the class name but it is inconsistent with the rest of the code base.
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java
File fe/src/main/java/org/apache/impala/service/JniFrontend.java:
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@773
PS2, Line 773: * @param beCfg
:* @return
:* @throws InternalException
Finish the documentation
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@778
PS2, Line 778: throws InternalException {
nit: indent 4
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@794
PS2, Line 794: +authzProvider
nit: add space around the `+`
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@812
PS2, Line 812: throws InternalException {
nit: indent 4
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/JniFrontend.java@822
PS2, Line 822: +authzFactoryClassName
nit: add space around the `+`
--
To view, visit http://gerrit.cloudera.org:8080/12901
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19
Gerrit-Change-Number: 12901
Gerrit-PatchSet: 2
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Fix stress test tracking of past metrics values
Thomas Marshall has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12903 Change subject: Fix stress test tracking of past metrics values .. Fix stress test tracking of past metrics values The stress test keeps track of various metrics about queries that it has already run, such as the number of completed queries and the number of errors. The stress test also forks off several processes to handle running of queries. The intention is that the metrics should be shared across all of the processes. However, a recent patch (IMPALA-6662) changed these metrics to be stored in a dict and in the process caused them to no longer be shared. This patch solves the problem by storing the value in the dict as 'Value' objects, which are shared. Synchronization of access to these values is handled by taking '_query_runners_lock'. Testing: - Ran the stress test for 1000 queries on tpch 500 on a 10 node cluster. Previously, this would reach a point where there were 0 'Active' queries and hang. Change-Id: I03b5e24b41fff1d33f9bb64b14a6d0b7a23a56e2 --- M tests/stress/concurrent_select.py 1 file changed, 6 insertions(+), 4 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/12903/1 -- To view, visit http://gerrit.cloudera.org:8080/12903 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I03b5e24b41fff1d33f9bb64b14a6d0b7a23a56e2 Gerrit-Change-Number: 12903 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Marshall
[native-toolchain-CR] Parallelize image building process
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12864 ) Change subject: Parallelize image building process .. Parallelize image building process Images can now be built in parallel when using the new buildall.py script. Other than the parallelism and logging changes, the script behaves the same way. Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 Reviewed-on: http://gerrit.cloudera.org:8080/12864 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M docker/README.md A docker/buildall.py D docker/buildall.sh 3 files changed, 57 insertions(+), 39 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 Gerrit-Change-Number: 12864 Gerrit-PatchSet: 5 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Fix debian:7 and debian:8 image builds
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12863 ) Change subject: Fix debian:7 and debian:8 image builds .. Fix debian:7 and debian:8 image builds Wheezy and Jessie were moved out of the main debian mirrors: https://lists.debian.org/debian-devel-announce/2019/03/msg6.html So we're fetching from archive now. Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 Reviewed-on: http://gerrit.cloudera.org:8080/12863 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- A docker/debian/7/sources.list A docker/debian/8/sources.list M docker/debian7.df M docker/debian8.df 4 files changed, 14 insertions(+), 0 deletions(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12863 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 Gerrit-Change-Number: 12863 Gerrit-PatchSet: 5 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Work around flatbuffers flaky build
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12886 ) Change subject: Work around flatbuffers flaky build .. Patch Set 4: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57e95617ccfc2d7d63cc277ab2b858f28ccad44c Gerrit-Change-Number: 12886 Gerrit-PatchSet: 4 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:20:44 + Gerrit-HasComments: No
[native-toolchain-CR] Parallelize image building process
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12864 ) Change subject: Parallelize image building process .. Patch Set 4: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 Gerrit-Change-Number: 12864 Gerrit-PatchSet: 4 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:20:13 + Gerrit-HasComments: No
[native-toolchain-CR] Work around flatbuffers flaky build
Tim Armstrong has submitted this change and it was merged. (
http://gerrit.cloudera.org:8080/12886 )
Change subject: Work around flatbuffers flaky build
..
Work around flatbuffers flaky build
The flatbuffers build ocasionally fails with the following error:
/flatbuffers/flatbuffers-1.6.0/src/idl_parser.cpp
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:19:17:
error: 'MyGame' has not been declared
using namespace MyGame::Sample;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:19:25:
error: 'Sample' is not a namespace-name
using namespace MyGame::Sample;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:19:31:
error: expected namespace-name before ';' token
using namespace MyGame::Sample;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp: In
function 'int main(int, const char**)':
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:25:3:
error: 'flatbuffers' has not been declared
flatbuffers::FlatBufferBuilder builder;
^
/mnt/source/flatbuffers/flatbuffers-1.6.0/samples/sample_binary.cpp:28:26:
error: 'builder' was not declared in this scope
auto weapon_one_name = builder.CreateString("Sword");
^
...
Disabling the BUILD_TESTS target removes the flakiness and makes the
compilation faster.
Change-Id: I57e95617ccfc2d7d63cc277ab2b858f28ccad44c
Reviewed-on: http://gerrit.cloudera.org:8080/12886
Reviewed-by: Tim Armstrong
Tested-by: Tim Armstrong
---
M source/flatbuffers/build.sh
1 file changed, 6 insertions(+), 1 deletion(-)
Approvals:
Tim Armstrong: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/12886
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: native-toolchain
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I57e95617ccfc2d7d63cc277ab2b858f28ccad44c
Gerrit-Change-Number: 12886
Gerrit-PatchSet: 5
Gerrit-Owner: Hector Acosta
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Include python-dev in all build images
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12882 ) Change subject: Include python-dev in all build images .. Patch Set 4: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 Gerrit-Change-Number: 12882 Gerrit-PatchSet: 4 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:20:28 + Gerrit-HasComments: No
[native-toolchain-CR] Fix debian:7 and debian:8 image builds
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12863 ) Change subject: Fix debian:7 and debian:8 image builds .. Patch Set 4: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12863 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 Gerrit-Change-Number: 12863 Gerrit-PatchSet: 4 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:19:51 + Gerrit-HasComments: No
[native-toolchain-CR] Prepend docker image to both stderr and stdout
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12861 ) Change subject: Prepend docker image to both stderr and stdout .. Prepend docker image to both stderr and stdout Sometimes, it may be hard to debug which distro produced an error, this commit ensures that both stderr and stdout get piped through sed. Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 Reviewed-on: http://gerrit.cloudera.org:8080/12861 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M Makefile 1 file changed, 1 insertion(+), 1 deletion(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 Gerrit-Change-Number: 12861 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Install newer nss version in centos 6.6
Tim Armstrong has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer nss version in centos 6.6 .. Install newer nss version in centos 6.6 Centos 6.6 ships with an nss version that can't clone from github. To fix it we include a new Centos-Vault repo file with disabled repositories and explicitly install nss from centos 6.9, which works. Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Reviewed-on: http://gerrit.cloudera.org:8080/12862 Reviewed-by: Tim Armstrong Tested-by: Tim Armstrong --- M docker/all/assert-dependencies-present.py A docker/redhat/Centos6-Vault.repo M docker/redhat6.df 3 files changed, 23 insertions(+), 1 deletion(-) Approvals: Tim Armstrong: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 5 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong
[native-toolchain-CR] Work around flatbuffers flaky build
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12886 ) Change subject: Work around flatbuffers flaky build .. Patch Set 3: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12886 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I57e95617ccfc2d7d63cc277ab2b858f28ccad44c Gerrit-Change-Number: 12886 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:18:29 + Gerrit-HasComments: No
[native-toolchain-CR] Install newer nss version in centos 6.6
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12862 ) Change subject: Install newer nss version in centos 6.6 .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12862 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I66549477ec9378faeda46a98cf6556a3207979eb Gerrit-Change-Number: 12862 Gerrit-PatchSet: 4 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Laszlo Gaal Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:18:03 + Gerrit-HasComments: No
[native-toolchain-CR] Parallelize image building process
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12864 ) Change subject: Parallelize image building process .. Patch Set 3: Verified+1 Code-Review+2 (1 comment) http://gerrit.cloudera.org:8080/#/c/12864/3/docker/buildall.py File docker/buildall.py: http://gerrit.cloudera.org:8080/#/c/12864/3/docker/buildall.py@49 PS3, Line 49: sys.exit(exit) I didn't realise you could pass strings to sys.exit() so I looked at the docs and learned something new. -- To view, visit http://gerrit.cloudera.org:8080/12864 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I526b0088020bb7c647f1918b3eb3258cc1e3ddf9 Gerrit-Change-Number: 12864 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:15:08 + Gerrit-HasComments: Yes
[Impala-ASF-CR] Improve error handling for validation of unified backend executable
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/12895 ) Change subject: Improve error handling for validation of unified backend executable .. Patch Set 1: Code-Review+1 LGTM -- To view, visit http://gerrit.cloudera.org:8080/12895 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia84cd074be79bc9d99b0621a1ae216f3debf5833 Gerrit-Change-Number: 12895 Gerrit-PatchSet: 1 Gerrit-Owner: Joe McDonnell Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 01 Apr 2019 16:14:20 + Gerrit-HasComments: No
[native-toolchain-CR] Fix debian:7 and debian:8 image builds
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12863 ) Change subject: Fix debian:7 and debian:8 image builds .. Patch Set 3: Verified+1 Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12863 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Id85298dc83e419aa8fbff6bd89b13f446adc93a8 Gerrit-Change-Number: 12863 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:09:49 + Gerrit-HasComments: No
[native-toolchain-CR] Prepend docker image to both stderr and stdout
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12861 ) Change subject: Prepend docker image to both stderr and stdout .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 Gerrit-Change-Number: 12861 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:08:44 + Gerrit-HasComments: No
[native-toolchain-CR] Prepend docker image to both stderr and stdout
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12861 ) Change subject: Prepend docker image to both stderr and stdout .. Patch Set 2: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12861 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I16dd725b0c98599867892801c7146c4fe60f46e1 Gerrit-Change-Number: 12861 Gerrit-PatchSet: 2 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:08:49 + Gerrit-HasComments: No
[native-toolchain-CR] Include python-dev in all build images
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12882 ) Change subject: Include python-dev in all build images .. Patch Set 3: Hector sent me a pointer to the completed build. -- To view, visit http://gerrit.cloudera.org:8080/12882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 Gerrit-Change-Number: 12882 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:07:13 + Gerrit-HasComments: No
[native-toolchain-CR] Include python-dev in all build images
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/12882 ) Change subject: Include python-dev in all build images .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/12882 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: native-toolchain Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Iac6186317b943e3bb7609c9309d7bfdde245e4d8 Gerrit-Change-Number: 12882 Gerrit-PatchSet: 3 Gerrit-Owner: Hector Acosta Gerrit-Reviewer: Hector Acosta Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 01 Apr 2019 16:06:46 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7211: Fix the between predicate for decimals
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12902 ) Change subject: IMPALA-7211: Fix the between predicate for decimals .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2605/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12902 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Iac89d62082052b41bfa698e4de09aec26df5c312 Gerrit-Change-Number: 12902 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 01 Apr 2019 15:23:35 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-7211: Fix the between predicate for decimals
Hello Vuk Ercegovac, Impala Public Jenkins, I'd like you to do a code review. Please visit http://gerrit.cloudera.org:8080/12902 to review the following change. Change subject: IMPALA-7211: Fix the between predicate for decimals .. IMPALA-7211: Fix the between predicate for decimals Before this patch, some queries would fail where the inputs to the between predicate were decimal types that are not compatible with each other. We would needlessly cast them to the same type in the FE. This was not necessary because we are able to handle decimals of different types in the backend already for this predicate. This patch should even result in a slight performance improvement. Testing: - Added some tests to AnalyzeExprsTest. Change-Id: Iac89d62082052b41bfa698e4de09aec26df5c312 Reviewed-on: http://gerrit.cloudera.org:8080/10898 Reviewed-by: Vuk Ercegovac Tested-by: Impala Public Jenkins --- M fe/src/main/java/org/apache/impala/analysis/Analyzer.java M fe/src/main/java/org/apache/impala/analysis/BetweenPredicate.java M fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java 3 files changed, 66 insertions(+), 63 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/02/12902/3 -- To view, visit http://gerrit.cloudera.org:8080/12902 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: Iac89d62082052b41bfa698e4de09aec26df5c312 Gerrit-Change-Number: 12902 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Vuk Ercegovac
[Impala-ASF-CR](2.x) IMPALA-7211: Fix the between predicate for decimals
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/12902 ) Change subject: IMPALA-7211: Fix the between predicate for decimals .. Patch Set 3: Cherry-pick conflicts: fe/src/test/java/org/apache/impala/analysis/AnalyzeExprsTest.java How to resolve the conflicts: We should explictly enable decimal_v2 in decimal cast relative tests so AnalysisError and AnalysisOk can work as expected. -- To view, visit http://gerrit.cloudera.org:8080/12902 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: Iac89d62082052b41bfa698e4de09aec26df5c312 Gerrit-Change-Number: 12902 Gerrit-PatchSet: 3 Gerrit-Owner: Quanlong Huang Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Vuk Ercegovac Gerrit-Comment-Date: Mon, 01 Apr 2019 14:47:29 + Gerrit-HasComments: No
[Impala-ASF-CR] Make Docker-based tests available on more hosting OS platforms
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12892 ) Change subject: Make Docker-based tests available on more hosting OS platforms .. Patch Set 3: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2604/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d326611772facc193d8e509a827515f10207808 Gerrit-Change-Number: 12892 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 01 Apr 2019 12:49:52 + Gerrit-HasComments: No
[Impala-ASF-CR] Make Docker-based tests available on more hosting OS platforms
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12892 ) Change subject: Make Docker-based tests available on more hosting OS platforms .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2603/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d326611772facc193d8e509a827515f10207808 Gerrit-Change-Number: 12892 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Laszlo Gaal Gerrit-Comment-Date: Mon, 01 Apr 2019 12:37:54 + Gerrit-HasComments: No
[Impala-ASF-CR] Make Docker-based tests available on more hosting OS platforms
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12892 to look at the new patch set (#2). Change subject: Make Docker-based tests available on more hosting OS platforms .. Make Docker-based tests available on more hosting OS platforms Tweak the test driver and the monitoring code to be compatible with more OS versions: 1. Change the code collecting memory statistics to go directly to /proc/meminfo instead of parsing the output of 'free', which requires different arguments on different OS distributions. This makes test-with-docker.py depend on the MemAvailable field in /proc/meminfo, which should be present on kernels v3.14 and above. This also foregoes forking 'free' once per second. 2. Relax the check for a valid timezone specifier. On Ubuntu 14.04 /etc/localtime is a copy of a timezone file instead of being a symlink to a timezone file. Add an alternative check that accepts this option for timezone synchronization between the host and the Docker containers. 3. Centos 6 did not like the leading dash in the 'ps' command argument list. Change this to a universally accepted form by dropping the leading dash. Tested by running 'ps axho' manually on all OSs supported for Docker-based tests. Change-Id: I6d326611772facc193d8e509a827515f10207808 --- M docker/entrypoint.sh M docker/monitor.py M docker/test-with-docker.py 3 files changed, 51 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/12892/2 -- To view, visit http://gerrit.cloudera.org:8080/12892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6d326611772facc193d8e509a827515f10207808 Gerrit-Change-Number: 12892 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Make Docker-based tests available on more hosting OS platforms
Laszlo Gaal has posted comments on this change. (
http://gerrit.cloudera.org:8080/12892 )
Change subject: Make Docker-based tests available on more hosting OS platforms
..
Patch Set 3:
(3 comments)
Fixed flake8 style findings.
http://gerrit.cloudera.org:8080/#/c/12892/1/docker/monitor.py
File docker/monitor.py:
http://gerrit.cloudera.org:8080/#/c/12892/1/docker/monitor.py@75
PS1, Line 75:
> flake8: E501 line too long (93 > 90 characters)
Done
http://gerrit.cloudera.org:8080/#/c/12892/1/docker/monitor.py@81
PS1, Line 81:
> flake8: E201 whitespace after '('
Done
http://gerrit.cloudera.org:8080/#/c/12892/1/docker/monitor.py@88
PS1, Line 88:
> flake8: E302 expected 2 blank lines, found 1
Done
--
To view, visit http://gerrit.cloudera.org:8080/12892
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I6d326611772facc193d8e509a827515f10207808
Gerrit-Change-Number: 12892
Gerrit-PatchSet: 3
Gerrit-Owner: Laszlo Gaal
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Laszlo Gaal
Gerrit-Comment-Date: Mon, 01 Apr 2019 12:07:02 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] Make Docker-based tests available on more hosting OS platforms
Hello Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12892 to look at the new patch set (#3). Change subject: Make Docker-based tests available on more hosting OS platforms .. Make Docker-based tests available on more hosting OS platforms Tweak the test driver and the monitoring code to be compatible with more OS versions: 1. Change the code collecting memory statistics to go directly to /proc/meminfo instead of parsing the output of 'free', which requires different arguments on different OS distributions. This makes test-with-docker.py depend on the MemAvailable field in /proc/meminfo, which should be present on kernels v3.14 and above. This also foregoes forking 'free' once per second. 2. Relax the check for a valid timezone specifier. On Ubuntu 14.04 /etc/localtime is a copy of a timezone file instead of being a symlink to a timezone file. Add an alternative check that accepts this option for timezone synchronization between the host and the Docker containers. 3. Centos 6 did not like the leading dash in the 'ps' command argument list. Change this to a universally accepted form by dropping the leading dash. Tested by running 'ps axho' manually on all OSs supported for Docker-based tests. Change-Id: I6d326611772facc193d8e509a827515f10207808 --- M docker/entrypoint.sh M docker/monitor.py M docker/test-with-docker.py 3 files changed, 51 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/92/12892/3 -- To view, visit http://gerrit.cloudera.org:8080/12892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I6d326611772facc193d8e509a827515f10207808 Gerrit-Change-Number: 12892 Gerrit-PatchSet: 3 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] Make Docker-based tests available on more hosting OS platforms
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12892 ) Change subject: Make Docker-based tests available on more hosting OS platforms .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/12892/2/docker/monitor.py File docker/monitor.py: http://gerrit.cloudera.org:8080/#/c/12892/2/docker/monitor.py@75 PS2, Line 75: flake8: W291 trailing whitespace http://gerrit.cloudera.org:8080/#/c/12892/2/docker/monitor.py@75 PS2, Line 75: https://gitlab.com/procps-ng/procps/blob/master/proc/sysinfo.c#L795), line has trailing whitespace -- To view, visit http://gerrit.cloudera.org:8080/12892 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6d326611772facc193d8e509a827515f10207808 Gerrit-Change-Number: 12892 Gerrit-PatchSet: 2 Gerrit-Owner: Laszlo Gaal Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 01 Apr 2019 12:05:34 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2602/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 2 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 01 Apr 2019 10:02:09 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/12901 )
Change subject: IMPALA-8309: add user authorization_provider flag
..
Patch Set 2:
(10 comments)
> Patch Set 1:
>
> (8 comments)
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/BackendConfig.java
File fe/src/main/java/org/apache/impala/service/BackendConfig.java:
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/main/java/org/apache/impala/service/BackendConfig.java@141
PS2, Line 141: public @Nullable String getAuthorizationFactoryClassOrNull() {
Not sure if there's a standard for handling optional flags... at least I didn't
really see one.
We could alternatively return an `Optional` (generally my preference) or just
return the raw value that the be populates and let callers decide what is
"present" or not.
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/main/java/org/apache/impala/service/JniCatalog.java
File fe/src/main/java/org/apache/impala/service/JniCatalog.java:
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/main/java/org/apache/impala/service/JniCatalog.java@113
PS1, Line 113: final AuthorizationFactory authzFactory
> line too long (99 > 90)
Ack
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java
File fe/src/test/java/org/apache/impala/service/JniFrontendTest.java:
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java@92
PS1, Line 92: Class
authorization_factory_class,
> line too long (105 > 90)
Ack
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java@93
PS1, Line 93: String authorization_provider) {
> line too long (91 > 90)
Ack
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java@110
PS1, Line 110: JniFrontend.authzFactoryClassNameFrom(authCfg("",
"ranger"))
> line too long (102 > 90)
Ack
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java@113
PS1, Line 113:
> line too long (98 > 90)
Ack
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java@118
PS1, Line 118: SentryAuthorizationFactory.class.getCanonicalName(),
> line too long (98 > 90)
Ack
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java@123
PS1, Line 123:
> line too long (98 > 90)
Ack
http://gerrit.cloudera.org:8080/#/c/12901/1/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java@128
PS1, Line 128: )
> line too long (98 > 90)
Ack
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java
File fe/src/test/java/org/apache/impala/service/JniFrontendTest.java:
http://gerrit.cloudera.org:8080/#/c/12901/2/fe/src/test/java/org/apache/impala/service/JniFrontendTest.java@47
PS2, Line 47: origFlags = BackendConfig.INSTANCE.getBackendCfg();
This seems to be what other tests do
--
To view, visit http://gerrit.cloudera.org:8080/12901
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19
Gerrit-Change-Number: 12901
Gerrit-PatchSet: 2
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Austin Nobis
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Todd Lipcon
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Mon, 01 Apr 2019 09:38:38 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Hello Austin Nobis, Fredy Wijaya, Todd Lipcon, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/12901 to look at the new patch set (#2). Change subject: IMPALA-8309: add user authorization_provider flag .. IMPALA-8309: add user authorization_provider flag This commit adds a `authorization_provider` user-facing flag in order to provide a more human-readable alternative to the `authorization_factory_class` for internally-provided authorization strategies. The `authorization_factory_class` flag is retained, but no longer takes a default value if not specified. The default for `authorization_provider` is "sentry" in order to retain backwards-compatibility. If specified, `authorization_factory_class` will take precedence. Testing: Manually started minicluster with each of following flags and verified correct authorization strategy chosen: - provider='' factory='' => sentry - provider=sentry factory='' => sentry - provider=ranger factory='' => ranger - provider='' factory=sentry => sentry - provider='' factory=ranger => ranger - provider=sentry factory=sentry => sentry - provider=ranger factory=sentry => sentry - provider=sentry factory=ranger => ranger - provider=ranger factory=ranger => ranger Wrote unit tests to capture above assertions Ran fe unit and e2e tests Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 --- M be/src/service/frontend.cc M be/src/util/backend-gflag-util.cc M common/thrift/BackendGflags.thrift M fe/src/main/java/org/apache/impala/authorization/AuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/AuthorizationProvider.java M fe/src/main/java/org/apache/impala/authorization/NoneAuthorizationFactory.java M fe/src/main/java/org/apache/impala/authorization/ranger/RangerAuthorizationConfig.java M fe/src/main/java/org/apache/impala/authorization/sentry/SentryAuthorizationConfig.java M fe/src/main/java/org/apache/impala/catalog/CatalogServiceCatalog.java M fe/src/main/java/org/apache/impala/service/BackendConfig.java M fe/src/main/java/org/apache/impala/service/JniCatalog.java M fe/src/main/java/org/apache/impala/service/JniFrontend.java M fe/src/test/java/org/apache/impala/common/FrontendTestBase.java M fe/src/test/java/org/apache/impala/service/JniFrontendTest.java 14 files changed, 217 insertions(+), 57 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/12901/2 -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 2 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8309: add user authorization provider flag
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12901 ) Change subject: IMPALA-8309: add user authorization_provider flag .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/2601/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12901 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I001c20505ba4f0562b60fdef73d15308e8500c19 Gerrit-Change-Number: 12901 Gerrit-PatchSet: 1 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Austin Nobis Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Todd Lipcon Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 01 Apr 2019 06:04:21 + Gerrit-HasComments: No
