[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Bharath Vissapragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. Patch Set 11: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 11 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Aug 2019 05:28:05 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. Patch Set 12: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4820/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 12 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Aug 2019 05:28:14 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. Patch Set 12: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 12 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Tue, 20 Aug 2019 05:28:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8869: Fix handling of HTTP keep-alive when returning 401
Todd Lipcon has posted comments on this change. ( http://gerrit.cloudera.org:8080/14076 ) Change subject: IMPALA-8869: Fix handling of HTTP keep-alive when returning 401 .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14076 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3d5f80dbcde5b623a1d0586b5d763a062dd21afa Gerrit-Change-Number: 14076 Gerrit-PatchSet: 1 Gerrit-Owner: Thomas Tauber-Marshall Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Tim Armstrong Gerrit-Reviewer: Todd Lipcon Gerrit-Comment-Date: Tue, 20 Aug 2019 02:53:12 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14039 ) Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. Patch Set 7: (7 comments) http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/buffered-tuple-stream-test.cc File be/src/runtime/buffered-tuple-stream-test.cc: http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/buffered-tuple-stream-test.cc@323 PS7, Line 323: "-1" nit: use the name of the test instead. Same below. http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.h File be/src/runtime/spillable-row-batch-queue.h: http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.h@100 PS7, Line 100: BufferedTupleStream* batch_queue_; Shouldn't this use unique_ptr ? May be I am missing something but as the code stands now, aren't we leaking 'batch_queue_' ? http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.cc File be/src/runtime/spillable-row-batch-queue.cc: http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.cc@53 PS7, Line 53: batch_queue_ = new Use unique_ptr http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/service/query-options.cc File be/src/service/query-options.cc: http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/service/query-options.cc@907 PS7, Line 907: query_options->max_pinned_result_spooling_memory nits: code seems more legible if we factor out these two options into local variables: int64_t max_pinned_mem = ... int64_t max_unpinned_mem = ... http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/service/query-options.cc@921 PS7, Line 921: if (query_options->max_pinned_result_spooling_memory == 0 : && query_options->max_unpinned_result_spooling_memory != 0) May be I mis-read it but isn't this the same check as the one on line 907 ? http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/service/query-options.cc@928 PS7, Line 928: if (query_options->max_unpinned_result_spooling_memory != 0 : && query_options->max_unpinned_result_spooling_memory : < query_options->max_pinned_result_spooling_memory) Same question: isn't this the same as the one at line 912 ? http://gerrit.cloudera.org:8080/#/c/14039/7/common/thrift/ImpalaService.thrift File common/thrift/ImpalaService.thrift: http://gerrit.cloudera.org:8080/#/c/14039/7/common/thrift/ImpalaService.thrift@419 PS7, Line 419: MAX_PINNED_RESULT_SPOOLING_MEMORY = 86 Also, please specify the behavior when setting this to 0. Same for any value < 0. -- To view, visit http://gerrit.cloudera.org:8080/14039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 Gerrit-Change-Number: 14039 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Tue, 20 Aug 2019 00:49:51 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8858: Add observability of idle executor groups
Andrew Sherman has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add observability of idle executor groups .. Patch Set 1: This looks like the right approach. Would there be any value by having a test in cluster-membership-mgr-test.cc ? -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 19 Aug 2019 23:49:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/13748 ) Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution .. Patch Set 19: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4307/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/13748 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643 Gerrit-Change-Number: 13748 Gerrit-PatchSet: 19 Gerrit-Owner: radford nguyen Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Fredy Wijaya Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: radford nguyen Gerrit-Comment-Date: Mon, 19 Aug 2019 23:11:47 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
Hello Bharath Vissapragada, Andrew Sherman, Fredy Wijaya, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/13748
to look at the new patch set (#19).
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
IMPALA-8571[WIP]: improve QueryEventHook execution
(WIP because still need to implement polling of metrics by backend)
This commit hardens guarantees around QueryEventHook execution by
adding the following features:
*hook execution*
Query event hooks are still executed in a thread-pool, but you can
now configure the type of (Java) threads used to execute them via
the newly-added backend flag `query_event_hook_use_daemon_threads`,
which takes a true/false value.
See Java Thread.setDaemon(boolean) for what a "daemon thread" entails.
*hook timeout/cancellation*
A timeout for hook execution can be configured through the backend
flag `query_event_hook_timeout_s`, which specified a timeout value
in seconds. If a hook has not completed execution within this timeout
(measured from hook submission, not execution) then the hook task
will be cancelled in order to free up resources.
*hook rejection*
The hook execution engine now has a fixed-capacity work queue whose
capacity can be configured through the backend flag
`query_event_hook_queue_capacity`. This queue is used to store
hook tasks that are submitted when there are no free threads
available for hook execution. All hook tasks submitted when the
queue is at capacity will be rejected and logged without affecting
the result of the query.
*hook performance metrics*
The following hook metrics are captured:
*query-event-hook.${hook_method}.execution-rejections*
Counter indicating how many submitted tasks have been rejected
due to a full work queue
*query-event-hook.${hook_method}.execution-exceptions*
Counter indicating how many tasks have thrown an exception
during execution
*query-event-hook.${hook_method}.execution-timeouts*
Counter indicating how many tasks have been cancelled due to
not completing within {@code hookTimeout_s} of submission.
*query-event-hook.${hook_method}.execution-submissions*
Counter indicating the number of times ${hookClass}.${method}
has been submitted for execution.
*query-event-hook.${hook_method}.mean-execution-time*
Mean time in [ns] that ${hook_name} has taken to complete,
whether normally or by error (e.g. timeout or exception).
*query-event-hook.${hook_method}.mean-queued-time*
Mean time in [ns] between hook task submission
and hook task execution. This indicates how long a hook
task has been queued waiting to execute.
Testing:
- added unit tests for new features
- re-ran existing E2E tests
Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
---
M be/src/service/frontend.cc
M be/src/service/frontend.h
M be/src/service/impala-server.cc
M be/src/util/CMakeLists.txt
M be/src/util/backend-gflag-util.cc
A be/src/util/hook-metrics-test.cc
A be/src/util/hook-metrics.cc
A be/src/util/hook-metrics.h
M common/thrift/BackendGflags.thrift
M common/thrift/Frontend.thrift
M common/thrift/metrics.json
A fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
M fe/src/main/java/org/apache/impala/hooks/QueryEventHookManager.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
M fe/src/main/java/org/apache/impala/service/Frontend.java
M fe/src/main/java/org/apache/impala/service/JniFrontend.java
A
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
M fe/src/test/java/org/apache/impala/hooks/QueryEventHookManagerTest.java
M query-event-hook-api/src/main/java/org/apache/impala/hooks/QueryEventHook.java
19 files changed, 1,364 insertions(+), 114 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/48/13748/19
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 19
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
Patch Set 18:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/13748/18/be/src/util/hook-metrics.cc
File be/src/util/hook-metrics.cc:
http://gerrit.cloudera.org:8080/#/c/13748/18/be/src/util/hook-metrics.cc@82
PS18, Line 82: void QueryEventHookMetricContainer::Update(const
TQueryEventHookMetrics& from) {
Is this all I really have to do to make sure the metrics will be published?
--
To view, visit http://gerrit.cloudera.org:8080/13748
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibb88422f7cfe86947d11ce57d2b4c63e57d1b643
Gerrit-Change-Number: 13748
Gerrit-PatchSet: 18
Gerrit-Owner: radford nguyen
Gerrit-Reviewer: Andrew Sherman
Gerrit-Reviewer: Bharath Vissapragada
Gerrit-Reviewer: Fredy Wijaya
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: radford nguyen
Gerrit-Comment-Date: Mon, 19 Aug 2019 22:16:07 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8571[WIP]: improve QueryEventHook execution
radford nguyen has posted comments on this change. (
http://gerrit.cloudera.org:8080/13748 )
Change subject: IMPALA-8571[WIP]: improve QueryEventHook execution
..
Patch Set 18:
(10 comments)
Thanks for the feedback, asherman. Replies inline
http://gerrit.cloudera.org:8080/#/c/13748/18//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/13748/18//COMMIT_MSG@31
PS18, Line 31:
> Do you want to discuss --query_event_hook_use_daemon_threads here? How woul
Yes, this is something that should be included in the commit message
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java
File
fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java:
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@51
PS18, Line 51: * Rejections
> This javadoc is great, and just the sort of thing I was hoping to see.
I was not planning on publishing the html; I have just been in the habit of
using html because eclipse/Idea/etc will render it when displaying the javadoc
in a tooltip window. Everything just turns into one giant line in there if you
don't format.
Anyway, I'll switch to markdown, since it's both human-readable and can be
rendered by an IDE plugin or javadoc tool if needed.
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@102
PS18, Line 102: * ${hookClass}.${method}.execution.exceptions
> I think the metrics now have the "query-event-hook" prefix.
re: "query-event-hook" prefix
-
So this is where the split b/w backend/frontend metrics is kind of confusing.
This javadoc describes the metric names as they are collected in the frontend,
and unfortunately the names do not exactly align with the backend. Part of
this is due to the fact that the Java metrics are objects (e.g. Timers) that
can provide multiple metrics (e.g. mean time, count).
In passing the metrics to the backend, it seemed to me that the convention was
to pass the raw data (e.g. mean time, count) as needed, instead of say
translating the entire metric object to some similarly-capable object in the
backend.
The "query-event-hook" prefix only exists in the backend metric names.
re: hookClass vs. method
-
In this context, ${hookClass} is always the fully-qualified name of the java
class that implements `QueryEventHook`, and ${hookMethod} the method name
invoked on that hook. An example metric name would be:
org.foo.bar.MyQueryEventHook.onQueryComplete.execution.exceptions
I actually tried to make this very predictable and therefore amenable to
automation/generation, but if there is some way I can improve that further,
please let me know.
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@176
PS18, Line 176: // ArrayBlockingQueue contructor performs bounds-check on
queue size for us
> typo: constructor
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@184
PS18, Line 184: // this executor cancels any hook tasks that
> This may seem picky but in Imapla we like comments with Capitals at the beg
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@255
PS18, Line 255:* {@link ExecutionException}. This behavior is consistent
with how futures normally
> We could be more concise by not saying this about how Futures normally beha
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/main/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutor.java@286
PS18, Line 286: LOG.warn("QueryEventHook {}.{} execution rejected because
the " +
> I sometimes weaselly write "probably because" as you never know :-)
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java
File
fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java:
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@1
PS18, Line 1: /**
> Use // style comments for the apache license please
Done
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@80
PS18, Line 80: // make this longer to reasonably guarantee a timeout
> Is this a FIXME note, not sure I understand what this keans
Reworded as: "Sleep much longer than `hookTimeout` to reasonably guarantee a
timeout."
http://gerrit.cloudera.org:8080/#/c/13748/18/fe/src/test/java/org/apache/impala/hooks/FixedCapacityQueryHookExecutorTest.java@115
PS18, Line 115:
[Impala-ASF-CR] IMPALA-8845: Cancel receiver's streams on exchange node's EOS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14101 ) Change subject: IMPALA-8845: Cancel receiver's streams on exchange node's EOS .. Patch Set 2: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4306/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c805e9d63ed8af9f458bf71e8ef5ea9376b939 Gerrit-Change-Number: 14101 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Aug 2019 21:47:19 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8866: Bump versions of dependencies
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14083 ) Change subject: IMPALA-8866: Bump versions of dependencies .. IMPALA-8866: Bump versions of dependencies Bumped derby to next patch version, 10.14.2.0 Bumped jackson-databind to next patch version, 2.9.9.2 Bump jquery to 3.4.1, the most recent release. Testing: Manually inspected debug page, clicked through most of the screens to sanity check them. Ran exhaustive tests with Hive 2 and Hive 3 configs. Change-Id: Ia60d1ffec034790205404550c4bac54b247547d7 Reviewed-on: http://gerrit.cloudera.org:8080/14083 Reviewed-by: Impala Public Jenkins Tested-by: Impala Public Jenkins --- M .gitattributes M bin/rat_exclude_files.txt M fe/pom.xml M impala-parent/pom.xml M www/common-header.tmpl D www/jquery/jquery-1.12.4.min.js A www/jquery/jquery-3.4.1.min.js 7 files changed, 19 insertions(+), 8 deletions(-) Approvals: Impala Public Jenkins: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ia60d1ffec034790205404550c4bac54b247547d7 Gerrit-Change-Number: 14083 Gerrit-PatchSet: 4 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8866: Bump versions of dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14083 ) Change subject: IMPALA-8866: Bump versions of dependencies .. Patch Set 3: Verified+1 -- To view, visit http://gerrit.cloudera.org:8080/14083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia60d1ffec034790205404550c4bac54b247547d7 Gerrit-Change-Number: 14083 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 21:35:49 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Tim Armstrong has posted comments on this change. (
http://gerrit.cloudera.org:8080/14039 )
Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS
..
Patch Set 7:
(9 comments)
Nice! Had some relatively minor comments.
I think a few of them come from a philosophical thing about error handling - I
think I'd prefer if SpillableRowBatchQueue's invariants were stricter about
when it was valid to call different methods in the lifecycle - currently there
are a few runtime checks that would only be triggered if there was a bug in the
calling code.
There's a couple of reasons I prefer it this way (and why we've generally done
it this way in Impala). First, if the error-handling code isn't actually
reachable, then we can't really test that it works correctly. Second, too many
defensive checks add complexity and runtime overhead if the pattern is repeated
- and it does actually get confusing whether something is an invariant of the
system or a runtime error.
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/exec/exec-node.h
File be/src/exec/exec-node.h:
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/exec/exec-node.h@196
PS7, Line 196: const std::string label() const;
I think the const is unnecessary for a return by value (if you're returning by
value, the caller gets its own copy anyway, so can choose whether it wants to
mutate it).
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/buffered-tuple-stream.h
File be/src/runtime/buffered-tuple-stream.h:
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/buffered-tuple-stream.h@370
PS7, Line 370: int64_t BytesUnpinned() const {
I'd probably just call it bytes_unpinned() cause it's a plain accessor (there's
a bit of a grey area in naming conventions here, but plain accessors usually
have are lower case with underscores).
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.h
File be/src/runtime/spillable-row-batch-queue.h:
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.h@71
PS7, Line 71: /// successfully added, returns an error Status if the queue is
full, has already been
Why not just make it invalid to append to the queue when it's full? We could
then make it a DCHECK instead of a runtime check
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.h@79
PS7, Line 79: /// Returns and removes the RowBatch at the head of the queue.
Returns Status::OK() if
Same thing - if it's not valid to call GetBatch() on an empty queue, just make
it part of the contract and use a DCHECK instead of the runtime check.
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.cc
File be/src/runtime/spillable-row-batch-queue.cc:
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.cc@67
PS7, Line 67: if (UNLIKELY(closed_)) return Status("SpillableRowBatchQueue is
already closed.");
If it's a bug to call this when it's closed, let's just make it a DCHECK.
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.cc@112
PS7, Line 112: if (UNLIKELY(closed_)) return false;
I think it would be a bug to call this when it's closed, so we could document
it as a precondition and make it a DCHECK
http://gerrit.cloudera.org:8080/#/c/14039/7/be/src/runtime/spillable-row-batch-queue.cc@130
PS7, Line 130: if (batch_queue_ != nullptr)
Missing braces for multi-line if
http://gerrit.cloudera.org:8080/#/c/14039/7/common/thrift/ImpalaService.thrift
File common/thrift/ImpalaService.thrift:
http://gerrit.cloudera.org:8080/#/c/14039/7/common/thrift/ImpalaService.thrift@415
PS7, Line 415: // The maximum amount of pinned memory used when spooling
query results. If this value
It's probably better to avoid renumbering things in thrift files. I think it's
ok in this case since it's not really exposed in public APIs, but it's probably
a good habit to avoid doing it.
http://gerrit.cloudera.org:8080/#/c/14039/7/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java
File fe/src/main/java/org/apache/impala/planner/PlanRootSink.java:
http://gerrit.cloudera.org:8080/#/c/14039/7/fe/src/main/java/org/apache/impala/planner/PlanRootSink.java@88
PS7, Line 88: maxMemReservationBytes = Math.max(maxMemReservationBytes,
minMemReservationBytes);
Move this up to l74 so that the calculation is all in one place?
--
To view, visit http://gerrit.cloudera.org:8080/14039
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9
Gerrit-Change-Number: 14039
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim
[Impala-ASF-CR] IMPALA-8803: Coordinator should release admitted memory per-backend
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14104 ) Change subject: IMPALA-8803: Coordinator should release admitted memory per-backend .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4305/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I88bb11e0ede7574568020e0277dd8ac8d2586dc9 Gerrit-Change-Number: 14104 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Bikramjeet Vig Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 21:20:23 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8845: Cancel receiver's streams on exchange node's EOS
Michael Ho has uploaded a new patch set (#2). ( http://gerrit.cloudera.org:8080/14101 ) Change subject: IMPALA-8845: Cancel receiver's streams on exchange node's EOS .. IMPALA-8845: Cancel receiver's streams on exchange node's EOS When an exchange node reaches its row count limit, the current code will not notify the sender fragments about it. Consequently, sender fragments may keep sending row batches to the exchange node but they won't be dequeued anymore. The sender fragments may end up blocking in the RPC indefinitely until either the query is cancelled or closed. This change fixes the problem above by cancelling the underlying receiver's streams of an exchange node once it reaches the row count limit. This will unblock all senders whose TransmitData() RPCs haven't been replied to yet. Any future row batches sent to this receiver will also be immediately replied to with a response indicating that this receiver is already closed so the sender will stop sending any more row batches to it. Change-Id: I10c805e9d63ed8af9f458bf71e8ef5ea9376b939 --- M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/runtime/krpc-data-stream-recvr.cc M be/src/runtime/krpc-data-stream-recvr.h A tests/custom_cluster/test_exchange_eos.py 5 files changed, 91 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/14101/2 -- To view, visit http://gerrit.cloudera.org:8080/14101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I10c805e9d63ed8af9f458bf71e8ef5ea9376b939 Gerrit-Change-Number: 14101 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar
[Impala-ASF-CR] IMPALA-8845: Cancel receiver's streams on exchange node's EOS
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/14101 ) Change subject: IMPALA-8845: Cancel receiver's streams on exchange node's EOS .. Patch Set 2: (2 comments) http://gerrit.cloudera.org:8080/#/c/14101/1/tests/custom_cluster/test_exchange_eos.py File tests/custom_cluster/test_exchange_eos.py: http://gerrit.cloudera.org:8080/#/c/14101/1/tests/custom_cluster/test_exchange_eos.py@22 PS1, Line 22: from tests.verifiers.metric_verifier import MetricVerifier > flake8: F401 'tests.common.test_vector.ImpalaTestDimension' imported but un Done http://gerrit.cloudera.org:8080/#/c/14101/1/tests/custom_cluster/test_exchange_eos.py@25 PS1, Line 25: class TestExchangeEos(CustomClusterTestSuite): > flake8: E302 expected 2 blank lines, found 1 Done -- To view, visit http://gerrit.cloudera.org:8080/14101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c805e9d63ed8af9f458bf71e8ef5ea9376b939 Gerrit-Change-Number: 14101 Gerrit-PatchSet: 2 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Aug 2019 21:04:46 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8858: Add observability of idle executor groups
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14103 ) Change subject: IMPALA-8858: Add observability of idle executor groups .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4304/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig Gerrit-Reviewer: Andrew Sherman Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Comment-Date: Mon, 19 Aug 2019 20:54:54 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8803: Coordinator should release admitted memory per-backend
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14104 Change subject: IMPALA-8803: Coordinator should release admitted memory per-backend .. IMPALA-8803: Coordinator should release admitted memory per-backend Changes the Coordinator to release admitted memory when each Backend completes, rather than waiting for the entire query to complete before releasing admitted memory. When the Coordinator detects that a Backend has completed (via ControlService::ReportExecStatus) it updates the state of the Backend in Coordinator::BackendResourceState. BackendResourceState tracks the state of the admitted resources for each Backend and decides when the resources for a group of Backend states should be released. BackendResourceState defines a state machine to help coordinate the state of the admitted memory for each Backend. It guarantees that by the time the query is shutdown, all Backends release their admitted memory. BackendResourceState implements three rules to control the rate at which the Coordinator releases admitted memory from the AdmissionController: * Resources are released at most every 1 seconds, this prevents short lived queries from causing high load on the admission controller * Resources are released at most O(log(num_backends)) times; the BackendResourceStates can release multiple BackendStates from the AdmissionController at a time * All pending resources are released if the only remaining Backend is the Coordinator Backend; this is useful for result spooling where all Backends may complete, except for the Coordinator Backend Testing: * Ran core tests * Added new tests to test_result_spooling.py and test_admission_controller.py Change-Id: I88bb11e0ede7574568020e0277dd8ac8d2586dc9 --- M be/src/runtime/CMakeLists.txt A be/src/runtime/coordinator-backend-state-test.cc M be/src/runtime/coordinator-backend-state.cc M be/src/runtime/coordinator-backend-state.h M be/src/runtime/coordinator.cc M be/src/runtime/coordinator.h M be/src/scheduling/admission-controller-test.cc M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M tests/custom_cluster/test_admission_controller.py M tests/query_test/test_result_spooling.py A tests/util/web_pages_util.py 12 files changed, 862 insertions(+), 68 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/04/14104/1 -- To view, visit http://gerrit.cloudera.org:8080/14104 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I88bb11e0ede7574568020e0277dd8ac8d2586dc9 Gerrit-Change-Number: 14104 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. Patch Set 11: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4303/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 11 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Aug 2019 20:46:58 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8858: Add observability of idle executor groups
Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14103 Change subject: IMPALA-8858: Add observability of idle executor groups .. IMPALA-8858: Add observability of idle executor groups This patch adds a metric that displays a comma seperated list of executor group names that are in a healthy state and are idle according to the coordinator (no queries admitted locally by the coordinator are running on them). It gets updated when either the cluster membership changes or a query gets admitted or released by the admission controller. Testing: Added a custom cluster test. Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 --- M be/src/scheduling/admission-controller.cc M be/src/scheduling/admission-controller.h M be/src/scheduling/cluster-membership-mgr.cc M be/src/scheduling/cluster-membership-mgr.h M common/thrift/metrics.json M tests/custom_cluster/test_executor_groups.py 6 files changed, 151 insertions(+), 21 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/03/14103/1 -- To view, visit http://gerrit.cloudera.org:8080/14103 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I58cde8699c33af8b87273437e9d8bf6371a34539 Gerrit-Change-Number: 14103 Gerrit-PatchSet: 1 Gerrit-Owner: Bikramjeet Vig
[Impala-ASF-CR] IMPALA-7935: Disable /catalog object in local catalog mode.
Anurag Mantripragada has posted comments on this change. ( http://gerrit.cloudera.org:8080/12443 ) Change subject: IMPALA-7935: Disable /catalog_object in local catalog mode. .. Patch Set 11: Missed a test that checks for catalog object page which doesn't exist after this change. -- To view, visit http://gerrit.cloudera.org:8080/12443 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia04797b32964c2edaa2e860dcf510d6f9cccd81c Gerrit-Change-Number: 12443 Gerrit-PatchSet: 11 Gerrit-Owner: Anurag Mantripragada Gerrit-Reviewer: Anurag Mantripragada Gerrit-Reviewer: Bharath Vissapragada Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Aug 2019 20:06:33 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14039 ) Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. Patch Set 7: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4302/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 Gerrit-Change-Number: 14039 Gerrit-PatchSet: 7 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 19:35:08 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14039 ) Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. Patch Set 6: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4301/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 Gerrit-Change-Number: 14039 Gerrit-PatchSet: 6 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 19:24:52 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Sahil Takiar has posted comments on this change. (
http://gerrit.cloudera.org:8080/14039 )
Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS
..
Patch Set 4:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/exec/buffered-plan-root-sink.h
File be/src/exec/buffered-plan-root-sink.h:
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/exec/buffered-plan-root-sink.h@49
PS4, Line 49: Intializes
> typo
Done
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/exec/buffered-plan-root-sink.cc
File be/src/exec/buffered-plan-root-sink.cc:
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/exec/buffered-plan-root-sink.cc@19
PS4, Line 19: #include "runtime/deque-row-batch-queue.h"
> Can be deleted ?
Done
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/exec/exec-node.cc
File be/src/exec/exec-node.cc:
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/exec/exec-node.cc@373
PS4, Line 373: node_type_name + " (id=" + std::to_string(id_) + ")"
> Use substitute for better legibility.
Done
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/buffered-tuple-stream.h
File be/src/runtime/buffered-tuple-stream.h:
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/buffered-tuple-stream.h@223
PS4, Line 223: const std::string
> Did you mean to pass-by-value so the compiler will do copy-elison ?
Changed to pass-by-reference to avoid an additional copy, but I think there is
still a copy happening during the assignment:
caller_label_ = caller_label
Not sure if there is a way to avoid that?
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/buffered-tuple-stream.cc
File be/src/runtime/buffered-tuple-stream.cc:
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/buffered-tuple-stream.cc@212
PS4, Line 212: string caller_label
> Doesn't match the signature in header
Done
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/deque-row-batch-queue.h
File be/src/runtime/deque-row-batch-queue.h:
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/deque-row-batch-queue.h@33
PS4, Line 33: class DequeRowBatchQueue {
> Can this class and files be deleted after this patch ?
Done
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/sorter.h
File be/src/runtime/sorter.h:
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/sorter.h@210
PS4, Line 210: //
> ///
Done
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/spillable-row-batch-queue.h
File be/src/runtime/spillable-row-batch-queue.h:
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/spillable-row-batch-queue.h@57
PS4, Line 57: std::string
> nit: std::string&
Done
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/spillable-row-batch-queue.h@107
PS4, Line 107: std::string
> nit: std::string& name
Done
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/spillable-row-batch-queue.cc
File be/src/runtime/spillable-row-batch-queue.cc:
http://gerrit.cloudera.org:8080/#/c/14039/4/be/src/runtime/spillable-row-batch-queue.cc@101
PS4, Line 101: DCHECK((eos && IsEmpty()) || (!eos && !IsEmpty()));
> DCHECK_EQ(eos, IsEmpty());
Done
--
To view, visit http://gerrit.cloudera.org:8080/14039
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9
Gerrit-Change-Number: 14039
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho
Gerrit-Reviewer: Sahil Takiar
Gerrit-Reviewer: Tim Armstrong
Gerrit-Comment-Date: Mon, 19 Aug 2019 18:54:45 +
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14039 to look at the new patch set (#7). Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. IMPALA-8818: Replace deque with spillable queue in BufferedPRS Replaces DequeRowBatchQueue with SpillableRowBatchQueue in BufferedPlanRootSink. A few changes to BufferedPlanRootSink were necessary for it to work with the spillable queue, however, all the synchronization logic is the same. SpillableRowBatchQueue is a wrapper around a BufferedTupleStream and a ReservationManager. It takes in a TBackendResourceProfile that specifies the max / min memory reservation the BufferedTupleStream can use to buffer rows. The 'max_unpinned_bytes' parameter limits the max number of bytes that can be unpinned in the BufferedTupleStream. The limit is a 'soft' limit because calls to AddBatch may push the amount of unpinned memory over the limit. The queue is non-blocking and not thread safe. It provides AddBatch and GetBatch methods. Calls to AddBatch spill if the BufferedTupleStream does not have enough reservation to fit the entire RowBatch. Adds two new query options: 'MAX_PINNED_RESULT_SPOOLING_MEMORY' and 'MAX_UNPINNED_RESULT_SPOOLING_MEMORY', which bound the amount of pinned and unpinned memory that a query can use for spooling, respectively. MAX_PINNED_RESULT_SPOOLING_MEMORY must be <= MAX_UNPINNED_RESULT_SPOOLING_MEMORY in order to allow all the pinned data in the BufferedTupleStream to be unpinned. This is enforced in a new method in QueryOptions called 'ValidateQueryOptions'. Planner Changes: PlanRootSink.java now computes a full ResourceProfile if result spooling is enabled. The min mem reservation is bounded by the size of the read and write pages used by the BufferedTupleStream. The max mem reservation is bounded by 'MAX_PINNED_RESULT_SPOOLING_MEMORY'. The mem estimate is computed by estimating the size of the result set using stats. BufferedTupleStream Re-Factoring: For the most part, using a BufferedTupleStream outside an ExecNode works properly. However, some changes were necessary: * The message for the MAX_ROW_SIZE error is ExecNode specific. In order to fix this, this patch introduces the concept of an ExecNode 'label' which is a more generic version of an ExecNode 'id'. * The definition of TBackendResourceProfile lived in PlanNodes.thrift, it was moved to its own file so it can be used by DataSinks.thrift. * Modified BufferedTupleStream so it internally tracks how many bytes are unpinned (necessary for 'MAX_UNPINNED_RESULT_SPOOLING_MEMORY'). Metrics: * Added a few of the metrics mentioned in IMPALA-8825 to BufferedPlanRootSink. Specifically, added timers to track how much time is spent waiting in the BufferedPlanRootSink 'Send' and 'GetNext' methods. * The BufferedTupleStream in the SpillableRowBatchQueue exposes several BufferPool metrics such as number of reserved and unpinned bytes. Bug Fixes: * Fixed a bug in BufferedPlanRootSink where the MemPool used by the expression evaluators was not being cleared incrementally. * Fixed a bug where the inactive timer was not being properly updated in BufferedPlanRootSink. * Fixed a bug where RowBatch memory was not freed if BufferedPlanRootSink::GetNext terminated early because it could not handle requests where num_results < BATCH_SIZE. Testing: * Added new tests to test_result_spooling.py. * Updated errors thrown in spilling-large-rows.test. * Ran exhaustive tests. Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/exec/analytic-eval-node.cc M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/data-sink.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/plan-root-sink.h M be/src/exec/sort-node.cc M be/src/runtime/CMakeLists.txt M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h D be/src/runtime/deque-row-batch-queue.cc D be/src/runtime/deque-row-batch-queue.h M be/src/runtime/sorter.cc M be/src/runtime/sorter.h A be/src/runtime/spillable-row-batch-queue.cc A be/src/runtime/spillable-row-batch-queue.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-server.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CMakeLists.txt M common/thrift/DataSinks.thrift M common/thrift/ImpalaInternalService.thrift M
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14039 to look at the new patch set (#6). Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. IMPALA-8818: Replace deque with spillable queue in BufferedPRS Replaces DequeRowBatchQueue with SpillableRowBatchQueue in BufferedPlanRootSink. A few changes to BufferedPlanRootSink were necessary for it to work with the spillable queue, however, all the synchronization logic is the same. SpillableRowBatchQueue is a wrapper around a BufferedTupleStream and a ReservationManager. It takes in a TBackendResourceProfile that specifies the max / min memory reservation the BufferedTupleStream can use to buffer rows. The 'max_unpinned_bytes' parameter limits the max number of bytes that can be unpinned in the BufferedTupleStream. The limit is a 'soft' limit because calls to AddBatch may push the amount of unpinned memory over the limit. The queue is non-blocking and not thread safe. It provides AddBatch and GetBatch methods. Calls to AddBatch spill if the BufferedTupleStream does not have enough reservation to fit the entire RowBatch. Adds two new query options: 'MAX_PINNED_RESULT_SPOOLING_MEMORY' and 'MAX_UNPINNED_RESULT_SPOOLING_MEMORY', which bound the amount of pinned and unpinned memory that a query can use for spooling, respectively. MAX_PINNED_RESULT_SPOOLING_MEMORY must be <= MAX_UNPINNED_RESULT_SPOOLING_MEMORY in order to allow all the pinned data in the BufferedTupleStream to be unpinned. This is enforced in a new method in QueryOptions called 'ValidateQueryOptions'. Planner Changes: PlanRootSink.java now computes a full ResourceProfile if result spooling is enabled. The min mem reservation is bounded by the size of the read and write pages used by the BufferedTupleStream. The max mem reservation is bounded by 'MAX_PINNED_RESULT_SPOOLING_MEMORY'. The mem estimate is computed by estimating the size of the result set using stats. BufferedTupleStream Re-Factoring: For the most part, using a BufferedTupleStream outside an ExecNode works properly. However, some changes were necessary: * The message for the MAX_ROW_SIZE error is ExecNode specific. In order to fix this, this patch introduces the concept of an ExecNode 'label' which is a more generic version of an ExecNode 'id'. * The definition of TBackendResourceProfile lived in PlanNodes.thrift, it was moved to its own file so it can be used by DataSinks.thrift. * Modified BufferedTupleStream so it internally tracks how many bytes are unpinned (necessary for 'MAX_UNPINNED_RESULT_SPOOLING_MEMORY'). Metrics: * Added a few of the metrics mentioned in IMPALA-8825 to BufferedPlanRootSink. Specifically, added timers to track how much time is spent waiting in the BufferedPlanRootSink 'Send' and 'GetNext' methods. * The BufferedTupleStream in the SpillableRowBatchQueue exposes several BufferPool metrics such as number of reserved and unpinned bytes. Bug Fixes: * Fixed a bug in BufferedPlanRootSink where the MemPool used by the expression evaluators was not being cleared incrementally. * Fixed a bug where the inactive timer was not being properly updated in BufferedPlanRootSink. * Fixed a bug where RowBatch memory was not freed if BufferedPlanRootSink::GetNext terminated early because it could not handle requests where num_results < BATCH_SIZE. Testing: * Added new tests to test_result_spooling.py. * Updated errors thrown in spilling-large-rows.test. * Ran exhaustive tests. Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/exec/analytic-eval-node.cc M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/data-sink.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/plan-root-sink.h M be/src/exec/sort-node.cc M be/src/runtime/CMakeLists.txt M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h M be/src/runtime/sorter.cc M be/src/runtime/sorter.h A be/src/runtime/spillable-row-batch-queue.cc A be/src/runtime/spillable-row-batch-queue.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-server.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CMakeLists.txt M common/thrift/DataSinks.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift A
[Impala-ASF-CR] IMPALA-8845: Cancel receiver's streams on exchange node's EOS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14101 ) Change subject: IMPALA-8845: Cancel receiver's streams on exchange node's EOS .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4300/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c805e9d63ed8af9f458bf71e8ef5ea9376b939 Gerrit-Change-Number: 14101 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Aug 2019 18:37:13 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14039 ) Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. Patch Set 5: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4299/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14039 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 Gerrit-Change-Number: 14039 Gerrit-PatchSet: 5 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Michael Ho Gerrit-Reviewer: Sahil Takiar Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 18:27:17 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8845: Cancel receiver's streams on exchange node's EOS
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14101 ) Change subject: IMPALA-8845: Cancel receiver's streams on exchange node's EOS .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14101/1/tests/custom_cluster/test_exchange_eos.py File tests/custom_cluster/test_exchange_eos.py: http://gerrit.cloudera.org:8080/#/c/14101/1/tests/custom_cluster/test_exchange_eos.py@22 PS1, Line 22: from tests.common.test_vector import ImpalaTestDimension flake8: F401 'tests.common.test_vector.ImpalaTestDimension' imported but unused http://gerrit.cloudera.org:8080/#/c/14101/1/tests/custom_cluster/test_exchange_eos.py@25 PS1, Line 25: class TestExchangeEos(CustomClusterTestSuite): flake8: E302 expected 2 blank lines, found 1 -- To view, visit http://gerrit.cloudera.org:8080/14101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I10c805e9d63ed8af9f458bf71e8ef5ea9376b939 Gerrit-Change-Number: 14101 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Aug 2019 18:11:56 + Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8845: Cancel receiver's streams on exchange node's EOS
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14101 Change subject: IMPALA-8845: Cancel receiver's streams on exchange node's EOS .. IMPALA-8845: Cancel receiver's streams on exchange node's EOS When an exchange node reaches its row count limit, the current code will not notify the sender fragments about it. Consequently, sender fragments may keep sending row batches to the exchange node but they won't be dequeued anymore. The sender fragments may end up blocking in the RPC indefinitely until either the query is cancelled or closed. This change fixes the problem above by cancelling the underlying receiver's streams of an exchange node once it reaches the row count limit. This will unblock all senders whose TransmitData() RPCs haven't been replied to yet. Any future row batches sent to this receiver will also be immediately replied to with a response indicating that this receiver is already closed so the sender will stop sending any more row batches to it. Change-Id: I10c805e9d63ed8af9f458bf71e8ef5ea9376b939 --- M be/src/exec/exchange-node.cc M be/src/exec/exchange-node.h M be/src/runtime/krpc-data-stream-recvr.cc M be/src/runtime/krpc-data-stream-recvr.h A tests/custom_cluster/test_exchange_eos.py 5 files changed, 91 insertions(+), 12 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/01/14101/1 -- To view, visit http://gerrit.cloudera.org:8080/14101 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I10c805e9d63ed8af9f458bf71e8ef5ea9376b939 Gerrit-Change-Number: 14101 Gerrit-PatchSet: 1 Gerrit-Owner: Michael Ho
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. [DOCS] A section on 3.3 Fixed Issues Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Reviewed-on: http://gerrit.cloudera.org:8080/14098 Reviewed-by: Quanlong Huang Tested-by: Impala Public Jenkins --- M docs/topics/impala_fixed_issues.xml 1 file changed, 8 insertions(+), 1 deletion(-) Approvals: Quanlong Huang: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. Patch Set 3: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/463/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 19 Aug 2019 17:51:12 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. Patch Set 2: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/462/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 19 Aug 2019 17:50:37 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 19 Aug 2019 17:48:04 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Hello Michael Ho, Tim Armstrong, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14039 to look at the new patch set (#5). Change subject: IMPALA-8818: Replace deque with spillable queue in BufferedPRS .. IMPALA-8818: Replace deque with spillable queue in BufferedPRS Replaces DequeRowBatchQueue with SpillableRowBatchQueue in BufferedPlanRootSink. A few changes to BufferedPlanRootSink were necessary for it to work with the spillable queue, however, all the synchronization logic is the same. SpillableRowBatchQueue is a wrapper around a BufferedTupleStream and a ReservationManager. It takes in a TBackendResourceProfile that specifies the max / min memory reservation the BufferedTupleStream can use to buffer rows. The 'max_unpinned_bytes' parameter limits the max number of bytes that can be unpinned in the BufferedTupleStream. The limit is a 'soft' limit because calls to AddBatch may push the amount of unpinned memory over the limit. The queue is non-blocking and not thread safe. It provides AddBatch and GetBatch methods. Calls to AddBatch spill if the BufferedTupleStream does not have enough reservation to fit the entire RowBatch. Adds two new query options: 'MAX_PINNED_RESULT_SPOOLING_MEMORY' and 'MAX_UNPINNED_RESULT_SPOOLING_MEMORY', which bound the amount of pinned and unpinned memory that a query can use for spooling, respectively. MAX_PINNED_RESULT_SPOOLING_MEMORY must be <= MAX_UNPINNED_RESULT_SPOOLING_MEMORY in order to allow all the pinned data in the BufferedTupleStream to be unpinned. This is enforced in a new method in QueryOptions called 'ValidateQueryOptions'. Planner Changes: PlanRootSink.java now computes a full ResourceProfile if result spooling is enabled. The min mem reservation is bounded by the size of the read and write pages used by the BufferedTupleStream. The max mem reservation is bounded by 'MAX_PINNED_RESULT_SPOOLING_MEMORY'. The mem estimate is computed by estimating the size of the result set using stats. BufferedTupleStream Re-Factoring: For the most part, using a BufferedTupleStream outside an ExecNode works properly. However, some changes were necessary: * The message for the MAX_ROW_SIZE error is ExecNode specific. In order to fix this, this patch introduces the concept of an ExecNode 'label' which is a more generic version of an ExecNode 'id'. * The definition of TBackendResourceProfile lived in PlanNodes.thrift, it was moved to its own file so it can be used by DataSinks.thrift. * Modified BufferedTupleStream so it internally tracks how many bytes are unpinned (necessary for 'MAX_UNPINNED_RESULT_SPOOLING_MEMORY'). Metrics: * Added a few of the metrics mentioned in IMPALA-8825 to BufferedPlanRootSink. Specifically, added timers to track how much time is spent waiting in the BufferedPlanRootSink 'Send' and 'GetNext' methods. * The BufferedTupleStream in the SpillableRowBatchQueue exposes several BufferPool metrics such as number of reserved and unpinned bytes. Bug Fixes: * Fixed a bug in BufferedPlanRootSink where the MemPool used by the expression evaluators was not being cleared incrementally. * Fixed a bug where the inactive timer was not being properly updated in BufferedPlanRootSink. * Fixed a bug where RowBatch memory was not freed if BufferedPlanRootSink::GetNext terminated early because it could not handle requests where num_results < BATCH_SIZE. Testing: * Added new tests to test_result_spooling.py. * Updated errors thrown in spilling-large-rows.test. * Ran exhaustive tests. Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9 --- M be/generated-sources/gen-cpp/CMakeLists.txt M be/src/exec/analytic-eval-node.cc M be/src/exec/blocking-plan-root-sink.cc M be/src/exec/blocking-plan-root-sink.h M be/src/exec/buffered-plan-root-sink.cc M be/src/exec/buffered-plan-root-sink.h M be/src/exec/data-sink.cc M be/src/exec/exec-node.cc M be/src/exec/exec-node.h M be/src/exec/grouping-aggregator-partition.cc M be/src/exec/grouping-aggregator.cc M be/src/exec/partial-sort-node.cc M be/src/exec/partitioned-hash-join-builder.cc M be/src/exec/partitioned-hash-join-builder.h M be/src/exec/partitioned-hash-join-node.cc M be/src/exec/plan-root-sink.h M be/src/exec/sort-node.cc M be/src/runtime/CMakeLists.txt M be/src/runtime/buffered-tuple-stream-test.cc M be/src/runtime/buffered-tuple-stream.cc M be/src/runtime/buffered-tuple-stream.h M be/src/runtime/sorter.cc M be/src/runtime/sorter.h A be/src/runtime/spillable-row-batch-queue.cc A be/src/runtime/spillable-row-batch-queue.h M be/src/service/impala-beeswax-server.cc M be/src/service/impala-server.cc M be/src/service/query-options-test.cc M be/src/service/query-options.cc M be/src/service/query-options.h M common/thrift/CMakeLists.txt M common/thrift/DataSinks.thrift M common/thrift/ImpalaInternalService.thrift M common/thrift/ImpalaService.thrift M common/thrift/PlanNodes.thrift A
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. Patch Set 3: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/463/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 19 Aug 2019 17:45:38 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14098 to look at the new patch set (#3). Change subject: [DOCS] A section on 3.3 Fixed Issues .. [DOCS] A section on 3.3 Fixed Issues Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 --- M docs/impala_keydefs.ditamap M docs/topics/impala_fixed_issues.xml 2 files changed, 9 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14098/3 -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 3 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14098/1/docs/topics/impala_fixed_issues.xml File docs/topics/impala_fixed_issues.xml: http://gerrit.cloudera.org:8080/#/c/14098/1/docs/topics/impala_fixed_issues.xml@48 PS1, Line 48: 3.1.0 > Should this be 3.3.0? Done http://gerrit.cloudera.org:8080/#/c/14098/1/docs/topics/impala_fixed_issues.xml@53 PS1, Line 53: impala31 > Should this be impala33? Done -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 19 Aug 2019 17:44:03 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Hello Quanlong Huang, Impala Public Jenkins, I'd like you to reexamine a change. Please visit http://gerrit.cloudera.org:8080/14098 to look at the new patch set (#2). Change subject: [DOCS] A section on 3.3 Fixed Issues .. [DOCS] A section on 3.3 Fixed Issues Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 --- M docs/impala_keydefs.ditamap M docs/topics/impala_fixed_issues.xml 2 files changed, 9 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14098/2 -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newpatchset Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. Patch Set 2: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/462/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 2 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 19 Aug 2019 17:43:43 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. Patch Set 1: (2 comments) http://gerrit.cloudera.org:8080/#/c/14098/1/docs/topics/impala_fixed_issues.xml File docs/topics/impala_fixed_issues.xml: http://gerrit.cloudera.org:8080/#/c/14098/1/docs/topics/impala_fixed_issues.xml@48 PS1, Line 48: 3.1.0 Should this be 3.3.0? http://gerrit.cloudera.org:8080/#/c/14098/1/docs/topics/impala_fixed_issues.xml@53 PS1, Line 53: impala31 Should this be impala33? -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 19 Aug 2019 17:39:59 + Gerrit-HasComments: Yes
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/461/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Comment-Date: Mon, 19 Aug 2019 17:34:21 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8866: Bump versions of dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14083 ) Change subject: IMPALA-8866: Bump versions of dependencies .. Patch Set 3: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4818/ DRY_RUN=false -- To view, visit http://gerrit.cloudera.org:8080/14083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia60d1ffec034790205404550c4bac54b247547d7 Gerrit-Change-Number: 14083 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 17:30:22 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8866: Bump versions of dependencies
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14083 ) Change subject: IMPALA-8866: Bump versions of dependencies .. Patch Set 3: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia60d1ffec034790205404550c4bac54b247547d7 Gerrit-Change-Number: 14083 Gerrit-PatchSet: 3 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 17:30:21 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14098 Change subject: [DOCS] A section on 3.3 Fixed Issues .. [DOCS] A section on 3.3 Fixed Issues Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 --- M docs/impala_keydefs.ditamap M docs/topics/impala_fixed_issues.xml 2 files changed, 9 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/98/14098/1 -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni
[Impala-ASF-CR] [DOCS] A section on 3.3 Fixed Issues
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14098 ) Change subject: [DOCS] A section on 3.3 Fixed Issues .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/461/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14098 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I3c24122138c6799f592f7b30ecdb6cf0fc5ee180 Gerrit-Change-Number: 14098 Gerrit-PatchSet: 1 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Aug 2019 17:28:13 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Update 3.3 release notes
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14097 ) Change subject: [DOCS] Update 3.3 release notes .. [DOCS] Update 3.3 release notes Include parquet/lz4 in the new features list. Change-Id: Ib99dc8031ff38634beccde51d07cfb08c00724e1 Reviewed-on: http://gerrit.cloudera.org:8080/14097 Reviewed-by: Alex Rodoni Tested-by: Impala Public Jenkins --- M docs/topics/impala_new_features.xml 1 file changed, 5 insertions(+), 0 deletions(-) Approvals: Alex Rodoni: Looks good to me, approved Impala Public Jenkins: Verified -- To view, visit http://gerrit.cloudera.org:8080/14097 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: Ib99dc8031ff38634beccde51d07cfb08c00724e1 Gerrit-Change-Number: 14097 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] WIP: IMPALA-7027: fix StringLiteral.localEquals()
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14096 ) Change subject: WIP: IMPALA-7027: fix StringLiteral.localEquals() .. Patch Set 1: Build Successful https://jenkins.impala.io/job/gerrit-code-review-checks/4298/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests. -- To view, visit http://gerrit.cloudera.org:8080/14096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I6a9dff514c2c4cc422343d1dfbd881917acca138 Gerrit-Change-Number: 14096 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Comment-Date: Mon, 19 Aug 2019 17:27:45 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8866: Bump versions of dependencies
Lars Volker has posted comments on this change. ( http://gerrit.cloudera.org:8080/14083 ) Change subject: IMPALA-8866: Bump versions of dependencies .. Patch Set 2: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia60d1ffec034790205404550c4bac54b247547d7 Gerrit-Change-Number: 14083 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 17:25:44 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Update 3.3 release notes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14097 ) Change subject: [DOCS] Update 3.3 release notes .. Patch Set 1: Verified+1 Build Successful https://jenkins.impala.io/job/gerrit-docs-auto-test/460/ : Doc tests passed. -- To view, visit http://gerrit.cloudera.org:8080/14097 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib99dc8031ff38634beccde51d07cfb08c00724e1 Gerrit-Change-Number: 14097 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 17:22:19 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Update 3.3 release notes
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/14097 ) Change subject: [DOCS] Update 3.3 release notes .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14097 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib99dc8031ff38634beccde51d07cfb08c00724e1 Gerrit-Change-Number: 14097 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 17:21:12 + Gerrit-HasComments: No
[Impala-ASF-CR] [DOCS] Update 3.3 release notes
Abhishek Rawat has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14097 Change subject: [DOCS] Update 3.3 release notes .. [DOCS] Update 3.3 release notes Include parquet/lz4 in the new features list. Change-Id: Ib99dc8031ff38634beccde51d07cfb08c00724e1 --- M docs/topics/impala_new_features.xml 1 file changed, 5 insertions(+), 0 deletions(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/14097/1 -- To view, visit http://gerrit.cloudera.org:8080/14097 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: Ib99dc8031ff38634beccde51d07cfb08c00724e1 Gerrit-Change-Number: 14097 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] [DOCS] Update 3.3 release notes
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14097 ) Change subject: [DOCS] Update 3.3 release notes .. Patch Set 1: Build Started https://jenkins.impala.io/job/gerrit-docs-auto-test/460/ Testing docs change - this change appears to modify docs/ and no code. This is experimental - please report any issues to tarmstr...@cloudera.com or on this JIRA: IMPALA-7317 -- To view, visit http://gerrit.cloudera.org:8080/14097 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ib99dc8031ff38634beccde51d07cfb08c00724e1 Gerrit-Change-Number: 14097 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 17:12:53 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8866: Bump versions of dependencies
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14083 ) Change subject: IMPALA-8866: Bump versions of dependencies .. Patch Set 2: Could you take another look? Should be super-quick. -- To view, visit http://gerrit.cloudera.org:8080/14083 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: Ia60d1ffec034790205404550c4bac54b247547d7 Gerrit-Change-Number: 14083 Gerrit-PatchSet: 2 Gerrit-Owner: Tim Armstrong Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Lars Volker Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 16:56:04 + Gerrit-HasComments: No
[Impala-ASF-CR] WIP: IMPALA-7027: fix StringLiteral.localEquals()
Tim Armstrong has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14096 Change subject: WIP: IMPALA-7027: fix StringLiteral.localEquals() .. WIP: IMPALA-7027: fix StringLiteral.localEquals() String literals with the same value but different types are not interchangeable. localEquals() should check the type, similar to a cast expression. Testing: TODO - Add some targeted tests with various string literals in select list, including CHAR and VARCHAR - Run exhaustive tests (need to make sure this doesn't regress anything) Change-Id: I6a9dff514c2c4cc422343d1dfbd881917acca138 --- M fe/src/main/java/org/apache/impala/analysis/StringLiteral.java 1 file changed, 2 insertions(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/14096/1 -- To view, visit http://gerrit.cloudera.org:8080/14096 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: newchange Gerrit-Change-Id: I6a9dff514c2c4cc422343d1dfbd881917acca138 Gerrit-Change-Number: 14096 Gerrit-PatchSet: 1 Gerrit-Owner: Tim Armstrong
[Impala-ASF-CR](2.x) IMPALA-8871: Upgrade Thrift version in fe
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14094 ) Change subject: IMPALA-8871: Upgrade Thrift version in fe .. Patch Set 1: Verified-1 Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/4817/ -- To view, visit http://gerrit.cloudera.org:8080/14094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: If62e6330f0a725ebd4213c200656fc12a0eccf06 Gerrit-Change-Number: 14094 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Aug 2019 16:41:23 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-8871: Upgrade Thrift version in fe
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14094 ) Change subject: IMPALA-8871: Upgrade Thrift version in fe .. Patch Set 1: Build Failed https://jenkins.impala.io/job/gerrit-code-review-checks/4297/ : Initial code review checks failed. See linked job for details on the failure. -- To view, visit http://gerrit.cloudera.org:8080/14094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: If62e6330f0a725ebd4213c200656fc12a0eccf06 Gerrit-Change-Number: 14094 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Aug 2019 16:11:46 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-8871: Upgrade Thrift version in fe
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14094 ) Change subject: IMPALA-8871: Upgrade Thrift version in fe .. Patch Set 1: Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/4817/ DRY_RUN=true -- To view, visit http://gerrit.cloudera.org:8080/14094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: If62e6330f0a725ebd4213c200656fc12a0eccf06 Gerrit-Change-Number: 14094 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Aug 2019 16:08:23 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-8871: Upgrade Thrift version in fe
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14094 ) Change subject: IMPALA-8871: Upgrade Thrift version in fe .. Patch Set 1: Should have uploaded this as a DRAFT, still testing this. -- To view, visit http://gerrit.cloudera.org:8080/14094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: comment Gerrit-Change-Id: If62e6330f0a725ebd4213c200656fc12a0eccf06 Gerrit-Change-Number: 14094 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Sahil Takiar Gerrit-Comment-Date: Mon, 19 Aug 2019 15:54:30 + Gerrit-HasComments: No
[Impala-ASF-CR](2.x) IMPALA-8871: Upgrade Thrift version in fe
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14094 Change subject: IMPALA-8871: Upgrade Thrift version in fe .. IMPALA-8871: Upgrade Thrift version in fe Upgrade fe/ Thrift version to 0.9.3-1. Testing: * Ran exhaustive tests. * Ran perf-AB-test, which showed no significant perf difference. Change-Id: If62e6330f0a725ebd4213c200656fc12a0eccf06 --- M bin/impala-config.sh 1 file changed, 1 insertion(+), 1 deletion(-) git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/94/14094/1 -- To view, visit http://gerrit.cloudera.org:8080/14094 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: 2.x Gerrit-MessageType: newchange Gerrit-Change-Id: If62e6330f0a725ebd4213c200656fc12a0eccf06 Gerrit-Change-Number: 14094 Gerrit-PatchSet: 1 Gerrit-Owner: Sahil Takiar
[Impala-ASF-CR] IMPALA-8683: Document parquet/lz4 compression codec
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14093 ) Change subject: IMPALA-8683: Document parquet/lz4 compression codec .. IMPALA-8683: Document parquet/lz4 compression codec Document lz4 compression codec for parquet. Change-Id: I304274842c8494a021816d68bc5d81810c353146 Reviewed-on: http://gerrit.cloudera.org:8080/14093 Tested-by: Impala Public Jenkins Reviewed-by: Quanlong Huang --- M docs/topics/impala_compression_codec.xml M docs/topics/impala_file_formats.xml M docs/topics/impala_parquet.xml 3 files changed, 15 insertions(+), 6 deletions(-) Approvals: Impala Public Jenkins: Verified Quanlong Huang: Looks good to me, approved -- To view, visit http://gerrit.cloudera.org:8080/14093 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I304274842c8494a021816d68bc5d81810c353146 Gerrit-Change-Number: 14093 Gerrit-PatchSet: 2 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8474: [DOCS] Impala 3.3 Release Notes
Quanlong Huang has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14054 ) Change subject: IMPALA-8474: [DOCS] Impala 3.3 Release Notes .. IMPALA-8474: [DOCS] Impala 3.3 Release Notes Change-Id: I02bb1944c57866b797b1856af139800445e4ac18 Reviewed-on: http://gerrit.cloudera.org:8080/14054 Tested-by: Impala Public Jenkins Tested-by: Quanlong Huang Reviewed-by: Quanlong Huang --- M docs/impala_keydefs.ditamap M docs/topics/impala_incompatible_changes.xml M docs/topics/impala_new_features.xml 3 files changed, 842 insertions(+), 951 deletions(-) Approvals: Impala Public Jenkins: Verified Quanlong Huang: Looks good to me, approved; Verified -- To view, visit http://gerrit.cloudera.org:8080/14054 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: merged Gerrit-Change-Id: I02bb1944c57866b797b1856af139800445e4ac18 Gerrit-Change-Number: 14054 Gerrit-PatchSet: 5 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong
[Impala-ASF-CR] IMPALA-8474: [DOCS] Impala 3.3 Release Notes
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14054 ) Change subject: IMPALA-8474: [DOCS] Impala 3.3 Release Notes .. Patch Set 4: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14054 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I02bb1944c57866b797b1856af139800445e4ac18 Gerrit-Change-Number: 14054 Gerrit-PatchSet: 4 Gerrit-Owner: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 15:34:31 + Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8683: Document parquet/lz4 compression codec
Quanlong Huang has posted comments on this change. ( http://gerrit.cloudera.org:8080/14093 ) Change subject: IMPALA-8683: Document parquet/lz4 compression codec .. Patch Set 1: Code-Review+2 -- To view, visit http://gerrit.cloudera.org:8080/14093 To unsubscribe, visit http://gerrit.cloudera.org:8080/settings Gerrit-Project: Impala-ASF Gerrit-Branch: master Gerrit-MessageType: comment Gerrit-Change-Id: I304274842c8494a021816d68bc5d81810c353146 Gerrit-Change-Number: 14093 Gerrit-PatchSet: 1 Gerrit-Owner: Abhishek Rawat Gerrit-Reviewer: Alex Rodoni Gerrit-Reviewer: Impala Public Jenkins Gerrit-Reviewer: Quanlong Huang Gerrit-Reviewer: Tim Armstrong Gerrit-Comment-Date: Mon, 19 Aug 2019 15:21:36 + Gerrit-HasComments: No
