[Impala-ASF-CR] IMPALA-10565: Check max spilled result spooling mem vs scratch limit

2021-03-10 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17166 )

Change subject: IMPALA-10565: Check max_spilled_result_spooling_mem vs 
scratch_limit
..


Patch Set 2:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8334/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17166
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I541f46e6911694e14c0fc25be1a6982fd929d3a9
Gerrit-Change-Number: 17166
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Thu, 11 Mar 2021 01:10:49 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10565: Check max spilled result spooling mem vs scratch limit

2021-03-10 Thread Riza Suminto (Code Review) 
Riza Suminto has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17166 )

Change subject: IMPALA-10565: Check max_spilled_result_spooling_mem vs 
scratch_limit
..


Patch Set 2:

(2 comments)

Patch set 2 disable result spooling in tests that set scratch_limit > -1.

http://gerrit.cloudera.org:8080/#/c/17166/1/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/17166/1/be/src/service/query-options.cc@
PS1, Line : un
> Agree. I will rephrase into this:
Done


http://gerrit.cloudera.org:8080/#/c/17166/1/be/src/service/query-options.cc@1124
PS1, Line 1124:   if (query_options->spool_query_results && scratch_limit != 
-1) {
> Good point. Quick git grep show some more places where scratch_limit is set
Done



--
To view, visit http://gerrit.cloudera.org:8080/17166
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I541f46e6911694e14c0fc25be1a6982fd929d3a9
Gerrit-Change-Number: 17166
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Thu, 11 Mar 2021 00:53:54 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10565: Check max spilled result spooling mem vs scratch limit

2021-03-10 Thread Riza Suminto (Code Review) 
Hello Aman Sinha, Bikramjeet Vig, Impala Public Jenkins,

I'd like you to reexamine a change. Please visit

http://gerrit.cloudera.org:8080/17166

to look at the new patch set (#2).

Change subject: IMPALA-10565: Check max_spilled_result_spooling_mem vs 
scratch_limit
..

IMPALA-10565: Check max_spilled_result_spooling_mem vs scratch_limit

IMPALA-9856 enables result spooling by default. However, if the query
option scratch_limit is set lower than max_spilled_result_spooling_mem,
the query might fail in the middle of execution due to insufficient
scratch space. This patch validation that when result spooling is
enabled, max_spilled_result_spooling_mem <= scratch_limit.

This patch also fix validation between max_result_spooling_mem and
max_spilled_result_spooling_mem that should treat both value 0 and -1 as
unbounded.

Testing:
- Lower max_spilled_result_spooling_mem in test_with_high_scratch_limit
  and test_with_low_scratch_limit.
- Toggle off spool_query_results in tests that set scratch_limit=0 to
  ensure that result spilling will not happen.
- Add test_with_scratch_limit_less_than_max_spilled_result_spooling_mem.
- Add be test QueryOptions.ResultSpoolingWithScratchLimit.
- Pass exhaustive tests.

Change-Id: I541f46e6911694e14c0fc25be1a6982fd929d3a9
---
M be/src/runtime/bufferpool/buffer-pool-test.cc
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M 
testdata/workloads/functional-query/queries/QueryTest/spilling-naaj-no-deny-reservation.test
M testdata/workloads/functional-query/queries/QueryTest/spilling-naaj.test
M testdata/workloads/tpch/queries/sort-reservation-usage-single-node.test
M tests/query_test/test_scratch_limit.py
7 files changed, 129 insertions(+), 12 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/17166/2
--
To view, visit http://gerrit.cloudera.org:8080/17166
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I541f46e6911694e14c0fc25be1a6982fd929d3a9
Gerrit-Change-Number: 17166
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto

[Impala-ASF-CR] IMPALA-10565: Check max spilled result spooling mem vs scratch limit

2021-03-10 Thread Riza Suminto (Code Review) 
Riza Suminto has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17166 )

Change subject: IMPALA-10565: Check max_spilled_result_spooling_mem vs 
scratch_limit
..


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17166/1/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/17166/1/be/src/service/query-options.cc@
PS1, Line : $1
> slightly misleading,does not necessarily have to be the same, could be 0 or
Agree. I will rephrase into this:
"If max_result_spooling_mem is set to unbounded ($0) 
max_spilled_result_spooling_mem must be set to unbounded (0 or -1) as well."


http://gerrit.cloudera.org:8080/#/c/17166/1/be/src/service/query-options.cc@1124
PS1, Line 1124:   if (query_options->spool_query_results && scratch_limit != 
-1) {
> since now we have spilling on by default, spool_query_results is set to 1GB
Good point. Quick git grep show some more places where scratch_limit is set to 
0, like some in spilling-naaj-no-deny-reservation.test. I will also set 
spool_query_results=0 in these places.



--
To view, visit http://gerrit.cloudera.org:8080/17166
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I541f46e6911694e14c0fc25be1a6982fd929d3a9
Gerrit-Change-Number: 17166
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 10 Mar 2021 23:28:30 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10565: Check max spilled result spooling mem vs scratch limit

2021-03-10 Thread Bikramjeet Vig (Code Review) 
Bikramjeet Vig has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17166 )

Change subject: IMPALA-10565: Check max_spilled_result_spooling_mem vs 
scratch_limit
..


Patch Set 1:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/17166/1/be/src/service/query-options.cc
File be/src/service/query-options.cc:

http://gerrit.cloudera.org:8080/#/c/17166/1/be/src/service/query-options.cc@
PS1, Line : $1
slightly misleading,does not necessarily have to be the same, could be 0 or -1, 
right?


http://gerrit.cloudera.org:8080/#/c/17166/1/be/src/service/query-options.cc@1124
PS1, Line 1124:   if (query_options->spool_query_results && scratch_limit != 
-1) {
since now we have spilling on by default, spool_query_results is set to 1GB, i 
wonder if this check will break older workloads that set the scratch limit to 
something and dont expect it to throw an error.



--
To view, visit http://gerrit.cloudera.org:8080/17166
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I541f46e6911694e14c0fc25be1a6982fd929d3a9
Gerrit-Change-Number: 17166
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Wed, 10 Mar 2021 21:54:31 +
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10565: Check max spilled result spooling mem vs scratch limit

2021-03-09 Thread Impala Public Jenkins (Code Review) 
Impala Public Jenkins has posted comments on this change. ( 
http://gerrit.cloudera.org:8080/17166 )

Change subject: IMPALA-10565: Check max_spilled_result_spooling_mem vs 
scratch_limit
..


Patch Set 1:

Build Successful

https://jenkins.impala.io/job/gerrit-code-review-checks/8323/ : Initial code 
review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun 
to run full precommit tests.


--
To view, visit http://gerrit.cloudera.org:8080/17166
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I541f46e6911694e14c0fc25be1a6982fd929d3a9
Gerrit-Change-Number: 17166
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto 
Gerrit-Reviewer: Aman Sinha 
Gerrit-Reviewer: Bikramjeet Vig 
Gerrit-Reviewer: Impala Public Jenkins 
Gerrit-Reviewer: Riza Suminto 
Gerrit-Comment-Date: Tue, 09 Mar 2021 20:36:22 +
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10565: Check max spilled result spooling mem vs scratch limit

2021-03-09 Thread Riza Suminto (Code Review) 
Riza Suminto has uploaded this change for review. ( 
http://gerrit.cloudera.org:8080/17166


Change subject: IMPALA-10565: Check max_spilled_result_spooling_mem vs 
scratch_limit
..

IMPALA-10565: Check max_spilled_result_spooling_mem vs scratch_limit

IMPALA-9856 enables result spooling by default. However, if the query
option scratch_limit is set lower than max_spilled_result_spooling_mem,
the query might fail in the middle of execution due to insufficient
scratch space. This patch validation that when result spooling is
enabled, max_spilled_result_spooling_mem <= scratch_limit.

This patch also fix validation between max_result_spooling_mem and
max_spilled_result_spooling_mem that should treat both value 0 and -1 as
unbounded.

Testing:
- Lower max_spilled_result_spooling_mem in test_with_high_scratch_limit
  and test_with_low_scratch_limit.
- Toggle off spool_query_results in
  test_with_zero_scratch_limit_no_memory_limit to ensure that spill will
  not happen.
- Add test_with_scratch_limit_less_than_max_spilled_result_spooling_mem.
- Add be test QueryOptions.ResultSpoolingWithScratchLimit.
- Pass exhaustive tests.

Change-Id: I541f46e6911694e14c0fc25be1a6982fd929d3a9
---
M be/src/service/query-options-test.cc
M be/src/service/query-options.cc
M tests/query_test/test_scratch_limit.py
3 files changed, 120 insertions(+), 11 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/66/17166/1
--
To view, visit http://gerrit.cloudera.org:8080/17166
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I541f46e6911694e14c0fc25be1a6982fd929d3a9
Gerrit-Change-Number: 17166
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto