[
https://issues.apache.org/jira/browse/IMPALA-10565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Riza Suminto updated IMPALA-10565:
----------------------------------
Description:
IMPALA-9856 enable result spooling by default. However, if query option
scratch_limit is set lower than the default max_spilled_result_spooling_mem,
the query might fail in the middle of execution due to insufficient scratch
space.
This is shown in intermittent failure of
TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit:
{code:java}
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
E MESSAGE: Memory limit exceeded: Could not free memory by spilling to disk:
scratch_limit is 0
E Error occurred on backend
impala-ec2-centos74-m5-4xlarge-ondemand-1827.vpc.cloudera.com:27000 by fragment
6f44a34948c173dd:a1a63d8000000000
E Memory left in process limit: 8.69 GB
E Query(6f44a34948c173dd:a1a63d8000000000): Reservation=100.00 MB
ReservationLimit=9.60 GB OtherMemory=14.25 MB Total=114.25 MB Peak=114.25 MB
E Fragment 6f44a34948c173dd:a1a63d8000000000: Reservation=100.00 MB
OtherMemory=14.25 MB Total=114.25 MB Peak=114.25 MB
E EXCHANGE_NODE (id=2): Reservation=14.23 MB OtherMemory=0 Total=14.23 MB
Peak=14.32 MB
E KrpcDeferredRpcs: Total=0 Peak=90.53 KB
E PLAN_ROOT_SINK: Reservation=100.00 MB ReservationLimit=100.00 MB
OtherMemory=8.00 KB Total=100.01 MB Peak=100.01 MB
E CodeGen: Total=263.00 B Peak=387.00 KB{code}
Before the query run, we need to validate that max_spilled_result_spooling_mem
<= scratch_limit.
was:
IMPALA-9856 enable result spooling by default. However, if query option
scratch_limit is set lower than the default max_spilled_result_spooling_mem,
the query might fail in the middle of execution due to insufficient scratch
space.
This is shown in intermittent failure of
TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit:
E ImpalaBeeswaxException: ImpalaBeeswaxException:
E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
E MESSAGE: Memory limit exceeded: Could not free memory by spilling to disk:
scratch_limit is 0
E Error occurred on backend
impala-ec2-centos74-m5-4xlarge-ondemand-1827.vpc.cloudera.com:27000 by fragment
6f44a34948c173dd:a1a63d8000000000
E Memory left in process limit: 8.69 GB
E Query(6f44a34948c173dd:a1a63d8000000000): Reservation=100.00 MB
ReservationLimit=9.60 GB OtherMemory=14.25 MB Total=114.25 MB Peak=114.25 MB
E Fragment 6f44a34948c173dd:a1a63d8000000000: Reservation=100.00 MB
OtherMemory=14.25 MB Total=114.25 MB Peak=114.25 MB
E EXCHANGE_NODE (id=2): Reservation=14.23 MB OtherMemory=0 Total=14.23 MB
Peak=14.32 MB
E KrpcDeferredRpcs: Total=0 Peak=90.53 KB
E PLAN_ROOT_SINK: Reservation=100.00 MB ReservationLimit=100.00 MB
OtherMemory=8.00 KB Total=100.01 MB Peak=100.01 MB
E CodeGen: Total=263.00 B Peak=387.00 KB
Before the query run, we need to validate that max_spilled_result_spooling_mem
<= scratch_limit.
> Validate max_spilled_result_spooling_mem against scratch_limit
> --------------------------------------------------------------
>
> Key: IMPALA-10565
> URL: https://issues.apache.org/jira/browse/IMPALA-10565
> Project: IMPALA
> Issue Type: Bug
> Components: Backend
> Affects Versions: Impala 4.0
> Reporter: Riza Suminto
> Assignee: Riza Suminto
> Priority: Major
>
> IMPALA-9856 enable result spooling by default. However, if query option
> scratch_limit is set lower than the default max_spilled_result_spooling_mem,
> the query might fail in the middle of execution due to insufficient scratch
> space.
> This is shown in intermittent failure of
> TestScratchLimit::test_with_zero_scratch_limit_no_memory_limit:
> {code:java}
> E ImpalaBeeswaxException: ImpalaBeeswaxException:
> E INNER EXCEPTION: <class 'beeswaxd.ttypes.BeeswaxException'>
> E MESSAGE: Memory limit exceeded: Could not free memory by spilling to
> disk: scratch_limit is 0
> E Error occurred on backend
> impala-ec2-centos74-m5-4xlarge-ondemand-1827.vpc.cloudera.com:27000 by
> fragment 6f44a34948c173dd:a1a63d8000000000
> E Memory left in process limit: 8.69 GB
> E Query(6f44a34948c173dd:a1a63d8000000000): Reservation=100.00 MB
> ReservationLimit=9.60 GB OtherMemory=14.25 MB Total=114.25 MB Peak=114.25 MB
> E Fragment 6f44a34948c173dd:a1a63d8000000000: Reservation=100.00 MB
> OtherMemory=14.25 MB Total=114.25 MB Peak=114.25 MB
> E EXCHANGE_NODE (id=2): Reservation=14.23 MB OtherMemory=0 Total=14.23
> MB Peak=14.32 MB
> E KrpcDeferredRpcs: Total=0 Peak=90.53 KB
> E PLAN_ROOT_SINK: Reservation=100.00 MB ReservationLimit=100.00 MB
> OtherMemory=8.00 KB Total=100.01 MB Peak=100.01 MB
> E CodeGen: Total=263.00 B Peak=387.00 KB{code}
> Before the query run, we need to validate that
> max_spilled_result_spooling_mem <= scratch_limit.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
