[
https://issues.apache.org/jira/browse/IMPALA-8818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16914748#comment-16914748
]
ASF subversion and git services commented on IMPALA-8818:
---------------------------------------------------------
Commit d037ac8304b43f6e4bb4c6ba2eb1910a9e921c24 in impala's branch
refs/heads/master from Sahil Takiar
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=d037ac8 ]
IMPALA-8818: Replace deque with spillable queue in BufferedPRS
Replaces DequeRowBatchQueue with SpillableRowBatchQueue in
BufferedPlanRootSink. A few changes to BufferedPlanRootSink were
necessary for it to work with the spillable queue, however, all the
synchronization logic is the same.
SpillableRowBatchQueue is a wrapper around a BufferedTupleStream and
a ReservationManager. It takes in a TBackendResourceProfile that
specifies the max / min memory reservation the BufferedTupleStream can
use to buffer rows. The 'max_unpinned_bytes' parameter limits the max
number of bytes that can be unpinned in the BufferedTupleStream. The
limit is a 'soft' limit because calls to AddBatch may push the amount of
unpinned memory over the limit. The queue is non-blocking and not thread
safe. It provides AddBatch and GetBatch methods. Calls to AddBatch spill
if the BufferedTupleStream does not have enough reservation to fit the
entire RowBatch.
Adds two new query options: 'MAX_PINNED_RESULT_SPOOLING_MEMORY' and
'MAX_UNPINNED_RESULT_SPOOLING_MEMORY', which bound the amount of pinned
and unpinned memory that a query can use for spooling, respectively.
MAX_PINNED_RESULT_SPOOLING_MEMORY must be <=
MAX_UNPINNED_RESULT_SPOOLING_MEMORY in order to allow all the pinned
data in the BufferedTupleStream to be unpinned. This is enforced in a
new method in QueryOptions called 'ValidateQueryOptions'.
Planner Changes:
PlanRootSink.java now computes a full ResourceProfile if result spooling
is enabled. The min mem reservation is bounded by the size of the read and
write pages used by the BufferedTupleStream. The max mem reservation is
bounded by 'MAX_PINNED_RESULT_SPOOLING_MEMORY'. The mem estimate is
computed by estimating the size of the result set using stats.
BufferedTupleStream Re-Factoring:
For the most part, using a BufferedTupleStream outside an ExecNode works
properly. However, some changes were necessary:
* The message for the MAX_ROW_SIZE error is ExecNode specific. In order to
fix this, this patch introduces the concept of an ExecNode 'label' which
is a more generic version of an ExecNode 'id'.
* The definition of TBackendResourceProfile lived in PlanNodes.thrift,
it was moved to its own file so it can be used by DataSinks.thrift.
* Modified BufferedTupleStream so it internally tracks how many bytes
are unpinned (necessary for 'MAX_UNPINNED_RESULT_SPOOLING_MEMORY').
Metrics:
* Added a few of the metrics mentioned in IMPALA-8825 to
BufferedPlanRootSink. Specifically, added timers to track how much time
is spent waiting in the BufferedPlanRootSink 'Send' and 'GetNext'
methods.
* The BufferedTupleStream in the SpillableRowBatchQueue exposes several
BufferPool metrics such as number of reserved and unpinned bytes.
Bug Fixes:
* Fixed a bug in BufferedPlanRootSink where the MemPool used by the
expression evaluators was not being cleared incrementally.
* Fixed a bug where the inactive timer was not being properly updated in
BufferedPlanRootSink.
* Fixed a bug where RowBatch memory was not freed if
BufferedPlanRootSink::GetNext terminated early because it could not
handle requests where num_results < BATCH_SIZE.
Testing:
* Added new tests to test_result_spooling.py.
* Updated errors thrown in spilling-large-rows.test.
* Ran exhaustive tests.
Change-Id: I10f9e72374cdf9501c0e5e2c5b39c13688ae65a9
Reviewed-on: http://gerrit.cloudera.org:8080/14039
Reviewed-by: Sahil Takiar <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Replace deque queue with spillable queue in BufferedPlanRootSink
> ----------------------------------------------------------------
>
> Key: IMPALA-8818
> URL: https://issues.apache.org/jira/browse/IMPALA-8818
> Project: IMPALA
> Issue Type: Sub-task
> Components: Backend
> Reporter: Sahil Takiar
> Assignee: Sahil Takiar
> Priority: Major
>
> Add a {{SpillableRowBatchQueue}} to replace the {{DequeRowBatchQueue}} in
> {{BufferedPlanRootSink}}. The {{SpillableRowBatchQueue}} will wrap a
> {{BufferedTupleStream}} and take in a {{TBackendResourceProfile}} created by
> {{PlanRootSink#computeResourceProfile}}.
> *BufferedTupleStream Usage*:
> The wrapped {{BufferedTupleStream}} should be created in 'attach_on_read'
> mode so that pages are attached to the output {{RowBatch}} in
> {{BufferedTupleStream::GetNext}}. The BTS should start off as pinned (e.g.
> all pages are pinned). If a call to {{BufferedTupleStream::AddRow}} returns
> false (it returns false if "the unused reservation was not sufficient to add
> a new page to the stream large enough to fit 'row' and the stream could not
> increase the reservation to get enough unused reservation"), it should unpin
> the stream ({{BufferedTupleStream::UnpinStream}}) and then add the row (if
> the row still could not be added, then an error must have occurred, perhaps
> an IO error, in which case return the error and fail the query).
> *Constraining Resources*:
> When result spooling is disabled, a user can run a {{select * from
> [massive-fact-table]}} and scroll through the results without affecting the
> health of the Impala cluster (assuming they close they query promptly).
> Impala will stream the results one batch at a time to the user.
> With result spooling, a naive implementation might try and buffer the enter
> fact table, and end up spilling all the contents to disk, which can
> potentially take up a large amount of space. So there needs to be
> restrictions on the memory and disk space used by the {{BufferedTupleStream}}
> in order to ensure a scan of a massive table does not consume all the memory
> or disk space of the Impala coordinator.
> This problem can be solved by placing a max size on the amount of unpinned
> memory (perhaps through a new config option
> {{MAX_PINNED_RESULT_SPOOLING_MEMORY}} (maybe set to a few GBs by default).
> The max amount of pinned memory should already be constrained by the
> reservation (see next paragraph). NUM_ROWS_PRODUCED_LIMIT already limits the
> number of rows returned by a query, and so it should limit the number of rows
> buffered by the BTS as well (although it is set to 0 by default).
> SCRATCH_LIMIT already limits the amount of disk space used for spilling
> (although it is set to -1 by default).
> The {{PlanRootSink}} should attempt to accurately estimate how much memory it
> needs to buffer all results in memory. This requires setting an accurate
> value of {{ResourceProfile#memEstimateBytes_}} in
> {{PlanRootSink#computeResourceProfile}}. If statistics are available, the
> estimate can be based on the number of estimated rows returned multiplied by
> the size of the rows returned. The min reservation should account for a read
> and write page for the {{BufferedTupleStream}}.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
