[
https://issues.apache.org/jira/browse/IMPALA-9225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17174137#comment-17174137
]
Quanlong Huang commented on IMPALA-9225:
----------------------------------------
I plan to add a new query option, safely_retry_queries (defaults to true), to
control the following behavior: retryable queries will turn on result spooling
and spool all results before returning any to the client. if all query results
cannot be contained in the allocated result spooling space (including unpinned
memory), results will be returned and query retry will be disable. The retried
queries won't enable result spooling if it's not turned on explicitly.
If safely_retry_queries is false, we fallback to the current behavior:
retryable queries will return results immediately and skip retrying the query
if it fails and client has fetches some results.
*Design*
* If retry_failed_queries and safely_retry_queries are true, FE sets the
ResourceProfile for the plan root sink (same as SPOOL_QUERY_RESULTS is true).
* BufferedPlanRootSink::GetNext will block sending the first batch until
(sender_state_ == SenderState::EOS || timeout || IsCancelledOrClosed ||
batch_queue_->IsFull()).
** If sender_state_ == SenderState::EOS, returns the results as normal
workflow.
** If (timeout || IsCancelledOrClosed), report errors.
** If batch_queue_->IsFull(), returns the results and mark a state about this.
Add information about this in QueryDriver::TryQueryRetry when we have to skip
retrying the query.
However, this requires the support for retrying a query in FINISHED state: when
"rows available" happens on a query, its state transits to FINISHED. This
doesn't depend on whether the client has called fetch() on the query. If the
client hasn't fetched any rows on the query, we should be able to retry it when
it fails. I create a separate Jira for this: IMPALA-10065
> Retryable queries should spool all results before returning any to the client
> -----------------------------------------------------------------------------
>
> Key: IMPALA-9225
> URL: https://issues.apache.org/jira/browse/IMPALA-9225
> Project: IMPALA
> Issue Type: Sub-task
> Reporter: Sahil Takiar
> Assignee: Quanlong Huang
> Priority: Critical
>
> If query retries are enabled, a query should not return any results to the
> client until all results are spooled. The issue is that once a query starts
> returning results, retrying the query becomes increasingly complex and is not
> supported in the initial version of IMPALA-9124. Retrying a query while
> returning results could cause incorrect results, especially for
> non-deterministic queries (e.g. when the results are not ordered).
> Since a query can fail anytime while results are being produced, transparent
> retries are most effective if they can be done during any point of query
> execution.
> The one edge case is what happens if all query results cannot be contained in
> the allocated result spooling memory (including unpinned memory). In this
> case, retries for the query should be transparently disabled.
> We should consider making this configurable, in case it leads to performance
> degradation. Although, I'm inclined to turn the flag on by default (e.g.
> always spool all returns before returning them), otherwise (depending on the
> query) query retries won't always be helpful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
