[
https://issues.apache.org/jira/browse/DRILL-6453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562262#comment-16562262
]
ASF GitHub Bot commented on DRILL-6453:
---------------------------------------
ilooner opened a new pull request #1408: DRILL-6453: Resolve deadlock when
reading from build and probe sides simultaneously in HashJoin
URL: https://github.com/apache/drill/pull/1408
# The Problem
Originally hash join sniffed the first data holding batch in the probe and
build side. Using the size statistics from both sides, memory calculations were
performed in order to determine when to spill data.
The issue with this is that fetching the first data holding batch from both
sides can cause a deadlock in the exchange operators. Details of how this can
happen have been included by others on the Jira.
# Theory of Operation
## Sniffing Batches
Batch sniffing is done in three phases.
1. Schema sniffing is done in buildSchema()
2. Before executing the build phase we sniff the first data holding build
side batch and use the stats to decide the number of partitions and do memory
calculations.
3. Before executing the probe phase we sniff the first data holding probe
side batch, and use the size statistics to do memory calculations that decide
when to spill.
## Memory Estimation
When sniffing the schema for the build and probe side, we may get lucky and
get data for the probe side. If this is the case then we cause use the probe
side data to estimate the optimal number of partitions to use in the join
operator. If we don't have probe side data when computing the number of
partitions to use we assume that the incoming probe batches will be less than
or equal to the configured batch size.
Since the number of partitions must be configured upfront before we may have
probe data, we may get stuck in a situation where we have too many partitions
to effectively process the probe side. In order to avoid this scenario we also
adjust the number of records in probe side partition batches after we receive
data from the probe side.
## Corner cases
While implementing this many corner cases had to be handled.
- Empty build side
- Empty probe side
- Empty probe and build sides
- Getting probe side data when retrieving the probe schema
- Not getting probe side data when retrieving the probe schema
## Testing
I added unit tests for all the corner cases, and have extracted logic for
predicting incoming and partition batch sizes into BatchSizePredictorImpl. In
unit tests various corner cases are tested by providing mock implementation of
BatchSizePredictorImpl.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> TPC-DS query 72 has regressed
> -----------------------------
>
> Key: DRILL-6453
> URL: https://issues.apache.org/jira/browse/DRILL-6453
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
> Affects Versions: 1.14.0
> Reporter: Khurram Faraaz
> Assignee: Timothy Farkas
> Priority: Blocker
> Fix For: 1.15.0
>
> Attachments: 24f75b18-014a-fb58-21d2-baeab5c3352c.sys.drill,
> jstack_29173_June_10_2018.txt, jstack_29173_June_10_2018.txt,
> jstack_29173_June_10_2018_b.txt, jstack_29173_June_10_2018_b.txt,
> jstack_29173_June_10_2018_c.txt, jstack_29173_June_10_2018_c.txt,
> jstack_29173_June_10_2018_d.txt, jstack_29173_June_10_2018_d.txt,
> jstack_29173_June_10_2018_e.txt, jstack_29173_June_10_2018_e.txt
>
>
> TPC-DS query 72 seems to have regressed, query profile for the case where it
> Canceled after 2 hours on Drill 1.14.0 is attached here.
> {noformat}
> On, Drill 1.14.0-SNAPSHOT
> commit : 931b43e (TPC-DS query 72 executed successfully on this commit, took
> around 55 seconds to execute)
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> TPC-DS query 72 executed successfully & took 47 seconds to complete execution.
> {noformat}
> {noformat}
> TPC-DS data in the below run has date values stored as DATE datatype and not
> VARCHAR type
> On, Drill 1.14.0-SNAPSHOT
> commit : 82e1a12
> SF1 parquet data on 4 nodes;
> planner.memory.max_query_memory_per_node = 10737418240.
> drill.exec.hashagg.fallback.enabled = true
> and
> alter system set `exec.hashjoin.num_partitions` = 1;
> TPC-DS query 72 executed for 2 hrs and 11 mins and did not complete, I had to
> Cancel it by stopping the Foreman drillbit.
> As a result several minor fragments are reported to be in
> CANCELLATION_REQUESTED state on UI.
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
