Boaz Ben-Zvi created DRILL-6861:
-----------------------------------
Summary: Hash-Join: Spilled partitions are skipped following an
empty probe side
Key: DRILL-6861
URL: https://issues.apache.org/jira/browse/DRILL-6861
Project: Apache Drill
Issue Type: Bug
Components: Execution - Relational Operators
Affects Versions: 1.14.0
Reporter: Boaz Ben-Zvi
Assignee: Boaz Ben-Zvi
Fix For: 1.15.0
Following DRILL-6755 (_Avoid building a hash table when the probe side is
empty_) - The special case of an empty spilled probe-partition was not handled.
When such a case happens, the Hash-Join terminates early (returns NONE) and
the remaining partitions are not processed/returned (which may lead to
incorrect results).
A test case - force tpcds/query95 to spill :
{code:java}
0: jdbc:drill:zk=local> alter system set `exec.hashjoin.max_batches_in_memory`
= 40;
+-------+-----------------------------------------------+
| ok | summary |
+-------+-----------------------------------------------+
| true | exec.hashjoin.max_batches_in_memory updated. |
+-------+-----------------------------------------------+
1 row selected (1.325 seconds)
0: jdbc:drill:zk=local> WITH ws_wh AS
. . . . . . . . . . . > (
. . . . . . . . . . . > SELECT ws1.ws_order_number,
. . . . . . . . . . . > ws1.ws_warehouse_sk wh1,
. . . . . . . . . . . > ws2.ws_warehouse_sk wh2
. . . . . . . . . . . > FROM dfs.`/data/tpcds/sf1/parquet/web_sales`
ws1,
. . . . . . . . . . . > dfs.`/data/tpcds/sf1/parquet/web_sales`
ws2
. . . . . . . . . . . > WHERE ws1.ws_order_number = ws2.ws_order_number
. . . . . . . . . . . > AND ws1.ws_warehouse_sk <>
ws2.ws_warehouse_sk)
. . . . . . . . . . . > SELECT
. . . . . . . . . . . > Count(DISTINCT ws1.ws_order_number) AS `order
count` ,
. . . . . . . . . . . > Sum(ws1.ws_ext_ship_cost) AS `total
shipping cost` ,
. . . . . . . . . . . > Sum(ws1.ws_net_profit) AS `total
net profit`
. . . . . . . . . . . > FROM dfs.`/data/tpcds/sf1/parquet/web_sales` ws1 ,
. . . . . . . . . . . > dfs.`/data/tpcds/sf1/parquet/date_dim` dd,
. . . . . . . . . . . > dfs.`/data/tpcds/sf1/parquet/customer_address`
ca,
. . . . . . . . . . . > dfs.`/data/tpcds/sf1/parquet/web_site` wbst
. . . . . . . . . . . > WHERE dd.d_date BETWEEN '2000-04-01' AND (
. . . . . . . . . . . > Cast('2000-04-01' AS DATE) + INTERVAL
'60' day)
. . . . . . . . . . . > AND ws1.ws_ship_date_sk = dd.d_date_sk
. . . . . . . . . . . > AND ws1.ws_ship_addr_sk = ca.ca_address_sk
. . . . . . . . . . . > AND ca.ca_state = 'IN'
. . . . . . . . . . . > AND ws1.ws_web_site_sk = wbst.web_site_sk
. . . . . . . . . . . > AND wbst.web_company_name = 'pri'
. . . . . . . . . . . > AND ws1.ws_order_number IN
. . . . . . . . . . . > (
. . . . . . . . . . . > SELECT ws_wh.ws_order_number
. . . . . . . . . . . > FROM ws_wh)
. . . . . . . . . . . > AND ws1.ws_order_number IN
. . . . . . . . . . . > (
. . . . . . . . . . . > SELECT wr.wr_order_number
. . . . . . . . . . . > FROM
dfs.`/data/tpcds/sf1/parquet/web_returns` wr,
. . . . . . . . . . . > ws_wh
. . . . . . . . . . . > WHERE wr.wr_order_number =
ws_wh.ws_order_number)
. . . . . . . . . . . > ORDER BY count(DISTINCT ws1.ws_order_number)
. . . . . . . . . . . > LIMIT 100;
+--------------+----------------------+---------------------+
| order count | total shipping cost | total net profit |
+--------------+----------------------+---------------------+
| 17 | 38508.130000000005 | 20822.3 |
+--------------+----------------------+---------------------+
1 row selected (105.621 seconds)
{code}
The correct results should be:
{code:java}
+--------------+----------------------+---------------------+
| order count | total shipping cost | total net profit |
+--------------+----------------------+---------------------+
| 34 | 63754.72 | 15919.009999999998 |
+--------------+----------------------+---------------------+
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
