[
https://issues.apache.org/jira/browse/DRILL-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16553474#comment-16553474
]
ASF GitHub Bot commented on DRILL-6626:
---------------------------------------
Ben-Zvi commented on a change in pull request #1394: DRILL-6626: Fixed an
IndexOutOfBoundException during aggregator rehash
URL: https://github.com/apache/drill/pull/1394#discussion_r204571826
##########
File path:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/common/HashTableTemplate.java
##########
@@ -816,7 +814,7 @@ private void resizeAndRehashIfNeeded() {
logger.debug("Bucket: {}, startIdx[ {} ] = {}.", i, i,
startIndices.getAccessor().get(i));
int startIdx = startIndices.getAccessor().get(i);
BatchHolder bh = batchHolders.get((startIdx >>> 16) & BATCH_MASK);
- bh.dump(idx);
+ bh.dump(startIdx);
Review comment:
Good catch !
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Hash Aggregate: Index out of bounds with small output batch size and spilling
> -----------------------------------------------------------------------------
>
> Key: DRILL-6626
> URL: https://issues.apache.org/jira/browse/DRILL-6626
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Relational Operators
> Affects Versions: 1.14.0
> Reporter: Boaz Ben-Zvi
> Assignee: salim achouche
> Priority: Major
> Labels: pull-request-available
>
> This new IOOB failure was seen while trying to recreate the NPE failure in
> DRILL-6622 (over TPC-DS SF1). The proposed fix for the latter (PR #1391) does
> not seem to make a difference.
> This IOOB can easily be created with other large Hash-Agg queries that need
> to spill.
> The IOOB was caused after restricting the output batch size (to force many),
> and the Hash Aggr memory (to force a spill):
> {code}
> 0: jdbc:drill:zk=local> alter system set
> `drill.exec.memory.operator.output_batch_size` = 262144;
> +-------+--------------------------------------------------------+
> | ok | summary |
> +-------+--------------------------------------------------------+
> | true | drill.exec.memory.operator.output_batch_size updated. |
> +-------+--------------------------------------------------------+
> 1 row selected (0.106 seconds)
> 0: jdbc:drill:zk=local>
> 0: jdbc:drill:zk=local> alter session set `exec.errors.verbose` = true;
> +-------+-------------------------------+
> | ok | summary |
> +-------+-------------------------------+
> | true | exec.errors.verbose updated. |
> +-------+-------------------------------+
> 1 row selected (0.081 seconds)
> 0: jdbc:drill:zk=local>
> 0: jdbc:drill:zk=local> alter session set `exec.hashagg.mem_limit` = 16777216;
> +-------+----------------------------------+
> | ok | summary |
> +-------+----------------------------------+
> | true | exec.hashagg.mem_limit updated. |
> +-------+----------------------------------+
> 1 row selected (0.089 seconds)
> 0: jdbc:drill:zk=local>
> 0: jdbc:drill:zk=local> SELECT c_customer_id FROM
> dfs.`/data/tpcds/sf1/parquet/customer`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT ca_address_id FROM
> dfs.`/data/tpcds/sf1/parquet/customer_address`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT cd_credit_rating FROM
> dfs.`/data/tpcds/sf1/parquet/customer_demographics`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT hd_buy_potential FROM
> dfs.`/data/tpcds/sf1/parquet/household_demographics`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT i_item_id FROM
> dfs.`/data/tpcds/sf1/parquet/item`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT p_promo_id FROM
> dfs.`/data/tpcds/sf1/parquet/promotion`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT t_time_id FROM
> dfs.`/data/tpcds/sf1/parquet/time_dim`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT d_date_id FROM
> dfs.`/data/tpcds/sf1/parquet/date_dim`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT s_store_id FROM
> dfs.`/data/tpcds/sf1/parquet/store`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT w_warehouse_id FROM
> dfs.`/data/tpcds/sf1/parquet/warehouse`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT sm_ship_mode_id FROM
> dfs.`/data/tpcds/sf1/parquet/ship_mode`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT r_reason_id FROM
> dfs.`/data/tpcds/sf1/parquet/reason`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT cc_call_center_id FROM
> dfs.`/data/tpcds/sf1/parquet/call_center`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT web_site_id FROM
> dfs.`/data/tpcds/sf1/parquet/web_site`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT wp_web_page_id FROM
> dfs.`/data/tpcds/sf1/parquet/web_page`
> . . . . . . . . . . . > UNION
> . . . . . . . . . . . > SELECT cp_catalog_page_id FROM
> dfs.`/data/tpcds/sf1/parquet/catalog_page`;
> Error: SYSTEM ERROR: IndexOutOfBoundsException: Index: 26474, Size: 7
> Fragment 4:0
> [Error Id: d44e64ea-f474-436e-94b0-61c61eec2227 on 172.30.8.176:31020]
> (java.lang.IndexOutOfBoundsException) Index: 26474, Size: 7
> java.util.ArrayList.rangeCheck():653
> java.util.ArrayList.get():429
>
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.rehash():293
>
> org.apache.drill.exec.physical.impl.common.HashTableTemplate$BatchHolder.access$1300():120
>
> org.apache.drill.exec.physical.impl.common.HashTableTemplate.resizeAndRehashIfNeeded():805
> org.apache.drill.exec.physical.impl.common.HashTableTemplate.put():682
>
> org.apache.drill.exec.physical.impl.aggregate.HashAggTemplate.checkGroupAndAggrValues():1379
> org.apache.drill.exec.physical.impl.aggregate.HashAggTemplate.doWork():604
> org.apache.drill.exec.physical.impl.aggregate.HashAggBatch.innerNext():273
> org.apache.drill.exec.record.AbstractRecordBatch.next():172
>
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():229
> org.apache.drill.exec.record.AbstractRecordBatch.next():119
> org.apache.drill.exec.record.AbstractRecordBatch.next():109
> org.apache.drill.exec.record.AbstractUnaryRecordBatch.innerNext():63
>
> org.apache.drill.exec.physical.impl.project.ProjectRecordBatch.innerNext():142
> org.apache.drill.exec.record.AbstractRecordBatch.next():172
>
> org.apache.drill.exec.physical.impl.validate.IteratorValidatorBatchIterator.next():229
> org.apache.drill.exec.physical.impl.BaseRootExec.next():103
>
> org.apache.drill.exec.physical.impl.SingleSenderCreator$SingleSenderRootExec.innerNext():93
> org.apache.drill.exec.physical.impl.BaseRootExec.next():93
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():294
> org.apache.drill.exec.work.fragment.FragmentExecutor$1.run():281
> java.security.AccessController.doPrivileged():-2
> javax.security.auth.Subject.doAs():422
> org.apache.hadoop.security.UserGroupInformation.doAs():1657
> org.apache.drill.exec.work.fragment.FragmentExecutor.run():281
> org.apache.drill.common.SelfCleaningRunnable.run():38
> java.util.concurrent.ThreadPoolExecutor.runWorker():1142
> java.util.concurrent.ThreadPoolExecutor$Worker.run():617
> java.lang.Thread.run():745 (state=,code=0)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
