[jira] [Commented] (DRILL-6254) IllegalArgumentException: the requested size must be non-negative
[
https://issues.apache.org/jira/browse/DRILL-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16420565#comment-16420565
]
ASF GitHub Bot commented on DRILL-6254:
---
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/1179
> IllegalArgumentException: the requested size must be non-negative
> -
>
> Key: DRILL-6254
> URL: https://issues.apache.org/jira/browse/DRILL-6254
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Khurram Faraaz
>Assignee: Padma Penumarthy
>Priority: Major
> Labels: ready-to-commit
> Fix For: 1.14.0
>
> Attachments: genAllTypesJSN.py
>
>
> Flatten query fails due to IllegalArgumentException: the requested size must
> be non-negative.
> Script to generate JSON data file is attached here.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_all_types_jsn_to_parquet AS
> . . . . . . . . . . . . . . > SELECT
> . . . . . . . . . . . . . . > CAST( col_int AS INT) col_int,
> . . . . . . . . . . . . . . > CAST( col_bigint AS BIGINT) col_bigint,
> . . . . . . . . . . . . . . > CAST( col_char AS CHAR(10)) col_char,
> . . . . . . . . . . . . . . > CAST( col_fxdln_str AS VARCHAR(256))
> col_fxdln_str,
> . . . . . . . . . . . . . . > CAST( col_varln_str AS VARCHAR(256))
> col_varln_str,
> . . . . . . . . . . . . . . > CAST( col_float AS FLOAT) col_float,
> . . . . . . . . . . . . . . > CAST( col_double AS DOUBLE PRECISION)
> col_double,
> . . . . . . . . . . . . . . > CAST( col_date AS DATE) col_date,
> . . . . . . . . . . . . . . > CAST( col_time AS TIME) col_time,
> . . . . . . . . . . . . . . > CAST( col_tmstmp AS TIMESTAMP) col_tmstmp,
> . . . . . . . . . . . . . . > CAST( col_boolean AS BOOLEAN) col_boolean,
> . . . . . . . . . . . . . . > col_binary,
> . . . . . . . . . . . . . . > array_of_ints from `all_supported_types.json`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 9 |
> +---++
> 1 row selected (0.29 seconds)
> {noformat}
> Reset all options and set slice_target=1
> alter system reset all;
> alter system set `planner.slice_target`=1;
> output_batch_size was set to its default value
> drill.exec.memory.operator.output_batch_size = 16777216
>
> {noformat}
> select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet;
> Error: SYSTEM ERROR: IllegalArgumentException: the requested size must be
> non-negative
> Fragment 0:0
> [Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2018-03-15 12:19:43,916 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa: select *, flatten(array_of_ints) from
> tbl_all_types_jsn_to_parquet
> 2018-03-15 12:19:43,952 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2018-03-15 12:19:43,953 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2018-03-15 12:19:43,966 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
> 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of
> 1 using 1 threads. Time: 2ms total, 2.927366ms avg, 2ms max.
> 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of
> 1 using 1 threads. Earliest start: 2.829000 μs, Latest start: 2.829000 μs,
> Average start: 2.829000 μs .
> 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Took 3 ms to read file metadata
> 2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State to report: RUNNING
> 2018-03-15 12:19:44,905 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State change
[jira] [Commented] (DRILL-6254) IllegalArgumentException: the requested size must be non-negative
[
https://issues.apache.org/jira/browse/DRILL-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416575#comment-16416575
]
ASF GitHub Bot commented on DRILL-6254:
---
Github user paul-rogers commented on a diff in the pull request:
https://github.com/apache/drill/pull/1179#discussion_r177619465
--- Diff:
exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/flatten/FlattenRecordBatch.java
---
@@ -237,7 +237,10 @@ protected IterOutcome doWork() {
private void handleRemainder() {
int remainingRecordCount =
flattener.getFlattenField().getAccessor().getInnerValueCount() - remainderIndex;
-if (!doAlloc(remainingRecordCount)) {
+
+// remainingRecordCount can be much higher than number of rows we will
have in outgoing batch.
+// Do memory allocation only for number of rows we are going to have
in the batch.
+if (!doAlloc(Math.min(remainingRecordCount,
flattenMemoryManager.getOutputRowCount( {
--- End diff --
Not related to this fix at all... If we run out of memory, we should throw
an `OutOfMemoryException` rather than trying to set flags and handle the case.
In particular, after the batch sizing fixes, if we can't allocate memory now,
something is very wrong and we'll never be able to. This code may be a vestige
of the old system where operators tried to negotiate via `OUT_OF_MEMORY`
statuses...
> IllegalArgumentException: the requested size must be non-negative
> -
>
> Key: DRILL-6254
> URL: https://issues.apache.org/jira/browse/DRILL-6254
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Khurram Faraaz
>Assignee: Padma Penumarthy
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: genAllTypesJSN.py
>
>
> Flatten query fails due to IllegalArgumentException: the requested size must
> be non-negative.
> Script to generate JSON data file is attached here.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_all_types_jsn_to_parquet AS
> . . . . . . . . . . . . . . > SELECT
> . . . . . . . . . . . . . . > CAST( col_int AS INT) col_int,
> . . . . . . . . . . . . . . > CAST( col_bigint AS BIGINT) col_bigint,
> . . . . . . . . . . . . . . > CAST( col_char AS CHAR(10)) col_char,
> . . . . . . . . . . . . . . > CAST( col_fxdln_str AS VARCHAR(256))
> col_fxdln_str,
> . . . . . . . . . . . . . . > CAST( col_varln_str AS VARCHAR(256))
> col_varln_str,
> . . . . . . . . . . . . . . > CAST( col_float AS FLOAT) col_float,
> . . . . . . . . . . . . . . > CAST( col_double AS DOUBLE PRECISION)
> col_double,
> . . . . . . . . . . . . . . > CAST( col_date AS DATE) col_date,
> . . . . . . . . . . . . . . > CAST( col_time AS TIME) col_time,
> . . . . . . . . . . . . . . > CAST( col_tmstmp AS TIMESTAMP) col_tmstmp,
> . . . . . . . . . . . . . . > CAST( col_boolean AS BOOLEAN) col_boolean,
> . . . . . . . . . . . . . . > col_binary,
> . . . . . . . . . . . . . . > array_of_ints from `all_supported_types.json`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 9 |
> +---++
> 1 row selected (0.29 seconds)
> {noformat}
> Reset all options and set slice_target=1
> alter system reset all;
> alter system set `planner.slice_target`=1;
> output_batch_size was set to its default value
> drill.exec.memory.operator.output_batch_size = 16777216
>
> {noformat}
> select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet;
> Error: SYSTEM ERROR: IllegalArgumentException: the requested size must be
> non-negative
> Fragment 0:0
> [Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2018-03-15 12:19:43,916 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa: select *, flatten(array_of_ints) from
> tbl_all_types_jsn_to_parquet
> 2018-03-15 12:19:43,952 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2018-03-15 12:19:43,953 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2018-03-15 12:19:43,966 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
> 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.M
[jira] [Commented] (DRILL-6254) IllegalArgumentException: the requested size must be non-negative
[
https://issues.apache.org/jira/browse/DRILL-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408615#comment-16408615
]
ASF GitHub Bot commented on DRILL-6254:
---
Github user priteshm commented on the issue:
https://github.com/apache/drill/pull/1179
@bitblender can you please review this?
> IllegalArgumentException: the requested size must be non-negative
> -
>
> Key: DRILL-6254
> URL: https://issues.apache.org/jira/browse/DRILL-6254
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Khurram Faraaz
>Assignee: Padma Penumarthy
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: genAllTypesJSN.py
>
>
> Flatten query fails due to IllegalArgumentException: the requested size must
> be non-negative.
> Script to generate JSON data file is attached here.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_all_types_jsn_to_parquet AS
> . . . . . . . . . . . . . . > SELECT
> . . . . . . . . . . . . . . > CAST( col_int AS INT) col_int,
> . . . . . . . . . . . . . . > CAST( col_bigint AS BIGINT) col_bigint,
> . . . . . . . . . . . . . . > CAST( col_char AS CHAR(10)) col_char,
> . . . . . . . . . . . . . . > CAST( col_fxdln_str AS VARCHAR(256))
> col_fxdln_str,
> . . . . . . . . . . . . . . > CAST( col_varln_str AS VARCHAR(256))
> col_varln_str,
> . . . . . . . . . . . . . . > CAST( col_float AS FLOAT) col_float,
> . . . . . . . . . . . . . . > CAST( col_double AS DOUBLE PRECISION)
> col_double,
> . . . . . . . . . . . . . . > CAST( col_date AS DATE) col_date,
> . . . . . . . . . . . . . . > CAST( col_time AS TIME) col_time,
> . . . . . . . . . . . . . . > CAST( col_tmstmp AS TIMESTAMP) col_tmstmp,
> . . . . . . . . . . . . . . > CAST( col_boolean AS BOOLEAN) col_boolean,
> . . . . . . . . . . . . . . > col_binary,
> . . . . . . . . . . . . . . > array_of_ints from `all_supported_types.json`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 9 |
> +---++
> 1 row selected (0.29 seconds)
> {noformat}
> Reset all options and set slice_target=1
> alter system reset all;
> alter system set `planner.slice_target`=1;
> output_batch_size was set to its default value
> drill.exec.memory.operator.output_batch_size = 16777216
>
> {noformat}
> select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet;
> Error: SYSTEM ERROR: IllegalArgumentException: the requested size must be
> non-negative
> Fragment 0:0
> [Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2018-03-15 12:19:43,916 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa: select *, flatten(array_of_ints) from
> tbl_all_types_jsn_to_parquet
> 2018-03-15 12:19:43,952 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2018-03-15 12:19:43,953 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2018-03-15 12:19:43,966 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
> 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of
> 1 using 1 threads. Time: 2ms total, 2.927366ms avg, 2ms max.
> 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of
> 1 using 1 threads. Earliest start: 2.829000 μs, Latest start: 2.829000 μs,
> Average start: 2.829000 μs .
> 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Took 3 ms to read file metadata
> 2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State change requested
> AWAITING_ALLOCATION --> RUNNING
> 2018-03-15 12:19:44,000 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
> o.a.d.e.w.f.FragmentStatusReporter -
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State to report: RUNNING
> 2018-03-15 12:19:44,905 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:frag:0:0] INFO
> o.a.d.e.w.fragment.FragmentExecutor -
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa:0:0: State c
[jira] [Commented] (DRILL-6254) IllegalArgumentException: the requested size must be non-negative
[
https://issues.apache.org/jira/browse/DRILL-6254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16408569#comment-16408569
]
ASF GitHub Bot commented on DRILL-6254:
---
GitHub user ppadma opened a pull request:
https://github.com/apache/drill/pull/1179
DRILL-6254: IllegalArgumentException: the requested size must be non-…
…negative
We should limit memory allocation to number of records that are going to be
in the next batch, not the total number of records remaining. For very large
remaining record count, when multiplied with high cardinality, integer
overflows and is becoming negative.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/ppadma/drill DRILL-6254
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/1179.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1179
commit c26e8f44b9c873d0d7acabac3623a3a8e19086eb
Author: Padma Penumarthy
Date: 2018-03-21T20:39:43Z
DRILL-6254: IllegalArgumentException: the requested size must be
non-negative
> IllegalArgumentException: the requested size must be non-negative
> -
>
> Key: DRILL-6254
> URL: https://issues.apache.org/jira/browse/DRILL-6254
> Project: Apache Drill
> Issue Type: Bug
> Components: Execution - Flow
>Affects Versions: 1.13.0
>Reporter: Khurram Faraaz
>Assignee: Padma Penumarthy
>Priority: Major
> Fix For: 1.14.0
>
> Attachments: genAllTypesJSN.py
>
>
> Flatten query fails due to IllegalArgumentException: the requested size must
> be non-negative.
> Script to generate JSON data file is attached here.
> {noformat}
> 0: jdbc:drill:schema=dfs.tmp> CREATE TABLE tbl_all_types_jsn_to_parquet AS
> . . . . . . . . . . . . . . > SELECT
> . . . . . . . . . . . . . . > CAST( col_int AS INT) col_int,
> . . . . . . . . . . . . . . > CAST( col_bigint AS BIGINT) col_bigint,
> . . . . . . . . . . . . . . > CAST( col_char AS CHAR(10)) col_char,
> . . . . . . . . . . . . . . > CAST( col_fxdln_str AS VARCHAR(256))
> col_fxdln_str,
> . . . . . . . . . . . . . . > CAST( col_varln_str AS VARCHAR(256))
> col_varln_str,
> . . . . . . . . . . . . . . > CAST( col_float AS FLOAT) col_float,
> . . . . . . . . . . . . . . > CAST( col_double AS DOUBLE PRECISION)
> col_double,
> . . . . . . . . . . . . . . > CAST( col_date AS DATE) col_date,
> . . . . . . . . . . . . . . > CAST( col_time AS TIME) col_time,
> . . . . . . . . . . . . . . > CAST( col_tmstmp AS TIMESTAMP) col_tmstmp,
> . . . . . . . . . . . . . . > CAST( col_boolean AS BOOLEAN) col_boolean,
> . . . . . . . . . . . . . . > col_binary,
> . . . . . . . . . . . . . . > array_of_ints from `all_supported_types.json`;
> +---++
> | Fragment | Number of records written |
> +---++
> | 0_0 | 9 |
> +---++
> 1 row selected (0.29 seconds)
> {noformat}
> Reset all options and set slice_target=1
> alter system reset all;
> alter system set `planner.slice_target`=1;
> output_batch_size was set to its default value
> drill.exec.memory.operator.output_batch_size = 16777216
>
> {noformat}
> select *, flatten(array_of_ints) from tbl_all_types_jsn_to_parquet;
> Error: SYSTEM ERROR: IllegalArgumentException: the requested size must be
> non-negative
> Fragment 0:0
> [Error Id: 480bae96-ae89-45a7-b937-011c0f87c14d on qa102-45.qa.lab:31010]
> (state=,code=0)
> 0: jdbc:drill:schema=dfs.tmp>
> {noformat}
> Stack trace from drillbit.log
> {noformat}
> 2018-03-15 12:19:43,916 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.drill.exec.work.foreman.Foreman - Query text for query id
> 255538af-bcd5-98ee-32e0-68d98fc4a6fa: select *, flatten(array_of_ints) from
> tbl_all_types_jsn_to_parquet
> 2018-03-15 12:19:43,952 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2018-03-15 12:19:43,953 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.dfs.FileSelection - FileSelection.getStatuses() took 0 ms,
> numFiles: 1
> 2018-03-15 12:19:43,966 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Took 0 ms to get file statuses
> 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.exec.store.parquet.Metadata - Fetch parquet metadata: Executed 1 out of
> 1 using 1 threads. Time: 2ms total, 2.927366ms avg, 2ms max.
> 2018-03-15 12:19:43,969 [255538af-bcd5-98ee-32e0-68d98fc4a6fa:foreman] INFO
> o.a.d.ex
