[jira] [Commented] (SPARK-22222) Fix the ARRAY_MAX in BufferHolder and add a test
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16244455#comment-16244455 ] Apache Spark commented on SPARK-2: -- User 'dongjoon-hyun' has created a pull request for this issue: https://github.com/apache/spark/pull/19697 > Fix the ARRAY_MAX in BufferHolder and add a test > > > Key: SPARK-2 > URL: https://issues.apache.org/jira/browse/SPARK-2 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: Feng Liu >Assignee: Feng Liu > Fix For: 2.3.0 > > > This is actually a followup for SPARK-22033, which set the `ARRAY_MAX` to > `Int.MaxValue - 8`. It is not a valid number because it will cause the > following line fail when such a large byte array is allocated: > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/BufferHolder.java#L86 > We need to make sure the new length is a multiple of 8. > We need to add one test for the fix. Note that the test should work > independently with the heap size of the test JVM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22222) Fix the ARRAY_MAX in BufferHolder and add a test
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16197620#comment-16197620 ] Apache Spark commented on SPARK-2: -- User 'liufengdb' has created a pull request for this issue: https://github.com/apache/spark/pull/19460 > Fix the ARRAY_MAX in BufferHolder and add a test > > > Key: SPARK-2 > URL: https://issues.apache.org/jira/browse/SPARK-2 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: Feng Liu > > This is actually a followup for SPARK-22033, which set the `ARRAY_MAX` to > `Int.MaxValue - 8`. It is not a valid number because it will cause the > following line fail when such a large byte array is allocated: > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/BufferHolder.java#L86 > We need to make sure the new length is a multiple of 8. > We need to add one test for the fix. Note that the test should work > independently with the heap size of the test JVM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-22222) Fix the ARRAY_MAX in BufferHolder and add a test
[ https://issues.apache.org/jira/browse/SPARK-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16196564#comment-16196564 ] Sean Owen commented on SPARK-2: --- Ah, I see. I think we'd have to round this down to Int.MaxValue - 15 in this case then, and any others introduced in that change where the array length would matter. > Fix the ARRAY_MAX in BufferHolder and add a test > > > Key: SPARK-2 > URL: https://issues.apache.org/jira/browse/SPARK-2 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.1 >Reporter: Feng Liu > > This is actually a followup for SPARK-22033, which set the `ARRAY_MAX` to > `Int.MaxValue - 8`. It is not a valid number because it will cause the > following line fail when such a large byte array is allocated: > https://github.com/apache/spark/blob/master/sql/catalyst/src/main/java/org/apache/spark/sql/catalyst/expressions/codegen/BufferHolder.java#L86 > We need to make sure the new length is a multiple of 8. > We need to add one test for the fix. Note that the test should work > independently with the heap size of the test JVM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
