[jira] [Updated] (SPARK-46092) Overflow in Parquet row group filter creation causes incorrect results
[
https://issues.apache.org/jira/browse/SPARK-46092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-46092:
--
Fix Version/s: 3.3.4
> Overflow in Parquet row group filter creation causes incorrect results
> --
>
> Key: SPARK-46092
> URL: https://issues.apache.org/jira/browse/SPARK-46092
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Johan Lasperas
>Assignee: Johan Lasperas
>Priority: Major
> Labels: correctness, pull-request-available
> Fix For: 4.0.0, 3.5.1, 3.3.4, 3.4.3
>
>
> While the parquet readers don't support reading parquet values into larger
> Spark types, it's possible to trigger an overflow when creating a Parquet row
> group filter that will then incorrectly skip row groups and bypass the
> exception in the reader,
> Repro:
> {code:java}
> Seq(0).toDF("a").write.parquet(path)
> spark.read.schema("a LONG").parquet(path).where(s"a <
> ${Long.MaxValue}").collect(){code}
> This succeeds and returns no results. This should either fail if the Parquet
> reader doesn't support the upcast from int to long or produce result `[0]` if
> it does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46092) Overflow in Parquet row group filter creation causes incorrect results
[
https://issues.apache.org/jira/browse/SPARK-46092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-46092:
--
Fix Version/s: 3.4.3
> Overflow in Parquet row group filter creation causes incorrect results
> --
>
> Key: SPARK-46092
> URL: https://issues.apache.org/jira/browse/SPARK-46092
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Johan Lasperas
>Assignee: Johan Lasperas
>Priority: Major
> Labels: correctness, pull-request-available
> Fix For: 4.0.0, 3.5.1, 3.4.3
>
>
> While the parquet readers don't support reading parquet values into larger
> Spark types, it's possible to trigger an overflow when creating a Parquet row
> group filter that will then incorrectly skip row groups and bypass the
> exception in the reader,
> Repro:
> {code:java}
> Seq(0).toDF("a").write.parquet(path)
> spark.read.schema("a LONG").parquet(path).where(s"a <
> ${Long.MaxValue}").collect(){code}
> This succeeds and returns no results. This should either fail if the Parquet
> reader doesn't support the upcast from int to long or produce result `[0]` if
> it does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46092) Overflow in Parquet row group filter creation causes incorrect results
[
https://issues.apache.org/jira/browse/SPARK-46092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-46092:
--
Fix Version/s: 3.5.1
> Overflow in Parquet row group filter creation causes incorrect results
> --
>
> Key: SPARK-46092
> URL: https://issues.apache.org/jira/browse/SPARK-46092
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Johan Lasperas
>Assignee: Johan Lasperas
>Priority: Major
> Labels: correctness, pull-request-available
> Fix For: 4.0.0, 3.5.1
>
>
> While the parquet readers don't support reading parquet values into larger
> Spark types, it's possible to trigger an overflow when creating a Parquet row
> group filter that will then incorrectly skip row groups and bypass the
> exception in the reader,
> Repro:
> {code:java}
> Seq(0).toDF("a").write.parquet(path)
> spark.read.schema("a LONG").parquet(path).where(s"a <
> ${Long.MaxValue}").collect(){code}
> This succeeds and returns no results. This should either fail if the Parquet
> reader doesn't support the upcast from int to long or produce result `[0]` if
> it does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46092) Overflow in Parquet row group filter creation causes incorrect results
[
https://issues.apache.org/jira/browse/SPARK-46092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-46092:
---
Labels: correctness pull-request-available (was: pull-request-available)
> Overflow in Parquet row group filter creation causes incorrect results
> --
>
> Key: SPARK-46092
> URL: https://issues.apache.org/jira/browse/SPARK-46092
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Johan Lasperas
>Priority: Major
> Labels: correctness, pull-request-available
>
> While the parquet readers don't support reading parquet values into larger
> Spark types, it's possible to trigger an overflow when creating a Parquet row
> group filter that will then incorrectly skip row groups and bypass the
> exception in the reader,
> Repro:
> {code:java}
> Seq(0).toDF("a").write.parquet(path)
> spark.read.schema("a LONG").parquet(path).where(s"a <
> ${Long.MaxValue}").collect(){code}
> This succeeds and returns no results. This should either fail if the Parquet
> reader doesn't support the upcast from int to long or produce result `[0]` if
> it does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46092) Overflow in Parquet row group filter creation causes incorrect results
[
https://issues.apache.org/jira/browse/SPARK-46092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-46092:
---
Labels: pull-request-available (was: )
> Overflow in Parquet row group filter creation causes incorrect results
> --
>
> Key: SPARK-46092
> URL: https://issues.apache.org/jira/browse/SPARK-46092
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Johan Lasperas
>Priority: Major
> Labels: pull-request-available
>
> While the parquet readers don't support reading parquet values into larger
> Spark types, it's possible to trigger an overflow when creating a Parquet row
> group filter that will then incorrectly skip row groups and bypass the
> exception in the reader,
> Repro:
> {code:java}
> Seq(0).toDF("a").write.parquet(path)
> spark.read.schema("a LONG").parquet(path).where(s"a <
> ${Long.MaxValue}").collect(){code}
> This succeeds and returns no results. This should either fail if the Parquet
> reader doesn't support the upcast from int to long or produce result `[0]` if
> it does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-46092) Overflow in Parquet row group filter creation causes incorrect results
[
https://issues.apache.org/jira/browse/SPARK-46092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Johan Lasperas updated SPARK-46092:
---
Description:
While the parquet readers don't support reading parquet values into larger
Spark types, it's possible to trigger an overflow when creating a Parquet row
group filter that will then incorrectly skip row groups and bypass the
exception in the reader,
Repro:
{code:java}
Seq(0).toDF("a").write.parquet(path)
spark.read.schema("a LONG").parquet(path).where(s"a <
${Long.MaxValue}").collect(){code}
This succeeds and returns no results. This should either fail if the Parquet
reader doesn't support the upcast from int to long or produce result `[0]` if
it does.
was:
While the parquet readers don't support reading parquet values into larger
Spark types, it's possible to trigger an overflow when creating a Parquet row
group filter that will then incorrectly skip row groups and bypass the
exception in the reader,
Repro:
```
Seq(0).toDF("a").write.parquet(path)
spark.read.schema("a LONG").parquet(path).where(s"a <
${Long.MaxValue}").collect()
```
This succeeds and returns no results. This should either fail if the Parquet
reader doesn't support the upcast from int to long or produce result `[0]` if
it does.
> Overflow in Parquet row group filter creation causes incorrect results
> --
>
> Key: SPARK-46092
> URL: https://issues.apache.org/jira/browse/SPARK-46092
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 3.5.0
>Reporter: Johan Lasperas
>Priority: Major
>
> While the parquet readers don't support reading parquet values into larger
> Spark types, it's possible to trigger an overflow when creating a Parquet row
> group filter that will then incorrectly skip row groups and bypass the
> exception in the reader,
> Repro:
> {code:java}
> Seq(0).toDF("a").write.parquet(path)
> spark.read.schema("a LONG").parquet(path).where(s"a <
> ${Long.MaxValue}").collect(){code}
> This succeeds and returns no results. This should either fail if the Parquet
> reader doesn't support the upcast from int to long or produce result `[0]` if
> it does.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
