Thanks Yijie! Also cc the user list.
Cheng
On 4/13/15 9:19 AM, Yijie Shen wrote:
I opened a new Parquet JIRA ticket here:
https://issues.apache.org/jira/browse/PARQUET-251
Yijie
On April 12, 2015 at 11:48:57 PM, Cheng Lian (lian.cs....@gmail.com
<mailto:lian.cs....@gmail.com>) wrote:
Thanks for reporting this! Would you mind to open JIRA tickets for both
Spark and Parquet?
I'm not sure whether Parquet declares somewhere the user mustn't reuse
byte arrays when using binary type. If it does, then it's a Spark bug.
Anyway, this should be fixed.
Cheng
On 4/12/15 1:50 PM, Yijie Shen wrote:
> Hi,
>
> Suppose I create a dataRDD which extends RDD[Row], and each row is
> GenericMutableRow(Array(Int, Array[Byte])). A same Array[Byte]
object is
> reused among rows but has different content each time. When I
convert it to
> a dataFrame and save it as Parquet File, the file's row group
statistic(max
> & min) of Binary column would be wrong.
>
>
>
> Here is the reason: In Parquet, BinaryStatistic just keep max & min as
> parquet.io.api.Binary references, Spark sql would generate a new
Binary
> backed by the same Array[Byte] passed from row.
> reference backed max:
Binary---------->ByteArrayBackedBinary---------->
> Array[Byte]
>
> Therefore, each time parquet updating row group's statistic, max & min
> would always refer to the same Array[Byte], which has new content each
> time. When parquet decides to save it into file, the last row's
content
> would be saved as both max & min.
>
>
>
> It seems it is a parquet bug because it's parquet's responsibility to
> update statistics correctly.
> But not quite sure. Should I report it as a bug in parquet JIRA?
>
>
> The spark JIRA is https://issues.apache.org/jira/browse/SPARK-6859
>
