[
https://issues.apache.org/jira/browse/SPARK-32486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Prashant Sharma updated SPARK-32486:
------------------------------------
Priority: Major (was: Blocker)
> Issue with deserialization and persist api in latest spark java versions
> ------------------------------------------------------------------------
>
> Key: SPARK-32486
> URL: https://issues.apache.org/jira/browse/SPARK-32486
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 2.4.4, 2.4.5, 2.4.6, 3.0.0
> Environment: It's happening on all the os and java8
> Reporter: Dinesh Kumar
> Priority: Major
>
> Hey Team, We have class level object instantiations in one of our Classes.
> When we want to persist that data into the Dataset of this class Type it's
> not persisting the null values instead it's taking class level precedence.
> i.e. It's showing as new object.
> Eg:
> _Test.class has below class level attributes:_
> _private Test1 testNumber = new Test1();_
> _private Test2 testNumber2;_
>
> String inputLocation = "src/test/resources/pipeline/test.parquet";
> Dataset<Row> ds = this.session.read().parquet(inputLocation);
> ds.printSchema();
> ds.foreach(input->{
> System.out.println(input); // When we verified it's showing testNumber,
> testNumber2 as null
> });
> Dataset<Test> inputDataSet = ds.as(Encoders.bean(Test.class));
> inputDataSet.foreach(input->{
> System.out.println(input); // When we verified it's showing testNumber as
> new Test1(), testNumber2 as null
> });
>
>
> This is the same issue with dataset.persist() call aswell. It is happening
> with all 2.4.4 and higher versions. Can you please fix it?
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
