[
https://issues.apache.org/jira/browse/SPARK-49727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Asif updated SPARK-49727:
-------------------------
Description:
If a Bean class contains a serializable POJO ( without any getter/setter)
property, the encoder does not convert it into Binary Type, and looses the
information.
Using that encoder , we can convert the Dataset of Bean to dataframe.
But When the dataframe is converted back to dataset, the serializable Pojo
field in the bean remains null.
The issue is addressed along with the PR for issue
[https://issues.apache.org/jira/browse/SPARK-46679|https://issues.apache.org/jira/browse/SPARK-46679]
Will be adding the bug test in the PR
was:
If a Bean class contains a serializable POJO ( without any getter/setter)
property, the encoder does not convert it into Binary Type, and looses the
information.
Using that encoder , we can convert the Dataset of Bean to dataframe.
But When the dataframe is converted back to dataset, the serializable Pojo
field in the bean remains null.
> A Bean class with a serializable POJO field looses data when converted back
> from dataframe, as dataset
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-49727
> URL: https://issues.apache.org/jira/browse/SPARK-49727
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.0.0, 3.5.2
> Reporter: Asif
> Priority: Major
>
> If a Bean class contains a serializable POJO ( without any getter/setter)
> property, the encoder does not convert it into Binary Type, and looses the
> information.
> Using that encoder , we can convert the Dataset of Bean to dataframe.
> But When the dataframe is converted back to dataset, the serializable Pojo
> field in the bean remains null.
> The issue is addressed along with the PR for issue
> [https://issues.apache.org/jira/browse/SPARK-46679|https://issues.apache.org/jira/browse/SPARK-46679]
> Will be adding the bug test in the PR
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
