bjornjorgensen commented on pull request #35296:
URL: https://github.com/apache/spark/pull/35296#issuecomment-1025144269
@zero323 This PR won't have any impact on your first example because there
is at least one value that is not None. In your second example it will be
without this PR.
`df = ps.DataFrame.from_dict({"id": [1, 2], "val": [np.nan, np.nan]})
df
id val
0 1 NaN
1 2 NaN
df.to_json("testdf.json", num_files=1)
df2 = ps.read_json("testdf.json/*")
df2
id val
0 1 None
1 2 None
df2.to_json("testdf2.json", ignoreNullFields=True, num_files=1)
df3 = ps.read_json("testdf2.json/*")
df3
id
0 1
1 2`
I will change what we agree on, but for now I think this has something to do
with None values.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
