Hi Chetan,
Sadly, you can not; Spark is configured to ignore the null values when
writing JSON. (check JacksonMessageWriter and find
JsonInclude.Include.NON_NULL from the code.) If you want that
functionality, it would be much better to file the problem to JIRA.
Best,
Dongjin
On Mon, Mar 20,
Exactly.
On Sat, Mar 11, 2017 at 1:35 PM, Dongjin Lee wrote:
> Hello Chetan,
>
> Could you post some code? If I understood correctly, you are trying to
> save JSON like:
>
> {
> "first_name": "Dongjin",
> "last_name: null
> }
>
> not in omitted form, like:
>
> {
>
Hello Chetan,
Could you post some code? If I understood correctly, you are trying to save
JSON like:
{
"first_name": "Dongjin",
"last_name: null
}
not in omitted form, like:
{
"first_name": "Dongjin"
}
right?
- Dongjin
On Wed, Mar 8, 2017 at 5:58 AM, Chetan Khatri
Hello Dev / Users,
I am working with PySpark Code migration to scala, with Python - Iterating
Spark with dictionary and generating JSON with null is possible with
json.dumps() which will be converted to SparkSQL[Row] but in scala how can
we generate json will null values as a Dataframe ?
Thanks.