[
https://issues.apache.org/jira/browse/SPARK-16076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hyukjin Kwon updated SPARK-16076:
---------------------------------
Labels: bulk-closed (was: )
> Dataset - outer join nulls can sometimes combinate to default values
> --------------------------------------------------------------------
>
> Key: SPARK-16076
> URL: https://issues.apache.org/jira/browse/SPARK-16076
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.0.0
> Reporter: Richard Marscher
> Priority: Major
> Labels: bulk-closed
>
> Recently joinWith behavior was tweaked to properly pass through missing rows
> in an outer join to preserve it as a null. But circumstantially calling some
> combinator functions afterwards can undo this and cause the data to revert to
> default values.
> I have a reproducing Databricks notebook where two Dataset of type (Int, Int)
> are joined on the first element in the tuples. If a `map` call is made after
> the joinWith, the map call lifts the null values into default values for the
> given datatype (-1 for Int).
> https://databricks-prod-cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f173bcfc/160347920874755/4268263383756277/673639177603143/latest.html
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
