[
https://issues.apache.org/jira/browse/SPARK-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Josh Rosen updated SPARK-9139:
------------------------------
Description:
SQL's DataType.fromJson is a public API and thus must be backwards-compatible;
there are also backwards-compatibility concerns related to persistence of
DataType JSON in metastores.
Unfortunately, we do not have any backwards-compatibility tests which attempt
to read old JSON values that were written by earlier versions of Spark.
DataTypeSuite has "roundtrip" tests that test fromJson(toJson(foo)), but this
doesn't ensure compatibility.
I think that we should address this by capuring the JSON strings produced in
Spark 1.3's DataFrameSuite and adding test cases that try to create DataTypes
from those strings.
This might be a good starter task for someone who wants to contribute to SQL
tests.
was:
SQL's DataType.fromJson is a public API and thus must be backwards-compatible;
there are also backwards-compatibility concerns related to persistence of
DataType JSON in metastores.
Unfortunately, we do not have any backwards-compatibility tests which attempt
to read old JSON values that were written by earlier versions of Spark.
DataTypeSuite has "roundtrip" tests that test fromJson(toJson(x)), but this
doesn't ensure compatibility.
I think that we should address this by capuring the JSON strings produced in
Spark 1.3's DataFrameSuite and adding test cases that try to create DataTypes
from those strings.
This might be a good starter task for someone who wants to contribute to SQL
tests.
> Add backwards-compatibility tests for DataType.fromJson()
> ---------------------------------------------------------
>
> Key: SPARK-9139
> URL: https://issues.apache.org/jira/browse/SPARK-9139
> Project: Spark
> Issue Type: Test
> Components: SQL
> Reporter: Josh Rosen
> Priority: Critical
>
> SQL's DataType.fromJson is a public API and thus must be
> backwards-compatible; there are also backwards-compatibility concerns related
> to persistence of DataType JSON in metastores.
> Unfortunately, we do not have any backwards-compatibility tests which attempt
> to read old JSON values that were written by earlier versions of Spark.
> DataTypeSuite has "roundtrip" tests that test fromJson(toJson(foo)), but this
> doesn't ensure compatibility.
> I think that we should address this by capuring the JSON strings produced in
> Spark 1.3's DataFrameSuite and adding test cases that try to create DataTypes
> from those strings.
> This might be a good starter task for someone who wants to contribute to SQL
> tests.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
