GitHub user yhuai opened a pull request:
https://github.com/apache/spark/pull/14249
[SPARK-16515][SQL]set default record reader and writer for script
transformation
## What changes were proposed in this pull request?
In ScriptInputOutputSchema, we read default RecordReader and RecordWriter
from conf. Since Spark 2.0 has deleted those config keys from hive conf, we
have to set default reader/writer class name by ourselves. Otherwise we will
get None for LazySimpleSerde, the data written would not be able to read by
script. The test case added worked fine with previous version of Spark, but
would fail now.
## How was this patch tested?
added a test case in SQLQuerySuite.
Closes #14169
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yhuai/spark scriptTransformation
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/14249.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #14249
----
commit e4c7e027e9216bad42ac85916de05e26163452ea
Author: Daoyuan Wang <[email protected]>
Date: 2016-07-13T10:29:48Z
set default record reader and writer in script transformation
commit 8048732c56bd43dd0ca175a0a047a495ab06fcea
Author: Daoyuan Wang <[email protected]>
Date: 2016-07-18T07:45:28Z
address comment
commit 814f67926038ef136efa12b8e41d8c252411b9c9
Author: Daoyuan Wang <[email protected]>
Date: 2016-07-18T07:45:56Z
Update SparkSqlParser.scala
commit 0edfed48a1e6a438a18d488404b900a51475a2d5
Author: Daoyuan Wang <[email protected]>
Date: 2016-07-18T07:46:56Z
Update SparkSqlParser.scala
commit e999f2c381fbd7c25695b16a38b97a9518e107fc
Author: Yin Huai <[email protected]>
Date: 2016-07-18T18:54:55Z
Update
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
