GitHub user kanzhang opened a pull request:
https://github.com/apache/spark/pull/1916
[SPARK-2736] PySpark converter and example script for reading Avro files
This patch includes:
1. An Avro converter that converts Avro data types to Python. It handles
all 3 Avro data mappings (Generic, Specific and Reflect).
2. An example Python script for reading Avro files using AvroKeyInputFormat
and the converter.
3. Fixing a classloading issue.
cc @MLnick @JoshRosen @mateiz
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kanzhang/spark SPARK-2736
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/1916.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1916
----
commit 536876bb813baa104081c6e594963b3457a13da3
Author: Kan Zhang <[email protected]>
Date: 2014-08-06T01:20:53Z
[SPARK-2736] Adding Avro to Java converter
commit 2271a5bb865bb90cf7eaf6cfcdbc2c06144321c5
Author: Kan Zhang <[email protected]>
Date: 2014-08-07T21:42:16Z
[SPARK-2736] Using the right class loader to find Avro classes
commit c8e5881767310f61b8efd57a3d578484f9fdc7ee
Author: Kan Zhang <[email protected]>
Date: 2014-08-10T18:15:21Z
[SPARK-2736] Trying to work with all 3 Avro data models
commit 0be77619900d589e814952181264a833cb185f64
Author: Kan Zhang <[email protected]>
Date: 2014-08-11T23:28:40Z
[SPARK-2736] Example pyspark script and data files
commit 82cc505a7ebc6b5082207ac9c560c9793a470c5c
Author: Kan Zhang <[email protected]>
Date: 2014-08-13T02:03:31Z
[SPARK-2736] Update data sample
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
