Github user ericgarcia commented on the pull request:
https://github.com/apache/spark/pull/455#issuecomment-50541143
@rjurney For some reason I can't push my changes to a tagged release in
github. Here's the packaged distribution that I built:
https://dl.dropboxusercontent.com/u/145006/spark-1.0.0-bin-1.0.4.tgz it's built
for hadoop 1.0
If you want to create your own modified 1.0.0 release, what i did was:
check out the tagged 1.0.0 release https://github.com/apache/spark/tree/v1.0.0
and also check out this pull request and then copy the files from this pull
request into the corresponding directories in the 1.0.0 release. Then create
`org.apache.spark.api.python.AvroGenericConverter` in the same directory as the
other `org.apache.spark.api.python` namespace files, change the namespace on
`AvroGenericConverter` in the code accordingly, and build the project with
`./make_distribution.sh`
The modified code for loading the avro files in pyspark is
avroRdd = sc.newAPIHadoopFile(filename,
"org.apache.avro.mapreduce.AvroKeyInputFormat",
"org.apache.avro.mapred.AvroKey",
"org.apache.hadoop.io.NullWritable",
keyConverter="org.apache.spark.api.python.AvroGenericConverter")
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
