Hi Doug,
Thanks for your reply. The avro version I use is 1.4.1. I checked the
schema of avro file and found the name is "Weather" with the namespace
"test".
So, I changed the name of schema to "Weather" without namespace and
made it consistent among all the related files. However, I still see
the same errors. Please see the below.
Tatsuya
[1: schema of avro file]
java -jar /usr/lib/hadoop/lib/avro-tools-1.4.1.jar getschema weather.avro
{
"type" : "record",
"name" : "Weather",
"fields" : [ {
"name" : "station",
"type" : "string"
}, {
"name" : "time",
"type" : "int"
}, {
"name" : "temp",
"type" : "int"
} ]
}
[2: put the avro file to hdfs]
hadoop fs -put weather.avro /user/mori/
[3: the jason file I use]
{"type": "record",
"name": "Weather",
"doc": "A weather reading.",
"fields": [
{"name": "station", "type": "string", "order": "ignore"},
{"name": "time", "type": "int"},
{"name": "temp", "type": "int"}
]
}
[4: compile avsc to Java class (Weather.java)]
java -jar lib/avro-tools-1.4.1.jar compile schema weather.avsc src
[5: part of Weather.java]
public class Weather extends
org.apache.avro.specific.SpecificRecordBase implements
org.apache.avro.specific.SpecificRecord {
public static final org.apache.avro.Schema SCHEMA$ =
org.apache.avro.Schema.parse("{\"type\":\"record\",\"name\":\"Weather\",\"fields\":[{\"name\":\"station\",\"type\":\"string\",\"order\":\"ignore\"},{\"name\":\"time\",\"type\":\"int\"},{\"name\":\"temp\",\"type\":\"int\"}]}");
[6: compile all the classes]
javac -d classes -cp
lib/avro-1.4.1.jar:lib/avro-tools-1.4.1.jar:$HADOOP_HOME/hadoop-0.20.1+169.113/hadoop-0.20.1+169.113-*.jar
src/TestWeather.java src/Weather.java
[7: execute the hadoop job] => same errors
hadoop jar TestWeather.jar TestWeather
Exception in thread "main" java.lang.ClassCastException:
org.apache.avro.generic.GenericData$Record cannot be cast to Weather
at TestWeather.main(TestWeather.java:104)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:616)
at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
On Tue, Jan 11, 2011 at 5:34 AM, Doug Cutting <[email protected]> wrote:
> Tatsuya,
>
> Are you sure that the schema in weather.avro file is named "test.Weather"?
> In trunk, if SpecificDatumReader cannot find the class named in a schema,
> it will use GenericData$Record. What version of Avro are you using?
>
> Doug
>
> On 01/08/2011 12:35 AM, Tatsuya Mori wrote:
>>
>> Hi,
>>
>> I am trying to read avro data file on HDFS. The following code gives
>> the error attached at the bottom, even though I use
>> SpecificDatumReader. The Weather class is taken from test of
>> org.apache.avro.mapred.
>> Any comments/suggestions are highly appreciated. Thanks.
>>
>> Tatsuya
>>
>> [code]
>> String input = "hdfs:/user/mori/weather.avro";
>> Path inPath = new Path(input);
>>
>> FileSystem fs = FileSystem.get(URI.create(input),
>> new Configuration());
>> BufferedInputStream inStream =
>> new BufferedInputStream(fs.open(inPath));
>>
>> DatumReader<Weather> reader =
>> new SpecificDatumReader<Weather>(Weather.class);
>> DataFileStream<Weather> fileReader =
>> new DataFileStream<Weather>(inStream, reader);
>>
>> while (fileReader.hasNext()) {
>> Weather w = (Weather) FileReader.next();
>> System.out.printf("%s\n", w.station);
>> }
>>
>> [error]
>> Exception in thread "main" java.lang.ClassCastException:
>> org.apache.avro.generic.GenericData$Record cannot be cast to
>> test.Weather
>> at TestWeather.main(TestWeather.java:103)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:616)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:156)
>
