Subroto Sanyal created HIVE-4986:
------------------------------------
Summary: Problem in creating configration in
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils
Key: HIVE-4986
URL: https://issues.apache.org/jira/browse/HIVE-4986
Project: Hive
Issue Type: Bug
Reporter: Subroto Sanyal
While using AvroSerDe but, I am facing the problem:
{noformat}
java.lang.IllegalArgumentException: Wrong FS:
hdfs://xxx:8020/abc/bbc/test.avsc, expected: file:///
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:381)
at
org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:55)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:393)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:251)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.<init>(ChecksumFileSystem.java:125)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:427)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.getSchemaFromHDFS(AvroSerdeUtils.java:105)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:70)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrReturnErrorSchema(AvroSerdeUtils.java:87)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:59)
at test.subroto.HiveUtil.getDeserializer(HiveUtil.java:102)
{noformat}
While debugging the problem I saw a piece of code in:
{code:title=org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.java|borderStyle=solid}
public static Schema More ...determineSchemaOrThrowException(Properties
properties)
throws IOException, AvroSerdeException {
String schemaString = properties.getProperty(SCHEMA_LITERAL);
if(schemaString != null && !schemaString.equals(SCHEMA_NONE))
return Schema.parse(schemaString);
// Try pulling directly from URL
schemaString = properties.getProperty(SCHEMA_URL);
if(schemaString == null || schemaString.equals(SCHEMA_NONE))
throw new AvroSerdeException(EXCEPTION_MESSAGE);
try {
if(schemaString.toLowerCase().startsWith("hdfs://"))
return getSchemaFromHDFS(schemaString, new Configuration());
} catch(IOException ioe) {
throw new AvroSerdeException("Unable to read schema from HDFS: " +
schemaString, ioe);
}
return Schema.parse(new URL(schemaString).openStream());
}
{code}
Here the configuration passed from the Deserilaizer is completely ignored and a
new Configuration object is created.
In my opinion the AvroSerDe should honor the Configuration being passed to
Deserializer interface ; rather than creating a new one.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
