I was researching my ATLAS-381 errors and came across possibly a more
fundamental problem. My avro tables define their schema by reference to a
schema file on HDFS. The DDL includes:
TBLPROPERTIES
('avro.schema.url'='hdfs://redstackambari.vagrant.tgt:8020/user/vagrant/schema/foo.avsc');
When the bridge tries to read that schema I get the error below. The fact that
AvroSerdeUtils is calling java.net.URL() with the hdfs://... string means that
it could find load the schema directly:
https://github.com/apache/hive/blob/branch-1.2/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java#L115
>From past experiences, I would guess that the attempt to get the HDFS file
>system here isn't working due to kerberos authentication issues:
https://github.com/apache/hive/blob/branch-1.2/serde/src/java/org/apache/hadoop/hive/serde2/avro/AvroSerdeUtils.java#L131
I am trying to debug this further, but does anyone have other thoughts on why I
may be getting this error?
Thanks! -Aaron
org.apache.hadoop.hive.serde2.avro.AvroSerdeException: Unable to read schema
from given path:
hdfs://redstackambari.vagrant.tgt:8020/user/vagrant/schema/foo.avsc
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:119)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.determineSchemaOrReturnErrorSchema(AvroSerDe.java:167)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:103)
at
org.apache.hadoop.hive.serde2.avro.AvroSerDe.initialize(AvroSerDe.java:80)
at
org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDe(SerDeUtils.java:521)
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.getDeserializer(MetaStoreUtils.java:391)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:276)
at
org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:258)
at org.apache.hadoop.hive.ql.metadata.Table.getCols(Table.java:605)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.createTableInstance(HiveMetaStoreBridge.java:271)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.registerTable(HiveMetaStoreBridge.java:302)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importTables(HiveMetaStoreBridge.java:217)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importDatabases(HiveMetaStoreBridge.java:105)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.importHiveMetadata(HiveMetaStoreBridge.java:97)
at
org.apache.atlas.hive.bridge.HiveMetaStoreBridge.main(HiveMetaStoreBridge.java:508)
Caused by: java.net.MalformedURLException: unknown protocol: hdfs
at java.net.URL.<init>(URL.java:592)
at java.net.URL.<init>(URL.java:482)
at java.net.URL.<init>(URL.java:431)
at
org.apache.hadoop.hive.serde2.avro.AvroSerdeUtils.determineSchemaOrThrowException(AvroSerdeUtils.java:115)
... 14 more
