[
https://issues.apache.org/jira/browse/PIG-1748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12985030#action_12985030
]
Daniel Dai commented on PIG-1748:
---------------------------------
To Jokob:
I get 2 failure in TestAvroStorage: testArrayWithSame,
testRecordWithFieldSchema. Error message is similar:
could not instantiate 'org.apache.pig.piggybank.storage.avro.AvroStorage' with
arguments '[same,
src/test/java/org/apache/pig/piggybank/test/storage/avro/avro_test_files/test_array.avro]'
java.lang.RuntimeException: could not instantiate
'org.apache.pig.piggybank.storage.avro.AvroStorage' with arguments '[same,
src/test/java/org/apache/pig/piggybank/test/storage/avro/avro_test_files/test_array.avro]'
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:494)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.NonEvalFuncSpec(QueryParser.java:5660)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.StoreClause(QueryParser.java:4034)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.BaseExpr(QueryParser.java:1501)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Expr(QueryParser.java:1013)
at
org.apache.pig.impl.logicalLayer.parser.QueryParser.Parse(QueryParser.java:825)
at
org.apache.pig.impl.logicalLayer.LogicalPlanBuilder.parse(LogicalPlanBuilder.java:63)
at org.apache.pig.PigServer$Graph.parseQuery(PigServer.java:1708)
at org.apache.pig.PigServer$Graph.registerQuery(PigServer.java:1658)
at org.apache.pig.PigServer.registerQuery(PigServer.java:546)
at org.apache.pig.PigServer.registerQuery(PigServer.java:570)
at
org.apache.pig.piggybank.test.storage.avro.TestAvroStorage.testAvroStorage(Unknown
Source)
at
org.apache.pig.piggybank.test.storage.avro.TestAvroStorage.testArrayWithSame(Unknown
Source)
Caused by: java.lang.reflect.InvocationTargetException
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.pig.impl.PigContext.instantiateFuncFromSpec(PigContext.java:484)
Caused by: java.net.ConnectException: Call to
localhost.localdomain/127.0.0.1:57284 failed on connection exception:
java.net.ConnectException: Connection refused
at org.apache.hadoop.ipc.Client.wrapException(Client.java:767)
at org.apache.hadoop.ipc.Client.call(Client.java:743)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:220)
at $Proxy9.getProtocolVersion(Unknown Source)
at org.apache.hadoop.ipc.RPC.getProxy(RPC.java:359)
at
org.apache.hadoop.hdfs.DFSClient.createRPCNamenode(DFSClient.java:106)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:207)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:170)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:82)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1378)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:66)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1390)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:196)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:95)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:180)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.init(AvroStorage.java:372)
at
org.apache.pig.piggybank.storage.avro.AvroStorage.<init>(AvroStorage.java:110)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:574)
at
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:404)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:304)
at org.apache.hadoop.ipc.Client$Connection.access$1700(Client.java:176)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:860)
at org.apache.hadoop.ipc.Client.call(Client.java:720)
Know what happen?
> Add load/store function AvroStorage for avro data
> -------------------------------------------------
>
> Key: PIG-1748
> URL: https://issues.apache.org/jira/browse/PIG-1748
> Project: Pig
> Issue Type: Improvement
> Components: impl
> Reporter: lin guo
> Assignee: Jakob Homan
> Attachments: avro_storage.patch, avro_test_files.tar.gz,
> PIG-1748-2.patch
>
>
> We want to use Pig to process arbitrary Avro data and store results as Avro
> files. AvroStorage() extends two PigFuncs: LoadFunc and StoreFunc.
> Due to discrepancies of Avro and Pig data models, AvroStorage has:
> 1. Limited support for "record": we do not support recursively defined record
> because the number of fields in such records is data dependent.
> 2. Limited support for "union": we only accept nullable union like ["null",
> "some-type"].
> For simplicity, we also make the following assumptions:
> If the input directory is a leaf directory, then we assume Avro data files in
> it have the same schema;
> If the input directory contains sub-directories, then we assume Avro data
> files in all sub-directories have the same schema.
> AvroStorage takes no input parameters when used as a LoadFunc (except for
> "debug [debug-level]").
> Users can provide parameters to AvroStorage when used as a StoreFunc. If they
> don't, Avro schema of output data is derived from its
> Pig schema.
> Detailed documentation can be found in
> http://snaprojects.jira.com/wiki/display/HTOOLS/AvroStorage+-+Pig+support+for+Avro+data
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
