[
https://issues.apache.org/jira/browse/PIG-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sudhir Ranjan updated PIG-3656:
-------------------------------
Description:
Getting exception on storing null valued record/tupple as avro.
The input file having one column with long values (one of them is null means
nothing) and when I am trying to store the data in avro format ,it throws error.
Please suggest if I am missing any thing some where as per the bellow codebase
or else please provide the patch.
input file: /home/hadoop/work/sudhir/AvroAnalysis/input/TSV_uncompressed/part*
content:
2037179309
2037179338
2037179367
2037179433
2037179437
2037179449
2037179547
2037179631
2037179631
2037179309
2037179338
2037179367
2037179433
2037179437
2037179449
2037179547
2037179631
2037179631
******My code base.
REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/snappy-java-1.0.4.1.jar
REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/avro-1.7.5.jar
REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/json-simple-1.1.jar;
REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/piggybank.jar;
REGISTER
/home/hadoop/work/sudhir/AvroAnalysis/Avrojars/jackson-core-asl-1.5.5.jar;
REGISTER
/home/hadoop/work/sudhir/AvroAnalysis/Avrojars/jackson-mapper-asl-1.5.5.jar;
– The input file only have 1 column (normal TEXT data i.e TSV format) and the
file having a null value means nothing
A = load '/home/hadoop/work/sudhir/AvroAnalysis/input/TSV_uncompressed/part*'
using PigStorage('\t') as (USER_ID:long);
– The soutput to be stored in avro data format
STORE A INTO '/home/hadoop/work/sudhir/AvroAnalysis/output/AvroStore/' USING
org.apache.pig.piggybank.storage.avro.AvroStorage('schema','{"namespace":"com.sudhir.schema.users.avro","type":"long","name":"users_avro","doc":"Avro
storing with schema using Pig.","fields":[
{"name":"USER_ID","type":["null","long"],"default":null}
]}');
*******Getting Error like:
INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate
exception from backed error:
org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: null of long
ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
was:
Getting exception on storing null valued record/tupple as avro.
The input file having one column with long values (one of them is null means
nothing) and when I am trying to store the data in avro format ,it throws error.
Please suggest if I am missing any thing some where as per the bellow codebase
or else please provide the patch.
******My code base.
REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/snappy-java-1.0.4.1.jar
REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/avro-1.7.5.jar
REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/json-simple-1.1.jar;
REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/piggybank.jar;
REGISTER
/home/hadoop/work/sudhir/AvroAnalysis/Avrojars/jackson-core-asl-1.5.5.jar;
REGISTER
/home/hadoop/work/sudhir/AvroAnalysis/Avrojars/jackson-mapper-asl-1.5.5.jar;
– The input file only have 1 column (normal TEXT data i.e TSV format) and the
file having a null value means nothing
A = load '/home/hadoop/work/sudhir/AvroAnalysis/input/TSV_uncompressed/part*'
using PigStorage('\t') as (USER_ID:long);
– The soutput to be stored in avro data format
STORE A INTO '/home/hadoop/work/sudhir/AvroAnalysis/output/AvroStore/' USING
org.apache.pig.piggybank.storage.avro.AvroStorage('schema','{"namespace":"com.sudhir.schema.users.avro","type":"long","name":"users_avro","doc":"Avro
storing with schema using Pig.","fields":[
{"name":"USER_ID","type":["null","long"],"default":null}
]}');
*******Getting Error like:
INFO
org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
- 100% complete
ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate
exception from backed error:
org.apache.avro.file.DataFileWriter$AppendWriteException:
java.lang.NullPointerException: null of long
ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
> Exception on storing Null value through AvroStorage using PIG
> -------------------------------------------------------------
>
> Key: PIG-3656
> URL: https://issues.apache.org/jira/browse/PIG-3656
> Project: Pig
> Issue Type: Bug
> Environment: Hadoop 0.20.2-cdh3u5
> Apache Pig version 0.8.1-cdh3u5
> java version "1.6.0_27"
> Reporter: Sudhir Ranjan
>
> Getting exception on storing null valued record/tupple as avro.
> The input file having one column with long values (one of them is null means
> nothing) and when I am trying to store the data in avro format ,it throws
> error.
> Please suggest if I am missing any thing some where as per the bellow
> codebase or else please provide the patch.
> input file: /home/hadoop/work/sudhir/AvroAnalysis/input/TSV_uncompressed/part*
> content:
> 2037179309
> 2037179338
> 2037179367
> 2037179433
> 2037179437
> 2037179449
> 2037179547
> 2037179631
> 2037179631
> 2037179309
> 2037179338
> 2037179367
> 2037179433
> 2037179437
> 2037179449
> 2037179547
> 2037179631
> 2037179631
> ******My code base.
> REGISTER
> /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/snappy-java-1.0.4.1.jar
> REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/avro-1.7.5.jar
> REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/json-simple-1.1.jar;
> REGISTER /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/piggybank.jar;
> REGISTER
> /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/jackson-core-asl-1.5.5.jar;
> REGISTER
> /home/hadoop/work/sudhir/AvroAnalysis/Avrojars/jackson-mapper-asl-1.5.5.jar;
> – The input file only have 1 column (normal TEXT data i.e TSV format) and the
> file having a null value means nothing
> A = load '/home/hadoop/work/sudhir/AvroAnalysis/input/TSV_uncompressed/part*'
> using PigStorage('\t') as (USER_ID:long);
> – The soutput to be stored in avro data format
> STORE A INTO '/home/hadoop/work/sudhir/AvroAnalysis/output/AvroStore/' USING
> org.apache.pig.piggybank.storage.avro.AvroStorage('schema','{"namespace":"com.sudhir.schema.users.avro","type":"long","name":"users_avro","doc":"Avro
> storing with schema using Pig.","fields":[
> {"name":"USER_ID","type":["null","long"],"default":null}
> ]}');
> *******Getting Error like:
> INFO
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
> - 100% complete
> ERROR org.apache.pig.tools.pigstats.PigStats - ERROR 2997: Unable to recreate
> exception from backed error:
> org.apache.avro.file.DataFileWriter$AppendWriteException:
> java.lang.NullPointerException: null of long
> ERROR org.apache.pig.tools.pigstats.PigStatsUtil - 1 map reduce job(s) failed!
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
