[
https://issues.apache.org/jira/browse/HIVE-8687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14192633#comment-14192633
]
Sushanth Sowmyan commented on HIVE-8687:
----------------------------------------
Since the pre-commit tests will likely run only on the trunk version of the
patch, I ran hcat tests with the 0.14 branch, and had the following results
locally:
{noformat}
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Hive HCatalog ..................................... SUCCESS [ 2.747 s]
[INFO] Hive HCatalog Core ................................ SUCCESS [17:22 min]
[INFO] Hive HCatalog Pig Adapter ......................... SUCCESS [14:11 min]
[INFO] Hive HCatalog Server Extensions ................... SUCCESS [ 20.902 s]
[INFO] Hive HCatalog Webhcat Java Client ................. SUCCESS [ 32.510 s]
[INFO] Hive HCatalog Webhcat ............................. SUCCESS [ 12.214 s]
[INFO] Hive HCatalog Streaming ........................... SUCCESS [03:39 min]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 36:22 min
[INFO] Finished at: 2014-10-31T14:59:08-08:00
[INFO] Final Memory: 46M/135M
{noformat}
The important bits, though, are as follows:
{noformat}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hive.hcatalog.pig.TestE2EScenarios
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.69 sec - in
org.apache.hive.hcatalog.pig.TestE2EScenarios
Running org.apache.hive.hcatalog.pig.TestHCatLoader
Tests run: 60, Failures: 0, Errors: 0, Skipped: 3, Time elapsed: 216.043 sec -
in org.apache.hive.hcatalog.pig.TestHCatLoader
Running org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema
Tests run: 18, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 84.396 sec -
in org.apache.hive.hcatalog.pig.TestHCatLoaderComplexSchema
Running org.apache.hive.hcatalog.pig.TestHCatLoaderStorer
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.12 sec - in
org.apache.hive.hcatalog.pig.TestHCatLoaderStorer
Running org.apache.hive.hcatalog.pig.TestHCatStorer
Tests run: 162, Failures: 0, Errors: 0, Skipped: 28, Time elapsed: 482.723 sec
- in org.apache.hive.hcatalog.pig.TestHCatStorer
Running org.apache.hive.hcatalog.pig.TestHCatStorerMulti
Tests run: 18, Failures: 0, Errors: 0, Skipped: 6, Time elapsed: 28.469 sec -
in org.apache.hive.hcatalog.pig.TestHCatStorerMulti
Running org.apache.hive.hcatalog.pig.TestHCatStorerWrapper
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.437 sec - in
org.apache.hive.hcatalog.pig.TestHCatStorerWrapper
Running org.apache.hive.hcatalog.pig.TestPigHCatUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.439 sec - in
org.apache.hive.hcatalog.pig.TestPigHCatUtil
Running org.apache.hive.hcatalog.pig.TestUtil
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.052 sec - in
org.apache.hive.hcatalog.pig.TestUtil
Results :
Tests run: 266, Failures: 0, Errors: 0, Skipped: 43
{noformat}
The above is after removing Avro from test exclusions for HCatLoader and
HCatStorer. Note that HCatStorer still has some tests disabled on Avro for the
following reasons:
* testDateCharTypes : incorrect precision returned (returned "5.20" instead
of "5.2" )
* testWriteDecimalXY : incorrect precision returned (returned "1.20" instead
of "1.2" )
* testWriteSmallInt : Avro does not have a small, and uses an int instead,
and thus returned a value of 32768 instead of a null from an overflow
* testWriteTinyInt : Avro does not have a "tiny", and uses an int instead,
and thus returned a value of 300 isntead of a null from an overflow
* testWriteTimestamp : Avro does not support the timestamp datatype
Since the above are avro feature support issues, and not avro-hcatalog
integration issues, I kept these tests disabled for now, and we can re-enable
them later as and when we make those work.
> Support Avro through HCatalog
> -----------------------------
>
> Key: HIVE-8687
> URL: https://issues.apache.org/jira/browse/HIVE-8687
> Project: Hive
> Issue Type: Bug
> Components: HCatalog, Serializers/Deserializers
> Affects Versions: 0.14.0
> Environment: discovered in Pig, but it looks like the root cause
> impacts all non-Hive users
> Reporter: Sushanth Sowmyan
> Assignee: Sushanth Sowmyan
> Priority: Critical
> Fix For: 0.14.0
>
> Attachments: HIVE-8687.branch-0.14.patch, HIVE-8687.patch
>
>
> Attempting to write to a HCatalog defined table backed by the AvroSerde fails
> with the following stacktrace:
> {code}
> java.lang.ClassCastException: org.apache.hadoop.io.NullWritable cannot be
> cast to org.apache.hadoop.io.LongWritable
> at
> org.apache.hadoop.hive.ql.io.avro.AvroContainerOutputFormat$1.write(AvroContainerOutputFormat.java:84)
> at
> org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:253)
> at
> org.apache.hcatalog.mapreduce.FileRecordWriterContainer.write(FileRecordWriterContainer.java:53)
> at
> org.apache.hcatalog.pig.HCatBaseStorer.putNext(HCatBaseStorer.java:242)
> at org.apache.hcatalog.pig.HCatStorer.putNext(HCatStorer.java:52)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:139)
> at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat$PigRecordWriter.write(PigOutputFormat.java:98)
> at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.write(MapTask.java:559)
> at
> org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:85)
> {code}
> The proximal cause of this failure is that the AvroContainerOutputFormat's
> signature mandates a LongWritable key and HCat's FileRecordWriterContainer
> forces a NullWritable. I'm not sure of a general fix, other than redefining
> HiveOutputFormat to mandate a WritableComparable.
> It looks like accepting WritableComparable is what's done in the other Hive
> OutputFormats, and there's no reason AvroContainerOutputFormat couldn't also
> be changed, since it's ignoring the key. That way fixing things so
> FileRecordWriterContainer can always use NullWritable could get spun into a
> different issue?
> The underlying cause for failure to write to AvroSerde tables is that
> AvroContainerOutputFormat doesn't meaningfully implement getRecordWriter, so
> fixing the above will just push the failure into the placeholder RecordWriter.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
