Owen,
Thanks that is great.
>From that error do you have a trick I can do to tear apart this column which
>is a array of structs to see what I am missing for a null ?
java.lang.NullPointerException at java.lang.String.checkBounds(String.java:384)
at java.lang.String.<init>(String.java:545) at
org.apache.hadoop.hive.ql.exec.vector.BytesColumnVector.stringifyValue(BytesColumnVector.java:445)
at
org.apache.hadoop.hive.ql.exec.vector.StructColumnVector.stringifyValue(StructColumnVector.java:85)
at
org.apache.hadoop.hive.ql.exec.vector.ListColumnVector.stringifyValue(ListColumnVector.java:91)
at com.sovrn.scratch.Scratch.main(Scratch.java:152)
What a pain to convert avro to orc .....
From: Owen O'Malley <[email protected]>
To: [email protected]; Telco Phone <[email protected]>
Sent: Friday, October 13, 2017 3:57 PM
Subject: Re: Error with batch write
So, the error is a null pointer in a BytesColumnVector.vector[r] down in one of
the nested fields.
The relevant path through the schema looks like: struct -> list -> struct ->
struct -> string.
You could do something like:
VectorizedRowBatch batch = ...for(int r = 0; r < batch.size; ++r) {
System.out.println("Starting row " + r); for(int c=0; c < batch.cols.length;
++c) { System.out.println("Column " + c); // You'll get an exception on
the broken field StringBuilder builder = new StringBuilder();
batch.cols[c].stringifyValue(builder, r); }}
.. Owen
On Fri, Oct 13, 2017 at 2:02 PM, Telco Phone <[email protected]> wrote:
What is the best way to troubleshoot this kind of error?
java.lang.NullPointerException at java.lang.System.arraycopy( Native Method) at
org.apache.hadoop.io.Text.set( Text.java:225) at org.apache.orc.impl.
StringRedBlackTree.add( StringRedBlackTree.java:62) at org.apache.orc.impl.
WriterImpl$StringTreeWriter. writeBatch(WriterImpl.java: 1459) at
org.apache.orc.impl. WriterImpl$StructTreeWriter. writeFields(WriterImpl.java:
2106) at org.apache.orc.impl. WriterImpl$StructTreeWriter.
writeBatch(WriterImpl.java: 2120) at org.apache.orc.impl.
WriterImpl$ListTreeWriter. writeBatch(WriterImpl.java: 2221) at
org.apache.orc.impl. WriterImpl$StructTreeWriter. writeRootBatch(WriterImpl.
java:2098) at org.apache.orc.impl. WriterImpl.addRowBatch(
WriterImpl.java:2811) at com.sovrn.scratch.Scratch. main(Scratch.java:154)
Line 154 is writer.addRowBatch(batch);
I am not looking to have anyone fixe my current is but more on how do I setup
the code base to quickly trouble shoot these kind of issue ?
Is it my array code ? String ?
The logic loop that I have the code in seems to work well but from time to time
I will get the above.
Thoughts ?
