ROOBALJINDAL commented on issue #6348:
URL: https://github.com/apache/hudi/issues/6348#issuecomment-1210402767
I further checked and downloaded aws logs of the node and checked the exact
exception. Do you have any idea about what's going wrong? @nsivabalan
@pratyakshsharma
**Stack trace**
`22/08/10 07:31:24 ERROR HoodieWriteHandle: Error writing record
HoodieRecord{key=HoodieKey { recordKey=1201
partitionPath=receiptdt=2022/06/22}, currentLocation='null', newLocation='null'}
java.lang.UnsupportedOperationException: Cannot read strings longer than
2147483639 bytes
at org.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:305)
at
org.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:208)
at
org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:469)
at
org.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:459)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:191)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:187)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.readField(GenericDatumReader.java:259)
at
org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:247)
at
org.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:179)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:160)
at
org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at
org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro(HoodieAvroUtils.java:156)
at
org.apache.hudi.avro.HoodieAvroUtils.bytesToAvro(HoodieAvroUtils.java:146)
at
org.apache.hudi.common.model.OverwriteWithLatestAvroPayload.getInsertValue(OverwriteWithLatestAvroPayload.java:75)
at
org.apache.hudi.common.model.debezium.AbstractDebeziumAvroPayload.getInsertRecord(AbstractDebeziumAvroPayload.java:87)
at
org.apache.hudi.common.model.debezium.AbstractDebeziumAvroPayload.getInsertValue(AbstractDebeziumAvroPayload.java:58)
at
org.apache.hudi.common.model.HoodieRecordPayload.getInsertValue(HoodieRecordPayload.java:105)
at
org.apache.hudi.execution.HoodieLazyInsertIterable$HoodieInsertValueGenResult.<init>(HoodieLazyInsertIterable.java:90)
at
org.apache.hudi.execution.HoodieLazyInsertIterable.lambda$getTransformFunction$0(HoodieLazyInsertIterable.java:103)
at
org.apache.hudi.common.util.queue.BoundedInMemoryQueue.insertRecord(BoundedInMemoryQueue.java:190)
at
org.apache.hudi.common.util.queue.IteratorBasedQueueProducer.produce(IteratorBasedQueueProducer.java:46)
at
org.apache.hudi.common.util.queue.BoundedInMemoryExecutor.lambda$null$0(BoundedInMemoryExecutor.java:105)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
22/08/10 07:31:25 INFO MultipartUploadOutputStream: close closed:false
s3://hudi-multistreamer-roobal/hudi/rrmencounter/receiptdt=2022/06/22/c839dd87-e7e1-4878-bb38-999d331bed33-0_0-29-2008_20220810073102064.parquet
22/08/10 07:31:25 INFO MemoryStore: Block rdd_64_0 stored as values in
memory (estimated size 2.4 KiB, free 1843.7 MiB)
22/08/10 07:31:25 INFO Executor: Finished task 0.0 in stage 29.0 (TID 2008).
1593 bytes result sent to driver
22/08/10 07:31:25 INFO YarnCoarseGrainedExecutorBackend: Got assigned task
2009
22/08/10 07:31:25 INFO Executor: Running task 0.0 in stage 35.0 (TID 2009)
22/08/10 07:31:25 INFO TorrentBroadcast: Started reading broadcast variable
18 with 1 pieces (estimated total size 4.0 MiB)
22/08/10 07:31:25 INFO MemoryStore: Block broadcast_18_piece0 stored as
bytes in memory (estimated size 329.2 KiB, free 1843.4 MiB)
22/08/10 07:31:25 INFO TorrentBroadcast: Reading broadcast variable 18 took
7 ms
22/08/10 07:31:25 INFO MemoryStore: Block broadcast_18 stored as values in
memory (estimated size 1357.0 KiB, free 1842.1 MiB)
22/08/10 07:31:25 INFO BlockManager: Found block rdd_64_0 locally`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
