zhangxinjian123 edited a comment on issue #829: Structured Streaming read kafka URL: https://github.com/apache/incubator-hudi/issues/829#issuecomment-521086833 first json: {"database":"zhang","table":"hao","type":"insert","ts":1555984612,"xid":90202,"commit":true,"data":{"id":7,"Name":"zhang"}} I get 7 and Zhang as values by parsing JSON to get ID and name as schemas, respectively. This is writable second json: {"database":"zhang","table":"haow","type":"insert","ts":1555984612,"xid":90202,"commit":true,"data":{"id":6,"Name":"zhang","Age":13,"code":19}} 57947 [pool-17-thread-2] ERROR com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor - error consuming records java.lang.ArrayIndexOutOfBoundsException: 10 at org.apache.avro.generic.GenericData$Record.get(GenericData.java:212) at org.apache.avro.generic.GenericData.getField(GenericData.java:697) at org.apache.parquet.avro.AvroWriteSupport.writeRecordFields(AvroWriteSupport.java:188) at org.apache.parquet.avro.AvroWriteSupport.write(AvroWriteSupport.java:165) at org.apache.parquet.hadoop.InternalParquetRecordWriter.write(InternalParquetRecordWriter.java:128) at org.apache.parquet.hadoop.ParquetWriter.write(ParquetWriter.java:299) at com.uber.hoodie.io.storage.HoodieParquetWriter.writeAvro(HoodieParquetWriter.java:103) at com.uber.hoodie.io.HoodieMergeHandle.write(HoodieMergeHandle.java:292) at com.uber.hoodie.table.HoodieCopyOnWriteTable$UpdateHandler.consumeOneRecord(HoodieCopyOnWriteTable.java:452) at com.uber.hoodie.table.HoodieCopyOnWriteTable$UpdateHandler.consumeOneRecord(HoodieCopyOnWriteTable.java:442) at com.uber.hoodie.common.util.queue.BoundedInMemoryQueueConsumer.consume(BoundedInMemoryQueueConsumer.java:38) at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$2(BoundedInMemoryExecutor.java:126) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:748) 57979 [pool-17-thread-1] ERROR com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor - error producing records com.uber.hoodie.exception.HoodieException: operation has failed at com.uber.hoodie.common.util.queue.BoundedInMemoryQueue.throwExceptionIfFailed(BoundedInMemoryQueue.java:234) at com.uber.hoodie.common.util.queue.BoundedInMemoryQueue.insertRecord(BoundedInMemoryQueue.java:171) at com.uber.hoodie.common.util.queue.IteratorBasedQueueProducer.produce(IteratorBasedQueueProducer.java:45) at com.uber.hoodie.common.util.queue.BoundedInMemoryExecutor.lambda$null$0(BoundedInMemoryExecutor.java:96) at java.util.concurrent.FutureTask.run(FutureTask.java:266) At first I wrote correctly, but this error occurs when there are more new fields than the first one.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
