[GitHub] [hudi] lufzhangzitao commented on issue #7283: [SUPPORT] Schema evolution fails due to NPE with `hoodie.schema.on.read.enable` turned on

Mon, 28 Nov 2022 23:48:10 -0800 


lufzhangzitao commented on issue #7283:
URL: https://github.com/apache/hudi/issues/7283#issuecomment-1330218337

   @xiarixiaoyao 
    thx I tried hudi 0.12.1 schema evolution can work with 
hoodie.datasource.write.reconcile.schema ，
   But it seems the hudi document example cannot  work , I try the example  add 
a new string field and change the datatype of a field from int to long in hudi 
0.12.1, will throw  exception :
   org.apache.avro.AvroRuntimeException: cannot support rewrite value for 
schema type: "int" since the old schema type is: "long"
        at 
org.apache.hudi.avro.HoodieAvroUtils.rewritePrimaryTypeWithDiffSchemaType(HoodieAvroUtils.java:955)
        at 
org.apache.hudi.avro.HoodieAvroUtils.rewritePrimaryType(HoodieAvroUtils.java:869)
        at 
org.apache.hudi.avro.HoodieAvroUtils.rewriteRecordWithNewSchema(HoodieAvroUtils.java:820)
        at 
org.apache.hudi.avro.HoodieAvroUtils.rewriteRecordWithNewSchema(HoodieAvroUtils.java:818)
        at 
org.apache.hudi.avro.HoodieAvroUtils.rewriteRecordWithNewSchema(HoodieAvroUtils.java:772)
        at 
org.apache.hudi.avro.HoodieAvroUtils.rewriteRecordWithNewSchema(HoodieAvroUtils.java:734)
        at 
org.apache.hudi.avro.HoodieAvroUtils.rewriteRecordDeep(HoodieAvroUtils.java:1034)
        at 
org.apache.hudi.HoodieSparkUtils$.$anonfun$createRdd$4(HoodieSparkUtils.scala:109)
        at 
org.apache.hudi.HoodieSparkUtils$.$anonfun$createRdd$5(HoodieSparkUtils.scala:117)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at scala.collection.Iterator$$anon$10.next(Iterator.scala:461)
        at 
org.apache.spark.util.collection.ExternalSorter.insertAll(ExternalSorter.scala:199)
        at 
org.apache.spark.shuffle.sort.SortShuffleWriter.write(SortShuffleWriter.scala:63)
        at 
org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
        at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
        at 
org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
        at org.apache.spark.scheduler.Task.run(Task.scala:131)
        at 
org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:506)
        at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1491)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:509)
        at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
   
   
   
![企业微信截图_1669707358199](https://user-images.githubusercontent.com/77084940/204467890-ef9d7c91-2c18-461f-93e6-813d7aab3052.png)
   
   
![企业微信截图_16697074283529](https://user-images.githubusercontent.com/77084940/204467975-7d97b32a-1494-40c2-8b1f-5d0ef49d4a05.png)
   
   
![企业微信截图_16697074634013](https://user-images.githubusercontent.com/77084940/204468003-2f9260b8-02ad-41b9-8f80-fd5b8f160042.png)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to