If you update the partition(20200205) after adding fields. it will show the
add fields by using ` val hudiDF2 =
spark.read.format("org.apache.hudi").load("/tmp/hudi/drivers/*");
hudiDF2.show `, which needn't mergeSchema from all files.Igor Basko <[email protected]> 于2020年2月6日周四 下午8:35写道： > Thanks a lot for the answer. > I was sure Hudi would store the latest schema, instead of merging it from > all the files. > > On Thu, 6 Feb 2020 at 01:10, leesf <[email protected]> wrote: > > > Hi Igor, > > > > It is because the Spark ParquetFileFormat infer schema from the parquet > > file under 20200205 dir, and the file do not contains the added > > column(direction), you would just try `val hudiDF2 = > > spark.read.format("org.apache.hudi").option("mergeSchema", > > "true").load("/tmp/hudi/drivers/*")` to get schema merged from 20200205 > and > > 20200206, and it shows the added column, I do not know whether it is a > > common soulution but it solves the problem. > > > > Best, > > Leesf > > ` > > > > Igor Basko <[email protected]> 于2020年2月5日周三 下午3:33写道： > > > > > Hi All, > > > I've tried to write data with some schema changes using the Datasource > > > Writer. > > > The procedure was: > > > First I wrote an event with a specific schema. > > > After that I wrote a different event with the same schema but with one > > more > > > added field. > > > > > > When I read from the Hudi table, I get both the events, with the > original > > > schema. > > > I was expecting to get both events with the newer schema with some > > default > > > value in the new > > > field for the first event. > > > > > > I've created a gist that describes my experience: > > > https://gist.github.com/igorbasko01/4a1d0cf7c06a5b216382260efaa1f333 > > > > > > Would like to know, if schema evolution is supported using the > Datasource > > > Writer. > > > Or maybe I'm doing something wrong. > > > > > > Thanks a lot. > > > > > >
