danny0405 commented on a change in pull request #3257:
URL: https://github.com/apache/hudi/pull/3257#discussion_r672074150
##########
File path: docs/_docs/2_2_writing_data.md
##########
@@ -424,3 +424,192 @@ Here are some ways to efficiently manage the storage of
your Hudi tables.
- Intelligently tuning the [bulk insert
parallelism](/docs/configurations.html#withBulkInsertParallelism), can again in
nicely sized initial file groups. It is in fact critical to get this right,
since the file groups
once created cannot be deleted, but simply expanded as explained before.
- For workloads with heavy updates, the [merge-on-read
table](/docs/concepts.html#merge-on-read-table) provides a nice mechanism for
ingesting quickly into smaller files and then later merging them into larger
base files via compaction.
+
+
+## Schema Evolution
+
+Schema evolution is a very important aspect of data management.
+Hudi supports common schema evolution scenarios, such as adding a nullable
field or promoting a datatype of a field, out-of-the-box.
+Furthermore, the evolved schema is queryable across engines, such as Presto,
Hive and Spark SQL.
+The following table presents a summary of the types of schema changes
compatible with different Hudi table types.
+
+| Schema Change | COW | MOR | Remarks |
+| ----------- | ------- | ------- | ------- |
+| Add a new nullable column at root level at the end | Yes | Yes | `Yes` means
that a write with evolved schema succeeds and a read following the write
succeeds to read entire dataset. |
+| Add a new nullable column to inner struct (at the end) | Yes | Yes |
+| Add a new complex type field with default (map and array) | Yes | Yes | |
+| Add a new nullable column and change the ordering of fields | Yes | Yes | |
+| Add a custom nullable Hudi meta column, e.g. `_hoodie_meta_col` | Yes | Yes
| |
+| Promote datatype from `int` to `long` for a field at root level | Yes | Yes
| For other types, Hudi supports promotion as specified in [Avro schema
resolution](http://avro.apache.org/docs/current/spec.html#Schema+Resolution). |
+| Promote datatype from `int` to `long` for a nested field | Yes | Yes |
+| Promote datatype from `int` to `long` for a complex type (value of map or
array) | Yes | Yes | |
+| Add a new non-nullable column at root level at the end | No | No | In case
of MOR table with Spark data source, write succeeds but read fails. |
Review comment:
Hmm, i plan to add a new metadata column in HUDI-1771 named
`_hoodie_operation` to record the change flag. Very good news to see that this
expects to be compatible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
