KnightChess opened a new issue, #5568:
URL: https://github.com/apache/hudi/issues/5568
### diff:
**readSchame**:
```shell
# minValue
{"name":"minValue","type":["null",{"type":"string","avro.java.string":"String"}],"doc":"Minimum
value in the range. Based on user data table schema, we can convert this to
appropriate type"}
# maxValue
...
```
**writerSchame**:
```shell
# minValue
{"name":"minValue","type":["null",{"type":"record","name":"BooleanWrapper","doc":"A
record wrapping boolean type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"boolean"}]},{"type":"record","name":"IntWrapper","doc":"A
record wrapping int type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"int"}]},{"type":"record","name":"LongWrapper","doc":"A
record wrapping long type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"long"}]},{"type":"record","name":"FloatWrapper","doc":"A
record wrapping float type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"float"}]},{"type":"record","name":"DoubleWrapper","doc":"A
record wrapping double type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"double"}]},{"type":"record","name":"BytesWrapper","doc":"A
record wrapping bytes type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","
type":"bytes"}]},{"type":"record","name":"StringWrapper","doc":"A record
wrapping string type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":{"type":"string","avro.java.string":"String"}}]},{"type":"record","name":"DateWrapper","doc":"A
record wrapping Date logical type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"int"}]},{"type":"record","name":"DecimalWrapper","doc":"A
record wrapping Decimal logical type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":{"type":"bytes","logicalType":"decimal","precision":30,"scale":15}}]},{"type":"record","name":"TimeMicrosWrapper","doc":"A
record wrapping Time-micros logical type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":{"type":"long","logicalType":"time-micros"}}]},{"type":"record","name":"TimestampMicrosWrapper","doc":"A
record wrapping Timestamp-micros logical type to be able to be used it w/in
Avro's Union","fields":[
{"name":"value","type":"long"}]}],"doc":"Minimum value in the range. Based on
user data table schema, we can convert this to appropriate type","default":null}
# maxValue
...
```
### all:
```shell
Caused by: org.apache.hudi.exception.HoodieUpsertException: Failed upsert
schema compatibility check.
at
org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:743)
at
org.apache.hudi.client.SparkRDDWriteClient.upsertPreppedRecords(SparkRDDWriteClient.java:169)
at
org.apache.hudi.metadata.SparkHoodieBackedTableMetadataWriter.commit(SparkHoodieBackedTableMetadataWriter.java:166)
at
org.apache.hudi.metadata.HoodieBackedTableMetadataWriter.update(HoodieBackedTableMetadataWriter.java:924)
at
org.apache.hudi.table.action.BaseActionExecutor.lambda$writeTableMetadata$2(BaseActionExecutor.java:77)
at org.apache.hudi.common.util.Option.ifPresent(Option.java:97)
at
org.apache.hudi.table.action.BaseActionExecutor.writeTableMetadata(BaseActionExecutor.java:77)
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.finishRollback(BaseRollbackActionExecutor.java:255)
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.runRollback(BaseRollbackActionExecutor.java:124)
at
org.apache.hudi.table.action.rollback.BaseRollbackActionExecutor.execute(BaseRollbackActionExecutor.java:145)
at
org.apache.hudi.table.HoodieSparkCopyOnWriteTable.rollback(HoodieSparkCopyOnWriteTable.java:281)
at
org.apache.hudi.client.BaseHoodieWriteClient.rollback(BaseHoodieWriteClient.java:766)
... 101 more
Caused by: org.apache.hudi.exception.HoodieException: Failed schema
compatibility check for writerSchema :
{"type":"record","name":"HoodieMetadataRecord","namespace":"org.apache.hudi.avro.model","doc":"A
record saved within the Metadata
Table","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"key","type":{"type":"string","avro.java.string":"String"}},{"name":"type","type":"int","doc":"Type
of the metadata
record"},{"name":"filesystemMetadata","type":["null",{"type":"map","values":{"type":"record","name":"HoodieMetadataFileInfo","fields":[{"name":"size","type":"long","doc":"Size
of the file"},{"name":"isDeleted","type":"boolean","doc":"True if this file
has been deleted"}]},"avro.java.string":"String"}],"doc":"Contains info
rmation about partitions and files within the
dataset"},{"name":"BloomFilterMetadata","type":["null",{"type":"record","name":"HoodieMetadataBloomFilter","doc":"Data
file bloom filter
details","fields":[{"name":"type","type":{"type":"string","avro.java.string":"String"},"doc":"Bloom
filter type
code"},{"name":"timestamp","type":{"type":"string","avro.java.string":"String"},"doc":"Instant
timestamp when this metadata was
created/updated"},{"name":"bloomFilter","type":"bytes","doc":"Bloom filter
binary byte array"},{"name":"isDeleted","type":"boolean","doc":"Bloom filter
entry valid/deleted flag"}]}],"doc":"Metadata Index of bloom filters for all
data files in the user
table","default":null},{"name":"ColumnStatsMetadata","type":["null",{"type":"record","name":"HoodieMetadataColumnStats","doc":"Data
file column
statistics","fields":[{"name":"fileName","type":["null",{"type":"string","avro.java.string":"String"}],"doc":"File
name for which this column statistics applies","default":null},
{"name":"columnName","type":["null",{"type":"string","avro.java.string":"String"}],"doc":"Column
name for which this column statistics
applies","default":null},{"name":"minValue","type":["null",{"type":"record","name":"BooleanWrapper","doc":"A
record wrapping boolean type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"boolean"}]},{"type":"record","name":"IntWrapper","doc":"A
record wrapping int type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"int"}]},{"type":"record","name":"LongWrapper","doc":"A
record wrapping long type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"long"}]},{"type":"record","name":"FloatWrapper","doc":"A
record wrapping float type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"float"}]},{"type":"record","name":"DoubleWrapper","doc":"A
record wrapping double type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":
"double"}]},{"type":"record","name":"BytesWrapper","doc":"A record wrapping
bytes type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"bytes"}]},{"type":"record","name":"StringWrapper","doc":"A
record wrapping string type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":{"type":"string","avro.java.string":"String"}}]},{"type":"record","name":"DateWrapper","doc":"A
record wrapping Date logical type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"int"}]},{"type":"record","name":"DecimalWrapper","doc":"A
record wrapping Decimal logical type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":{"type":"bytes","logicalType":"decimal","precision":30,"scale":15}}]},{"type":"record","name":"TimeMicrosWrapper","doc":"A
record wrapping Time-micros logical type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":{"type":"long","logicalType":"time-micros"}}]},
{"type":"record","name":"TimestampMicrosWrapper","doc":"A record wrapping
Timestamp-micros logical type to be able to be used it w/in Avro's
Union","fields":[{"name":"value","type":"long"}]}],"doc":"Minimum value in the
range. Based on user data table schema, we can convert this to appropriate
type","default":null},{"name":"maxValue","type":["null","BooleanWrapper","IntWrapper","LongWrapper","FloatWrapper","DoubleWrapper","BytesWrapper","StringWrapper","DateWrapper","DecimalWrapper","TimeMicrosWrapper","TimestampMicrosWrapper"],"doc":"Maximum
value in the range. Based on user data table schema, we can convert it to
appropriate
type","default":null},{"name":"valueCount","type":["null","long"],"doc":"Total
count of
values","default":null},{"name":"nullCount","type":["null","long"],"doc":"Total
count of null
values","default":null},{"name":"totalSize","type":["null","long"],"doc":"Total
storage size on
disk","default":null},{"name":"totalUncompressedSize","type":["null","long"],"doc":"
Total uncompressed storage size on
disk","default":null},{"name":"isDeleted","type":"boolean","doc":"Column range
entry valid/deleted flag"}]}],"doc":"Metadata Index of column statistics for
all data files in the user table","default":null}]}, table schema
:{"type":"record","name":"HoodieMetadataRecord","namespace":"org.apache.hudi.avro.model","doc":"A
record saved within the Metadata
Table","fields":[{"name":"_hoodie_commit_time","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_commit_seqno","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_record_key","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_partition_path","type":["null","string"],"doc":"","default":null},{"name":"_hoodie_file_name","type":["null","string"],"doc":"","default":null},{"name":"key","type":{"type":"string","avro.java.string":"String"}},{"name":"type","type":"int","doc":"Type
of the metadata
record"},{"name":"filesystemMetadata","type":["null",{"type":"map",
"values":{"type":"record","name":"HoodieMetadataFileInfo","fields":[{"name":"size","type":"long","doc":"Size
of the file"},{"name":"isDeleted","type":"boolean","doc":"True if this file
has been deleted"}]},"avro.java.string":"String"}],"doc":"Contains information
about partitions and files within the
dataset"},{"name":"BloomFilterMetadata","type":["null",{"type":"record","name":"HoodieMetadataBloomFilter","doc":"Data
file bloom filter
details","fields":[{"name":"type","type":{"type":"string","avro.java.string":"String"},"doc":"Bloom
filter type
code"},{"name":"timestamp","type":{"type":"string","avro.java.string":"String"},"doc":"Instant
timestamp when this metadata was
created/updated"},{"name":"bloomFilter","type":"bytes","doc":"Bloom filter
binary byte array"},{"name":"isDeleted","type":"boolean","doc":"Bloom filter
entry valid/deleted flag"}]}],"doc":"Metadata Index of bloom filters for all
data files in the user
table","default":null},{"name":"ColumnStatsMetadata","type":["null
",{"type":"record","name":"HoodieMetadataColumnStats","doc":"Data file column
statistics","fields":[{"name":"fileName","type":["null",{"type":"string","avro.java.string":"String"}],"doc":"File
name for which this column statistics
applies"},{"name":"minValue","type":["null",{"type":"string","avro.java.string":"String"}],"doc":"Minimum
value in the range. Based on user data table schema, we can convert this to
appropriate
type"},{"name":"maxValue","type":["null",{"type":"string","avro.java.string":"String"}],"doc":"Maximum
value in the range. Based on user data table schema, we can convert it to
appropriate type"},{"name":"valueCount","type":["null","long"],"doc":"Total
count of values"},{"name":"nullCount","type":["null","long"],"doc":"Total count
of null values"},{"name":"totalSize","type":["null","long"],"doc":"Total
storage size on
disk"},{"name":"totalUncompressedSize","type":["null","long"],"doc":"Total
uncompressed storage size on disk"},{"name":"isDeleted","type":"boolean","d
oc":"Column range entry valid/deleted flag"}]}],"doc":"Metadata Index of
column statistics for all data files in the user table","default":null}]}, base
path
:hdfs://DClusterNmg2/user/prod_datalake_test/datalake_test/hive/datalake_test/auth_test_p_dt_cow/.hoodie/metadata
at
org.apache.hudi.table.HoodieTable.validateSchema(HoodieTable.java:735)
at
org.apache.hudi.table.HoodieTable.validateUpsertSchema(HoodieTable.java:741)
... 112 more
```
should I add the **string** type to backward compatible the change in
min/maxValue type? or delete metadata folder to create new one? which is
better?
add **string** to compaty may look strange with the others like:
BooleanWrapper、IntWrapper、StringWrapper、etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
