Rich-T-kid commented on code in PR #24227:
URL: https://github.com/apache/datafusion/pull/24227#discussion_r3962339698
##########
datafusion/datasource-parquet/src/schema_coercion.rs:
##########
@@ -135,6 +157,169 @@ pub fn apply_file_schema_type_coercions(
))
}
+// Find the value type that can represent both sides without narrowing offsets
+// or crossing string/binary families.
+fn common_dictionary_value_type(
+ field_type: &DataType,
+ dictionary_value_type: &DataType,
+) -> Option<DataType> {
+ let field_type = match field_type {
+ DataType::Dictionary(_, field_value_type) => field_value_type.as_ref(),
+ _ => field_type,
+ };
+
+ match (field_type, dictionary_value_type) {
+ (DataType::Utf8, DataType::Utf8) => Some(DataType::Utf8),
+ (DataType::Utf8 | DataType::LargeUtf8, DataType::Utf8 |
DataType::LargeUtf8) => {
+ Some(DataType::LargeUtf8)
+ }
+ (DataType::Binary, DataType::Binary) => Some(DataType::Binary),
+ (
+ DataType::Binary | DataType::LargeBinary,
+ DataType::Binary | DataType::LargeBinary,
+ ) => Some(DataType::LargeBinary),
+ _ => None,
+ }
+}
+
+// Same family (both signed or both unsigned): return the wider member.
Review Comment:
> Could this select UInt64 for that case, or reject the normalization
instead of silently narrowing it?
with
https://github.com/apache/datafusion/pull/24227/changes/4d607b74af918f514fb4f521b9b307a3458a5546
the type now widens instead of narrowing silently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
