jayzhan211 commented on issue #12622: URL: https://github.com/apache/datafusion/issues/12622#issuecomment-2599579247
> Unfortunately, using Scalar in ColumnarValue and Expr breaks all patterns that try to match the scalar for these types. While we can fix this in the DataFusion code base (we need a nested match or smth similar in many cases), all downstream dependencies must rewrite their pattern-matching code, which can be (I assume) a huge undertaking for many projects. Here is an [example](https://github.com/apache/datafusion/commit/454db7e63c6d6af1c8975f29ce9b7ee9c37ea287#diff-d4a05272894f059368463815799d47bc9c33fd3f861ffcea827839f71938846fR111) from the current version of logical-types. I think downstream project are not all forced to switch to `Scalar` they can keep their `ScalarValue` matching code, except for Utf8View and LargeUtf8 matching cases `Scalar` is introduced to separate the concept of arrow's DataType with the value itself. The following is equivalent ```rust ScalarValue::Utf8View Scalar { ScalarValue::Utf8(String), DataType::Utf8View } ``` ```rust ScalarValue::LargeUtf8 Scalar { ScalarValue::Utf8(String), DataType::LargeUtf8 } ``` > Furthermore, if we ever conclude that we do not need the physical type (e.g., because we can derive the physical type from the schema), these breaking changes could have been avoided If we can find such approach it would be great. `Scalar` is just a practical approach that w e can iterate on it with. There might be a better solution toward the goal, but I have no such idea in my mind. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
