andygrove commented on code in PR #654:
URL: https://github.com/apache/datafusion-comet/pull/654#discussion_r1676368801
##########
native/utils/src/lib.rs:
##########
@@ -34,3 +48,151 @@ pub fn down_cast_any_ref(any: &dyn Any) -> &dyn Any {
any
}
}
+
+/// Preprocesses input arrays to add timezone information from Spark to Arrow
array datatype or
+/// to apply timezone offset.
+//
+// We consider the following cases:
+//
+// | --------------------- | ------------ | ----------------- |
-------------------------------- |
+// | Conversion | Input array | Timezone | Output array
|
+// | --------------------- | ------------ | ----------------- |
-------------------------------- |
+// | Timestamp -> | Array in UTC | Timezone of input | A timestamp
with the timezone |
+// | Utf8 or Date32 | | | offset
applied and timezone |
+// | | | | removed
|
+// | --------------------- | ------------ | ----------------- |
-------------------------------- |
+// | Timestamp -> | Array in UTC | Timezone of input | Same as input
array |
+// | Timestamp w/Timezone| | |
|
+// | --------------------- | ------------ | ----------------- |
-------------------------------- |
+// | Timestamp_ntz -> | Array in | Timezone of input | Same as input
array |
+// | Utf8 or Date32 | timezone | |
|
+// | | session local| |
|
+// | | timezone | |
|
+// | --------------------- | ------------ | ----------------- |
-------------------------------- |
+// | Timestamp_ntz -> | Array in | Timezone of input | Array in UTC
and timezone |
+// | Timestamp w/Timezone | session local| | specified in
input |
+// | | timezone | |
|
+// | --------------------- | ------------ | ----------------- |
-------------------------------- |
+// | Timestamp(_ntz) -> |
|
+// | Any other type | Not Supported
|
+// | --------------------- | ------------ | ----------------- |
-------------------------------- |
+//
+pub fn array_with_timezone(
Review Comment:
I suppose some of these functions could just live in the spark-expr crate. I
will look at that as a follow on issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
