Re: [PR] feat: Support Spark expression: make_dt_interval [datafusion-comet]

2026-08-30 Thread via GitHub 


github-actions[bot] commented on PR #4338:
URL: 
https://github.com/apache/datafusion-comet/pull/4338#issuecomment-5472698134

   Thank you for your contribution. Unfortunately, this pull request is stale 
because it has been open 60 days with no activity. Please remove the stale 
label or comment or this will be closed in 7 days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] feat: Support Spark expression: make_dt_interval [datafusion-comet]

2026-05-20 Thread via GitHub 


YutaLin commented on code in PR #4338:
URL: https://github.com/apache/datafusion-comet/pull/4338#discussion_r3276720689


##
spark/src/main/scala/org/apache/comet/serde/datetime.scala:
##
@@ -413,6 +413,8 @@ object CometNextDay extends 
CometScalarFunction[NextDay]("next_day")
 
 object CometMakeDate extends CometScalarFunction[MakeDate]("make_date")
 
+object CometMakeDTInterval extends 
CometScalarFunction[MakeDTInterval]("make_dt_interval")

Review Comment:
   Hi @andygrove 
   Thanks for the review! The statement is true, i've marked it incompatible 
and filed an [issue](https://github.com/apache/datafusion/issues/22401).



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] feat: Support Spark expression: make_dt_interval [datafusion-comet]

2026-05-20 Thread via GitHub 


andygrove commented on code in PR #4338:
URL: https://github.com/apache/datafusion-comet/pull/4338#discussion_r3274882204


##
spark/src/main/scala/org/apache/comet/serde/datetime.scala:
##
@@ -413,6 +413,8 @@ object CometNextDay extends 
CometScalarFunction[NextDay]("next_day")
 
 object CometMakeDate extends CometScalarFunction[MakeDate]("make_date")
 
+object CometMakeDTInterval extends 
CometScalarFunction[MakeDTInterval]("make_dt_interval")

Review Comment:
   Could you implement `getSupportLevel` and mark this as incompatible. Also, 
we need `getIncompatReasons` so we generate documentation. 
   
   My AI review helper tells me that Spark's 
`IntervalUtils.makeDayTimeInterval` uses `Math.addExact` / `Math.multiplyExact` 
and always throws on overflow:
   
   ```scala
 catch {
   case _: ArithmeticException =>
 throw 
QueryExecutionErrors.withoutSuggestionIntervalArithmeticOverflowError(context)
 }
   ```
   The upstream `SparkMakeDtInterval` kernel uses `checked_mul` / `checked_add` 
and silently returns `NULL` on overflow. Inputs that throw 
`INTERVAL_ARITHMETIC_OVERFLOW` in Spark will
 produce a `NULL` row in Comet.
   
   Could you verify whether this is true or not? If it is then we can just mark 
it incompatible for now and file an upstream issue against the datafusion repo.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Re: [PR] feat: Support Spark expression: make_dt_interval [datafusion-comet]

2026-05-14 Thread via GitHub 


YutaLin commented on PR #4338:
URL: 
https://github.com/apache/datafusion-comet/pull/4338#issuecomment-4455971464

   This patch also help #4150 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

[PR] feat: Support Spark expression: make_dt_interval [datafusion-comet]

2026-05-14 Thread via GitHub 


YutaLin opened a new pull request, #4338:
URL: https://github.com/apache/datafusion-comet/pull/4338

   ## Which issue does this PR close?
   
   Closes #3098 
   
   ## Rationale for this change
   Support expression `make_dt_interval`
   
   ## What changes are included in this PR?
   * Support DateTimeIntervalType in `QueryPlanSerd`, `serd.rs` and 
`types.proto` 
   * Leverage make_dt_interval from datafusion-spark, so update `jni_api.rs`, 
`datatime.scala` and `QueryPlanSerd`
   * Support duration(microsend) in ArrowType
   * Support null duration
   
   ## How are these changes tested?
   Add `make_dt_interval` sql and test in spark3.4/3.5/4.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]