Copilot commented on code in PR #1657:
URL: https://github.com/apache/auron/pull/1657#discussion_r2554636186
##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala:
##########
@@ -467,9 +471,19 @@ object AuronConverters extends Logging {
relation.fileFormat match {
case p if p.getClass.getName.endsWith("ParquetFileFormat") =>
assert(enableScanParquet)
+ if (!enableScanParquetTimestamp) {
+ assert(
+ !exec.schema.exists(e => existTimestampType(e.dataType)),
+ "Parquet scan with timestamp type is not supported")
+ }
addRenameColumnsExec(Shims.get.createNativeParquetScanExec(exec))
case p if p.getClass.getName.endsWith("OrcFileFormat") =>
assert(enableScanOrc)
+ if (!enableScanOrcTimestamp) {
+ assert(
+ !exec.schema.exists(e => existTimestampType(e.dataType)),
Review Comment:
Consider using `requiredSchema` instead of `schema` for consistency with the
rest of the function. The function already extracts `requiredSchema` from
`exec.requiredSchema` at line 455 and logs it. Using `requiredSchema` (which
represents the fields actually being read) is more consistent with what's being
validated and matches the pattern used elsewhere in this function.
##########
spark-extension/src/main/scala/org/apache/spark/sql/auron/AuronConverters.scala:
##########
@@ -467,9 +471,19 @@ object AuronConverters extends Logging {
relation.fileFormat match {
case p if p.getClass.getName.endsWith("ParquetFileFormat") =>
assert(enableScanParquet)
+ if (!enableScanParquetTimestamp) {
+ assert(
+ !exec.schema.exists(e => existTimestampType(e.dataType)),
Review Comment:
Consider using `requiredSchema` instead of `schema` for consistency with the
rest of the function. The function already extracts `requiredSchema` from
`exec.requiredSchema` at line 455 and logs it. Using `requiredSchema` (which
represents the fields actually being read) is more consistent with what's being
validated and matches the pattern used elsewhere in this function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
