Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
JkSelf merged PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2921140089 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
JkSelf commented on code in PR #9733:
URL: https://github.com/apache/incubator-gluten/pull/9733#discussion_r2115034294
##
gluten-substrait/src/main/scala/org/apache/gluten/execution/WriteFilesExecTransformer.scala:
##
@@ -69,6 +71,83 @@ case class WriteFilesExecTransformer(
val caseInsensitiveOptions: CaseInsensitiveMap[String] =
CaseInsensitiveMap(options)
+ private def preProjectionNeeded(): Boolean = {
+if (partitionColumns == null || partitionColumns.isEmpty) {
Review Comment:
This change is necessary solely for the Velox backend. We need to implement
a check in BackendSettingsApi.scala to set it as disabled by default and enable
it specifically for VeloxBackendSettings. You can refer this code.
https://github.com/apache/incubator-gluten/blob/main/gluten-substrait/src/main/scala/org/apache/gluten/backendsapi/BackendSettingsApi.scala#L65C7-L65C24
https://github.com/apache/incubator-gluten/blob/main/backends-velox/src/main/scala/org/apache/gluten/backendsapi/velox/VeloxBackend.scala#L383
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
JkSelf commented on code in PR #9733:
URL: https://github.com/apache/incubator-gluten/pull/9733#discussion_r2115006289
##
gluten-substrait/src/main/scala/org/apache/gluten/execution/WriteFilesExecTransformer.scala:
##
@@ -69,6 +71,83 @@ case class WriteFilesExecTransformer(
val caseInsensitiveOptions: CaseInsensitiveMap[String] =
CaseInsensitiveMap(options)
+ private def preProjectionNeeded(): Boolean = {
+if (partitionColumns == null || partitionColumns.isEmpty) {
Review Comment:
Can we also skip the `partitionColumns.size() == 1` use case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2920555866 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2920469688 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2919712316 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2917778791 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2917078937 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2916716423 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-291450 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2913301033 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
JkSelf commented on code in PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#discussion_r2103723268 ## gluten-substrait/src/main/scala/org/apache/gluten/execution/WriteFilesExecTransformer.scala: ## @@ -118,7 +193,7 @@ case class WriteFilesExecTransformer( } RelBuilder.makeWriteRel( - input, + inputRelNode, Review Comment: @dmsuehir After establishing the correct partition order in the pre-project , it seems necessary to also update the order of the `typeNodes`, `nameList `and `columnTypeNodes`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2902642424 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8663][VL] Fix column directory structure for partitioned writes [incubator-gluten]
github-actions[bot] commented on PR #9733: URL: https://github.com/apache/incubator-gluten/pull/9733#issuecomment-2902641981 https://github.com/apache/incubator-gluten/issues/8663 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
