Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
jinchengchenghh merged PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-3018964635 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-3018513584 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-3018399645 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-3018210615 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-3018212707 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-3018080859 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-3018051608 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2921577511 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2918734690 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2918481965 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2918039708 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2917978989 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2917969140 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
zhztheplayer commented on code in PR #9634:
URL: https://github.com/apache/incubator-gluten/pull/9634#discussion_r2111802643
##
cpp/core/jni/JniWrapper.cc:
##
@@ -397,11 +397,15 @@
Java_org_apache_gluten_vectorized_PlanEvaluatorJniWrapper_nativeCreateKernelWith
jint partitionId,
jlong taskId,
jboolean enableDumping,
-jstring spillDir) {
+jstring spillDir,
+jboolean enableCudf) {
JNI_METHOD_START
auto ctx = getRuntime(env, wrapper);
- auto& conf = ctx->getConfMap();
+ auto conf = ctx->getConfMap();
+ if (enableCudf) {
+conf[kCudfEnabled] = "true";
+ }
Review Comment:
Looks good so far. Opened
https://github.com/apache/incubator-gluten/issues/9785 for further enhancement.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2915745499 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
zhztheplayer commented on code in PR #9634:
URL: https://github.com/apache/incubator-gluten/pull/9634#discussion_r2111356328
##
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:
##
@@ -198,6 +208,12 @@ trait TransformSupport extends ValidatablePlan {
// When true, it will not generate relNode, nor will it generate native
metrics.
def isNoop: Boolean = false
+
+ // If set the isCudf, the config will send to native backend,
+ // and enable the cudf plan or not in runtime stage
+ def setIsCudf: Unit = {
+isCudf = true
+ }
Review Comment:
Probably it's after all Gluten rules are executed but I don't recall
precisely. Can we turn this into a plan tag or so?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
jinchengchenghh commented on code in PR #9634:
URL: https://github.com/apache/incubator-gluten/pull/9634#discussion_r2110801224
##
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:
##
@@ -198,6 +208,12 @@ trait TransformSupport extends ValidatablePlan {
// When true, it will not generate relNode, nor will it generate native
metrics.
def isNoop: Boolean = false
+
+ // If set the isCudf, the config will send to native backend,
+ // and enable the cudf plan or not in runtime stage
+ def setIsCudf: Unit = {
+isCudf = true
+ }
Review Comment:
What's the rule sequence? Can we make the CudfNodeReplaceRule make effect
after reuse rule?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
zhztheplayer commented on code in PR #9634:
URL: https://github.com/apache/incubator-gluten/pull/9634#discussion_r2109426519
##
cpp/velox/compute/WholeStageResultIterator.cc:
##
@@ -572,6 +576,12 @@ std::unordered_map
WholeStageResultIterator::getQueryC
configs[velox::core::QueryConfig::kSparkLegacyStatisticalAggregate] =
std::to_string(veloxCfg_->get(kSparkLegacyStatisticalAggregate,
false));
+#ifdef GLUTEN_ENABLE_GPU
+if (!enableCudf_) {
Review Comment:
@jinchengchenghh
Is there any reason for passing `enableCudf` by method calls down to here,
instead of reading a configuration option? I am a bit concerned about such many
APIs have to be changed for adding the parameter?
##
gluten-substrait/src/main/scala/org/apache/gluten/execution/WholeStageTransformer.scala:
##
@@ -198,6 +208,12 @@ trait TransformSupport extends ValidatablePlan {
// When true, it will not generate relNode, nor will it generate native
metrics.
def isNoop: Boolean = false
+
+ // If set the isCudf, the config will send to native backend,
+ // and enable the cudf plan or not in runtime stage
+ def setIsCudf: Unit = {
+isCudf = true
+ }
Review Comment:
A flag like this will break Spark's plan immutability / equality? For
example, the subquery reuse rule will find a non-cudf node and use it to
replace a cudf node since they are actually equal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2897949063 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2895249905 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2890561814 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] [GLUTEN-8851][VL] Validate the plan and use runtime config to enable it [incubator-gluten]
github-actions[bot] commented on PR #9634: URL: https://github.com/apache/incubator-gluten/pull/9634#issuecomment-2887220520 Run Gluten Clickhouse CI on x86 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
