erenavsarogullari edited a comment on pull request #35536: URL: https://github.com/apache/spark/pull/35536#issuecomment-1042392854
Thanks @srowen for quick feedback. Please find some samples per operation as follows. I also attach whole sql response to jira in order to give clear view for future references: https://issues.apache.org/jira/secure/attachment/13040136/Spark_SQL_REST_Result_with-nodeDesc I think on following options: - We can exclude `WholeStageCodegen` and if (`nodeName == nodeDesc`) for all use-cases. - **Option 1:** We can expose SQLConf in order to include white-listed operators. With this option, the user can define the operator white-list. For example: ``` spark.sql.rest.api.include.operators.from.desc=SortMergeJoin,HashAggregate,Project ``` - **Option 2:** We can also expose another SQLConf if end user wants to see `nodeDesc` of all operators. For example: ``` spark.sql.rest.api.enable.all.operators.desc=false(default) ``` Sample Results: ``` "nodeName" : "SerializeFromObject", "nodeDesc" : "SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.status.api.v1.sql.Salary, true])).personId AS personId#17, knownnotnull(assertnotnull(input[0, org.apache.spark.status.api.v1.sql.Salary, true])).salary AS salary#18]", "nodeName" : "Project", "nodeDesc" : "Project [personId#17 AS id#24, salary#18]", "nodeName" : "Exchange", "nodeDesc" : "Exchange hashpartitioning(name#5, age#6, salary#18, 5), ENSURE_REQUIREMENTS, [id=#87]", "nodeName" : "Sort", "nodeDesc" : "Sort [id#24 ASC NULLS FIRST], false, 0", "nodeName" : "Scan", "nodeDesc" : "Scan[obj#3]", "nodeName" : "SortMergeJoin", "nodeDesc" : "SortMergeJoin [id#4], [id#24], Inner", "nodeName" : "HashAggregate", "nodeDesc" : "HashAggregate(keys=[name#5, age#6, knownfloatingpointnormalized(normalizenanandzero(salary#18)) AS salary#18], functions=[partial_avg(age#6), partial_avg(salary#18)])", "nodeName" : "Filter", "nodeDesc" : "Filter org.apache.spark.status.api.v1.sql.SqlResourceWithActualMetricsSuite$$Lambda$1666/[email protected]", // Following ones need exclusion "nodeName" : "WholeStageCodegen (3)", "nodeDesc" : "WholeStageCodegen (3)", "nodeName" : "Project", "nodeDesc" : "Project", ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
