erenavsarogullari edited a comment on pull request #35536: URL: https://github.com/apache/spark/pull/35536#issuecomment-1042392854
Thanks @srowen for quick feedback. Please find some samples per operation as follows. I also attach whole sql response to jira in order to give clear view for future references: https://issues.apache.org/jira/secure/attachment/13040136/Spark_SQL_REST_Result_with-nodeDesc I think on following options if makes sense and please feel free to extend: - We can exclude `WholeStageCodegen` and if (`nodeName == nodeDesc`) for all use-cases. - **Option 1:** We can expose SQLConf in order to include white-listed operators. With this option, the user can define the operator white-list. (We can also define default white-list and user can extend with this SQLConf). For example: ``` spark.sql.rest.api.include.operators.from.desc=SortMergeJoin,HashAggregate,Project ``` - **Option 2:** We can also expose another SQLConf if end user wants to see `nodeDesc` of all operators. For example: ``` spark.sql.rest.api.enable.all.operators.desc=false(default) ``` **Sample Node/Operator Descriptions:** ``` "nodeName" : "SerializeFromObject", "nodeDesc" : "SerializeFromObject [knownnotnull(assertnotnull(input[0, org.apache.spark.status.api.v1.sql.Salary, true])).personId AS personId#17, knownnotnull(assertnotnull(input[0, org.apache.spark.status.api.v1.sql.Salary, true])).salary AS salary#18]", "nodeName" : "Project", "nodeDesc" : "Project [personId#17 AS id#24, salary#18]", "nodeName" : "Exchange", "nodeDesc" : "Exchange hashpartitioning(name#5, age#6, salary#18, 5), ENSURE_REQUIREMENTS, [id=#87]", "nodeName" : "Sort", "nodeDesc" : "Sort [id#24 ASC NULLS FIRST], false, 0", "nodeName" : "Scan", "nodeDesc" : "Scan[obj#3]", "nodeName" : "SortMergeJoin", "nodeDesc" : "SortMergeJoin [id#4], [id#24], Inner", "nodeName" : "HashAggregate", "nodeDesc" : "HashAggregate(keys=[name#5, age#6, knownfloatingpointnormalized(normalizenanandzero(salary#18)) AS salary#18], functions=[partial_avg(age#6), partial_avg(salary#18)])", "nodeName" : "Filter", "nodeDesc" : "Filter org.apache.spark.status.api.v1.sql.SqlResourceWithActualMetricsSuite$$Lambda$1666/[email protected]", // Following ones need exclusion "nodeName" : "WholeStageCodegen (3)", "nodeDesc" : "WholeStageCodegen (3)", "nodeName" : "Project", "nodeDesc" : "Project", ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
