Github user MaxGekk commented on a diff in the pull request:
https://github.com/apache/spark/pull/22429#discussion_r218211903
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/QueryExecution.scala ---
@@ -250,5 +254,36 @@ class QueryExecution(val sparkSession: SparkSession,
val logical: LogicalPlan) {
def codegenToSeq(): Seq[(String, String)] = {
org.apache.spark.sql.execution.debug.codegenStringSeq(executedPlan)
}
+
+ /**
+ * Dumps debug information about query execution into the specified
file.
+ */
+ def toFile(path: String): Unit = {
+ val maxFields = SparkEnv.get.conf.getInt(Utils.MAX_TO_STRING_FIELDS,
+ Utils.DEFAULT_MAX_TO_STRING_FIELDS)
+ val filePath = new Path(path)
+ val fs = FileSystem.get(filePath.toUri,
sparkSession.sessionState.newHadoopConf())
+ val writer = new BufferedWriter(new
OutputStreamWriter(fs.create(filePath)))
+
+ try {
+ SparkEnv.get.conf.set(Utils.MAX_TO_STRING_FIELDS,
Int.MaxValue.toString)
--- End diff --
> It is generally a bad idea to change this conf as people expect that it
is immutable.
Maybe it is generally a bad idea but the purpose of the `toFile` is to call
it during debug of an issue. Currently when someone need to output all fields,
he/she has to set `spark.debug.maxToStringFields` before `explain()` and revert
it back after `explain()`. If you want, I can add a comment which tells to an
user that the `toFile` changes the config during its invoke.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
