amaliujia commented on code in PR #41811:
URL: https://github.com/apache/spark/pull/41811#discussion_r1248340507
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/StringUtils.scala:
##########
@@ -63,3 +66,39 @@ class StringConcat(val maxLength: Int =
ByteArrayUtils.MAX_ROUNDED_ARRAY_LENGTH)
result.toString
}
}
+
+object SparkStringUtils extends Logging {
+ /** Whether we have warned about plan string truncation yet. */
+ private val truncationWarningPrinted = new AtomicBoolean(false)
+
+ /**
+ * Format a sequence with semantics similar to calling .mkString(). Any
elements beyond
+ * maxNumToStringFields will be dropped and replaced by a "... N more
fields" placeholder.
+ *
+ * @return the trimmed and formatted string.
+ */
+ def truncatedString[T](
+ seq: Seq[T],
+ start: String,
+ sep: String,
+ end: String,
+ maxFields: Int): String = {
+ if (seq.length > maxFields) {
+ if (truncationWarningPrinted.compareAndSet(false, true)) {
+ logWarning(
+ "Truncated the string representation of a plan since it was too
large. This " +
+ s"behavior can be adjusted by setting
'spark.sql.debug.maxToStringFields'.")
Review Comment:
This is an example of that the error message actually only needs to access
the SQLConf key `${SQLConf.MAX_TO_STRING_FIELDS.key}` so I replaced that with
the key itself to eliminate the need of SQLConf.
@cloud-fan please let me know if this is the proper way that you were
thinking of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
