qlong commented on code in PR #53625:
URL: https://github.com/apache/spark/pull/53625#discussion_r2653193807
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/ui/SQLListener.scala:
##########
@@ -55,7 +57,12 @@ case class SparkListenerSQLExecutionStart(
time: Long,
modifiedConfigs: Map[String, String] = Map.empty,
jobTags: Set[String] = Set.empty,
- jobGroupId: Option[String] = None)
+ jobGroupId: Option[String] = None,
+ // A unique identifier for the query execution. For the first execution it
equals
+ // QueryExecution.queryId, for subsequent executions a new UUIDv7 is
generated.
+ // For backward compatibility, queryId is None when we parse event logs
+ // generated by old versions of Spark.
+ queryId: Option[UUID] = None)
Review Comment:
This queryID is from SQLExecution which tracks each execution. It is not the
queryID from QueryExecution. I think it should be renamed to executionUUID, and
add queryID in the future when we need to track query cross the whole life
cycle starting from parser
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/SQLExecution.scala:
##########
@@ -92,8 +93,17 @@ object SQLExecution extends Logging {
val sparkSession = queryExecution.sparkSession
val sc = sparkSession.sparkContext
val oldExecutionId = sc.getLocalProperty(EXECUTION_ID_KEY)
+ val oldQueryId = sc.getLocalProperty(QUERY_ID_KEY)
val executionId = SQLExecution.nextExecutionId
+ // Use the original queryId for the first execution, generate new ones for
+ // subsequent executions
+ val queryId = if (queryExecution.firstExecution.compareAndSet(true,
false)) {
+ queryExecution.queryId
+ } else {
+ UUIDv7Generator.generate()
+ }
Review Comment:
i think queryId should set to **queryExecution.queryId** so it it is
stable cross mutiple executions, and we use a **new exuecutionUUID** to track
each new execution. Otherwise we cannot track the queryId when we need it in
the future, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
