itholic commented on code in PR #46063:
URL: https://github.com/apache/spark/pull/46063#discussion_r1566678165
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/QueryContexts.scala:
##########
@@ -160,6 +160,8 @@ case class DataFrameQueryContext(
val pysparkFragment: String = pysparkErrorContext.map(_._1).getOrElse("")
val pysparkCallSite: String = pysparkErrorContext.map(_._2).getOrElse("")
+ PySparkCurrentOrigin.clear()
Review Comment:
I tried get and clear at the first time, but it seems not working expectedly:
```scala
def getAndClear(): Option[(String, String)] = {
val context = pysparkErrorContext.get()
// Clear the context after retrieving it
pysparkErrorContext.remove()
context
}
```
The error context is missing in somewhere, so it raises exception:
```
Expected PySpark fragment was 'divide', got ''
'' != 'divide'
Expected :'divide'
Actual :''
```
So I put the clear here right after getting the `pysparkFragment` and
`pysparkCallSite`, it works as expectedly.
I guess the item is removed when clear the ThreadLocal?
##########
sql/api/src/main/scala/org/apache/spark/sql/catalyst/trees/QueryContexts.scala:
##########
@@ -160,6 +160,8 @@ case class DataFrameQueryContext(
val pysparkFragment: String = pysparkErrorContext.map(_._1).getOrElse("")
val pysparkCallSite: String = pysparkErrorContext.map(_._2).getOrElse("")
+ PySparkCurrentOrigin.clear()
Review Comment:
I tried get and clear at the first time, but it seems not working expectedly:
```scala
def getAndClear(): Option[(String, String)] = {
val context = pysparkErrorContext.get()
// Clear the context after retrieving it
pysparkErrorContext.remove()
context
}
```
The error context is missing in somewhere, so it raises exception:
```
Expected PySpark fragment was 'divide', got ''
'' != 'divide'
Expected :'divide'
Actual :''
```
So I put the clear here right after getting the `pysparkFragment` and
`pysparkCallSite`, it works as expectedly.
I guess the item is removed when clearing the ThreadLocal?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
