HyukjinKwon commented on code in PR #46921:
URL: https://github.com/apache/spark/pull/46921#discussion_r1637258286
##########
python/pyspark/sql/streaming/listener.py:
##########
@@ -500,6 +505,19 @@ def fromJson(cls, j: Dict[str, Any]) ->
"StreamingQueryProgress":
else {},
)
+ def __getitem__(self, key: str) -> Any:
+ # Before Spark 4.0, StreamingQuery.lastProgress returns a dict, which
casts id and runId
+ # to string. But here they are UUID.
+ # To prevent breaking change, also cast them to string when accessed
with __getitem__.
+ if key == "id" or key == "runId":
+ return str(getattr(self, key))
+ else:
+ return getattr(self, key)
+
+ def __setitem__(self, key: str, value: Any) -> None:
Review Comment:
Can we remove the fixes of `__getitem__` and `__setitem__` but we do
`self.update(dict(id=id, runId=runId, ...))` at `__init__`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
