HeartSaVioR commented on code in PR #50689:
URL: https://github.com/apache/spark/pull/50689#discussion_r2059400557
##########
python/pyspark/sql/streaming/list_state_client.py:
##########
@@ -118,7 +130,24 @@ def append_value(self, state_name: str, value: Tuple) ->
None:
def append_list(self, state_name: str, values: List[Tuple]) -> None:
import pyspark.sql.streaming.proto.StateMessage_pb2 as stateMessage
- append_list_call = stateMessage.AppendList()
+ send_data_via_arrow = False
+
+ # To workaround mypy type assignment check.
+ values_as_bytes: Any = []
+ if len(values) == 100:
Review Comment:
This is simply a magic number assuming 100 is enough to have higher cost of
having all schema string into the pickled format of row - maybe I will file a
JIRA ticket for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
