Jungtaek Lim created SPARK-40414: ------------------------------------ Summary: Fix PythonArrowInput and PythonArrowOutput to be more generic to handle complicated type/data Key: SPARK-40414 URL: https://issues.apache.org/jira/browse/SPARK-40414 Project: Spark Issue Type: Improvement Components: PySpark Affects Versions: 3.4.0 Reporter: Jungtaek Lim
During the work of flatMapGroupsWithState in PySpark, we figured out that we are unable to reuse PythonArrowInput and PythonArrowOutput, as PythonArrowInput and PythonArrowOutput are too specific to the strict input data (row) and output data. To reuse the implementations we should make these traits more general to handle more generic type of data. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org