HeartSaVioR commented on code in PR #49751:
URL: https://github.com/apache/spark/pull/49751#discussion_r1945951586
##########
python/pyspark/sql/tests/pandas/test_pandas_transform_with_state.py:
##########
@@ -1470,6 +1470,39 @@ def check_exception(error):
check_exception=check_exception,
)
+ def test_not_nullable_fails(self):
Review Comment:
Yes and probably also no.
I agree moderate users may not ever try to get over and just stick with case
class or POJO or so. But "we" can imagine a way to get over, exactly the same
way how we could support PySpark:
```
override protected val stateEncoder: ExpressionEncoder[Any] =
ExpressionEncoder(stateType).resolveAndBind().asInstanceOf[ExpressionEncoder[Any]]
```
This is how we come up with state encoder for Python version of FMGWS. This
is to serde with Row interface - my rough memory says it's not InternalRow but
Row, so, most likely work with GenericRow, but we can try with both GenericRow
and InternalRow.
I'm OK with deferring this as follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
