HeartSaVioR commented on code in PR #54085:
URL: https://github.com/apache/spark/pull/54085#discussion_r2767805562
##########
python/pyspark/sql/streaming/python_streaming_source_runner.py:
##########
@@ -116,6 +138,77 @@ def send_batch_func(
write_int(EMPTY_PYARROW_RECORD_BATCHES, outfile)
+def check_support_func(reader: DataSourceStreamReader, outfile: IO) -> None:
+ support_flags = 0
+ if isinstance(reader, _SimpleStreamReaderWrapper):
Review Comment:
While I get how your proposal works, I'd argue that this is just another
workaround for doing something which we can't do from inheritance.
For example, you said SupportsTriggerAvailableNow isn't the important thing
- if we come to non-python language which has strong typing, explicitly
checking the type is mostly required except the hacky way like reflection in
Java - it is not a best practice. We do inspect for latestOffset but this is a
last resort approach, because python does not support method overloading so we
actually can't do the same evolution of API we did from Scala.
(It's still possible we can introduce a separate interface to achieve
similar thing but it is not about overloading - it's just that the latest
definition will just override the prior definition for the same name of method.)
Though I understand that in python it's duck typing and the type system is
crazy loose and the availability of function/method simply matters.
Maybe you might try to give a better approach for pythonic thinking. It's
just that both breaks inheritance and the proposal seems to break more than
what I do. I agree if you want to say this isn't pythonic, but then I could say
this is one of the pattern we heavily leverage in Scala, we call it as pattern
matching.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
