Github user jerryshao commented on the issue:
https://github.com/apache/spark/pull/20958
Thanks @tdas for your comments. I agree that socket source should only be
used in testing. But it doesn't mean that it can throw weird exception in
testing env. For example, if we're dumping socket source data to parquet in
test env, parquet source requires checkpoint (even in test env), but socket
source should not honor checkpoint data, this basically not works also in test
env. My thinking is that cases in test env should also be worked basically, but
doesn't have in-production features like checkpointing.
Maybe we can improve error log, but still cases like socket -> parquet
doesn't work.
Let me think about how to better address this issue. Thanks again.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
