Github user sardetushar commented on the issue:
https://github.com/apache/spark/pull/5008
Hi Can any one please help me, i am losing data when i killed the driver.
Below is my case
I have one JeroMQ publisher
Spark Streaming will start Custom JeroMQ receiver, WAL functionality is
enabled i can see the messages are being stored in receivedData dir in
checkpoint.
Not when i restart the spark streaming APP, if the receivedData messages
count is 25000 then only 400 are being processed in next restart.
I can confirm by looking at receivedData dir that i WAL received 25000
messages but after restart driver not able to fetch the records from WAL dir.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
