Jeff Nadler created SPARK-17539:
-----------------------------------
Summary: Streaming Backpressure Starves DirectStream When Used In
Combination With Receivers
Key: SPARK-17539
URL: https://issues.apache.org/jira/browse/SPARK-17539
Project: Spark
Issue Type: Bug
Components: Streaming
Affects Versions: 1.5.0
Environment: Spark 1.5.0 under CDH 5.5.1 w/ Kafka 0.8.2.1
Reporter: Jeff Nadler
Backpressure seems to work fine for Direct Streams when used alone.
We use Receivers (x4) to consume Topic A, and a Direct Stream to consume Topic
B in a single Streaming app. Both go through updateStateByKey and are later
joined.
In this configuration, backpressure works as expected for the Receivers but not
for the Direct Stream. After running for some time there is no scheduling
delay - there is some at startup, but it burns off in a minute or two.
The Direct Stream will get 'stuck' throttled at 1 msg / partition / sec. As
soon as I turn off backpressure this problem goes away.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
