[ https://issues.apache.org/jira/browse/SPARK-19593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15867214#comment-15867214 ]
Sarath Chandra Jiguru edited comment on SPARK-19593 at 2/15/17 5:52 AM: ------------------------------------------------------------------------ Type of the ticket is question. In KinesisReceiver.scala, currently it is possible to use default values of KinesisClientLibConfiguration. Due to this, even the stream is capable of serving the required read rate, the kinesis spark streaming consumer is not able to use it. Don't you think, it is highly needed to allow spark consumers to tweak these configurations? was (Author: sarathjiguru): See the Type of the ticket is question. In KinesisReceiver.scala, currently it is possible to use default values of KinesisClientLibConfiguration. Due to this, even the stream is capable of serving the required read rate, the kinesis spark streaming consumer is not able to use it. Don't you think, it is highly needed to allow spark consumers to tweak these configurations? > Records read per each kinesis transaction > ----------------------------------------- > > Key: SPARK-19593 > URL: https://issues.apache.org/jira/browse/SPARK-19593 > Project: Spark > Issue Type: Question > Components: DStreams > Affects Versions: 2.0.1 > Reporter: Sarath Chandra Jiguru > Priority: Trivial > > The question is related to spark streaming+kinesis integration > Is there a way to provide a kinesis consumer configuration. Ex: Number of > records read per each transaction etc. > Right now, even though, I am eligible to read 2.8G/minute, I am restricted to > read only 100MB/minute, as I am not able to increase the default number of > records in each transaction. > I have raised a question in stackoverflow as well, please look into it: > http://stackoverflow.com/questions/42107037/how-to-alter-kinesis-consumer-properties-in-spark-streaming > Kinesis stream setup: > open shards: 24 > write rate: 440K/minute > read rate: 1.42M/minute > read byte rate: 85 MB/minute. I am allowed to read around 2.8G/minute(24 > Shards*2 MB*60 Seconds) > Reference: > http://docs.aws.amazon.com/streams/latest/dev/kinesis-record-processor-additional-considerations.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org