[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-03-30 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 You're a gen @HyukjinKwon 💯 . I will wait for Tathagata and Burak's inputs then :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #17467: Ysharma/spark kinesis retries

2017-03-29 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Have got this JIRA ticket for the patch - https://issues.apache.org/jira/browse/SPARK-20140 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub

[GitHub] spark issue #17467: Ysharma/spark kinesis retries

2017-03-29 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Oh right you meant the PR title format. I will reject this pr and post a new one. Thanks @HyukjinKwon --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-03-29 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @HyukjinKwon What should be the next steps for this PR. Are there any Spark-Kinesis experts who can review the patch ? --- If your project is set up for it, you can reply to this email and have

[GitHub] spark pull request #17467: Ysharma/spark kinesis retries

2017-03-29 Thread yssharma
GitHub user yssharma opened a pull request: https://github.com/apache/spark/pull/17467 Ysharma/spark kinesis retries ## What changes were proposed in this pull request? The pull requests proposes to remove the hardcoded values for Amazon Kinesis - MIN_RETRY_WAIT_TIME_MS

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen - Yes, 2 objectives for the patch: - add new testcases for new api (that also tests aws session tokens) - ability to disable old api test cases For now I have added new test

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen - Have made following improvements to the patch to incorporate your points: - Did not remove old testcases for keeping the checks on old api - Added new testcases that test the new

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Can I get some feedback here please @tdas @brkyvz Thanks :) --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...

2017-04-01 Thread yssharma
GitHub user yssharma opened a pull request: https://github.com/apache/spark/pull/17506 [SPARK-20189][DStream] Fix spark kinesis testcases to remove deprecated createStream and use Builders ## What changes were proposed in this pull request? The spark-kinesis testcases

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 The Scala style check fail because of the double spaced lines probably. But that's how the existing code was so thought of keeping it that way. --- If your project is set up for it, you can reply

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @srowen - Could I get some love here as well. Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen - Thanks for the feedback. Appreciate it. Added a minimal simplified patch which fixes the testcases that fail with the old api. --- If your project is set up for it, you can reply

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen : It failed on the earlier patch with KinesisTestUtils.scala changes. This version is clean. Will wait for the next automated build :) Build logs: https://amplab.cs.berkeley.edu

[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...

2017-04-02 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17506#discussion_r109304601 --- Diff: external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisStreamSuite.java --- @@ -50,48 +56,86 @@ public void

[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...

2017-04-02 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17506#discussion_r109304590 --- Diff: external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisStreamSuite.java --- @@ -50,48 +56,86 @@ public void

[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...

2017-04-02 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17506#discussion_r109304637 --- Diff: external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisStreamSuite.java --- @@ -50,48 +56,86 @@ public void

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-11 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Waiting for review @brkyvz . Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-11 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen do you feel this patch could be merged now ? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-06 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 Thanks @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-05 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 @srowen - does the Jenkins re-test trigger automatically? else, could I request a retest on this patch please ? --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...

2017-04-12 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17506#discussion_r111292109 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -267,7 +267,7 @@ object

[GitHub] spark pull request #17506: [SPARK-20189][DStream] Fix spark kinesis testcase...

2017-04-12 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17506#discussion_r111291842 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisStreamSuite.scala --- @@ -233,11 +241,15 @@ abstract class

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-14 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 Thanks @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde @brkyvz - Implemented the review changes. Please review. - Using SparkConf for all the user parameters - removed kinesisWait to be val instead of var - fixed documentation

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-21 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112614624 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -83,7 +83,8 @@ class

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-19 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112346805 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -147,6 +152,14 @@ class

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde - thanks for taking time to review it. Appreciate it. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-19 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112347174 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -249,6 +252,17 @@ object

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-23 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @felixcheung : would love to get more review comments to improve the patch. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-23 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde - Added changes for the minor review comments and docs. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848401 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDDSuite.scala --- @@ -101,6 +103,36 @@ abstract class

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848762 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -135,7 +139,8 @@ class

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848373 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDDSuite.scala --- @@ -101,6 +103,36 @@ abstract class

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848363 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -135,7 +139,8 @@ class

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-04-23 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r112848855 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisBackedBlockRDD.scala --- @@ -135,7 +139,8 @@ class

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-22 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde - Not sure if we can exactly test the configured timeouts. I have debugged the flow in my testcase to verify that the custom values are passed down to the Kinesis backed block rdd

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-22 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde : Implemented review changes and checked scala code style checks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-08 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @brkyvz - thanks for taking time to review the patch. appreciate it. Implemented all your suggestions. Now passing a new map for the kinesis configs and added mechanism to use the builder

[GitHub] spark issue #17506: [SPARK-20189][DStream] Fix spark kinesis testcases to re...

2017-04-08 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17506 Is there anything else that can be done on this patch. The patch fixes all the deprecated api testcases that try to use the aws secret/id credentials instead of the builder. --- If your project

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-08 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Just for info, while trying to use the `sc` in the `KinesisBackedBlockRDD ` : `- Basic reading from Kinesis *** FAILED *** org.apache.spark.SparkException: Task not serializable

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-03-30 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @tdas @brkyvz - Added new changes for removing other constants hardcoded at multiple places. - Squashed 3 commits into single commit --- If your project is set up for it, you can reply

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-07-08 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 I really appreciate you taking time to review the changes. I know balancing work and foss contributions is difficult :) I have given a clean new stab at the issue with all the feedbacks. Let me

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-07-13 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Hi, @budde , @brkyvz would love to hear your thoughts on the new patch --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...

2017-07-23 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 @HyukjinKwon The PR is ready. Just waiting for some 👍 from the reviewers. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...

2017-07-23 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 I noticed that now. Yes I will post an updated patch today. Thanks @HyukjinKwon --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...

2017-07-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 Thanks @budde @srowen --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18029: [SPARK-20168][DStream] Add changes to use kinesis fetche...

2017-07-26 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Would love to work on the review feedback here @budde @brkyvz . Please have a look when you have time. --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark issue #18029: [SPARK-20168][DStream] Add changes to use kinesis fetche...

2017-07-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde @brkyvz - could I get some love here please. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18071: [SPARK-20855][Docs][DStream] Update the Spark kinesis do...

2017-07-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 Updated the patch @HyukjinKwon . Thanks for the note. 👍 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-26 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde would you like to share your thoughts on the new changes when you have time ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-04-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @brkyvz - Added new changes that adds - - A case class `KinesisReadConfigurations` that adds all the kinesis read configs in a single place - A test class that passes the kinesis configs

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-05 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @brkyvz , @budde, @HyukjinKwon could you please review this sometime. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks @budde for the review . Love the suggestions. 👍 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @brkyvz I have made the suggested modifications to the code. Please have a look when you get time. Thanks --- If your project is set up for it, you can reply to this email and have your reply

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-08-20 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r134138994 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Added review suggestions @budde ! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Will update and post another request seen. Thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-08-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Will wait for @brkyvz , @HyukjinKwon for final ☑️ --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-14 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @budde @brkyvz - Any feed back on this one please ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-16 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Awesome. Thanks @budde @brkyvz for reviews and patch improvements. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-09 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 For review @budde @brkyvz ^ this looks in a good state. @brkyvz - I still didn't quite get the code bit you shared, would love to make changes if you could explain that piece of your mind

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-19 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde @brkyvz could you suggest if the current patch seems ok, or I should make something similar to the case class/ trait ? --- If your project is set up for it, you can reply to this email

[GitHub] spark pull request #18028: [DStream][DOC]Add documentation for kinesis retry...

2017-05-18 Thread yssharma
GitHub user yssharma opened a pull request: https://github.com/apache/spark/pull/18028 [DStream][DOC]Add documentation for kinesis retry configurations ## What changes were proposed in this pull request? The changes were merged as part of - https://github.com/apache/spark

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-05-18 Thread yssharma
GitHub user yssharma opened a pull request: https://github.com/apache/spark/pull/18029 [SPARK-20168][WIP][DStream] Add changes to use kinesis fetches from specified timestamp ## What changes were proposed in this pull request? Kinesis client can resume from a specified

[GitHub] spark pull request #18071: [SPARK-20855][WIP][DStream] Update the Spark kine...

2017-05-23 Thread yssharma
GitHub user yssharma opened a pull request: https://github.com/apache/spark/pull/18071 [SPARK-20855][WIP][DStream] Update the Spark kinesis docs to use the KinesisInputDStream builder instead of deprecated KinesisUtils ## What changes were proposed in this pull request

[GitHub] spark issue #18071: [SPARK-20855][WIP][DStream] Update the Spark kinesis doc...

2017-05-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18071 Hey @budde , drawing your attention here please. Just wanted to update the docs with the new Builder interface by you. Few questions on top of my mind - - Is it compatible with the python

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-05-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Commit https://github.com/apache/spark/commit/424550c8450937f78ce608ff7b18e46f41478a8a should fix the timeouts mentioned in the https://github.com/apache/spark/commit

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-05-22 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Simplified how timestamp is passed to Kinesis for InitialPositionInStream.AT_TIMESTAMP. cc: @budde , @brkyvz --- If your project is set up for it, you can reply to this email and have your

[GitHub] spark issue #18028: [DStream][DOC]Add documentation for kinesis retry config...

2017-05-18 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18028 Thanks @brkyvz --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-05-18 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde @brkyvz would love to hear your thoughts if this is the best way to add this functionality --- If your project is set up for it, you can reply to this email and have your reply appear

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-05-31 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Could I get some love here please @budde @brkyvz This change is pretty useful for us and I would like to bake this patch well so others can take advantage of this feature. --- If your

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde Added a new method to pass the timestamp more elegantly. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde @brkyvz - Would appreciate if you could share some tips by which I could include these external kinesis modules on spark-shell to test this directly from shell (both for scala and python

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-03 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r119984045 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -38,6 +40,7 @@ private[kinesis] class

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-05 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 ``` ./bin/spark-shell --jars /Users/ysharma/.m2/repository/org/apache/spark/spark-streaming-kinesis-asl_2.11/2.3.0-SNAPSHOT/spark-streaming-kinesis-asl_2.11-2.3.0-SNAPSHOT.jar --packages

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-03 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r119986035 --- Diff: external/kinesis-asl/src/test/java/org/apache/spark/streaming/kinesis/JavaKinesisInputDStreamBuilderSuite.java --- @@ -45,7 +46,7 @@ public

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-03 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r119986280 --- Diff: external/kinesis-asl/src/test/scala/org/apache/spark/streaming/kinesis/KinesisInputDStreamBuilderSuite.scala --- @@ -111,5 +110,28 @@ class

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-13 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thoughts - @budde @brkyvz ? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled

[GitHub] spark issue #18029: [SPARK-20168][WIP][DStream] Add changes to use kinesis f...

2017-06-16 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde What would that mean for existing applications that are sending an `InitialPositionInStream.SOMETHING` to the the builder. We would need to provide multiple overloaded versions

[GitHub] spark pull request #17467: [SPARK-20140][DStream] Remove hardcoded kinesis r...

2017-05-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/17467#discussion_r115110773 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisReadConfigurations.scala --- @@ -0,0 +1,71

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-05 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Thanks for all the review comments @budde @brkyvz . Added new review changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 Thanks for the comments @budde @brkyvz . Would be adding the changes soon. I too liked pulling the values of spark conf directly and got it working with the private val

[GitHub] spark issue #17467: [SPARK-20140][DStream] Remove hardcoded kinesis retry wa...

2017-05-04 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/17467 @brkyvz - Thanks for the review comments. Updated the patch, please review. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r120234200 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -100,6 +103,7 @@ object

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r120235619 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -193,6 +197,21 @@ object

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r120236059 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -100,6 +103,7 @@ object

[GitHub] spark pull request #18029: [SPARK-20168][WIP][DStream] Add changes to use ki...

2017-06-05 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r120236128 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisInputDStream.scala --- @@ -193,6 +197,21 @@ object

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-09-20 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @budde could you please do one last review of this one. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-10-11 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks @brkyvz . Appreciate the review comments. I am facing some issues with the scala packaging. I will keep working on it and update the patch soon

[GitHub] spark issue #19427: [SparkStreaming] Reset spark.driver.bindAddress when sta...

2017-10-12 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/19427 I don't think there is an automated way. You could create a JIRA ticket and rename this title with the ticket id and component name

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-10-12 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks for the review @brkyvz . Updated the pull request with version 2.3.0. --- - To unsubscribe, e-mail: reviews-unsubscr

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-09 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137926490 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-06 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137235013 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-09-06 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Thanks for the comments @brkyvz . I will be working on the changes and update the PR very soon. --- - To unsubscribe, e-mail

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-06 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137245127 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-06 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137237166 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark pull request #18029: [SPARK-20168] [DStream] Add changes to use kinesi...

2017-09-07 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/18029#discussion_r137684968 --- Diff: external/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/InitialPosition.scala --- @@ -0,0 +1,104 @@ +/* + * Licensed

[GitHub] spark pull request #19416: [SPARK-22187][SS] Update unsaferow format for sav...

2017-10-03 Thread yssharma
Github user yssharma commented on a diff in the pull request: https://github.com/apache/spark/pull/19416#discussion_r142368186 --- Diff: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/FlatMapGroupsWithState_StateManager.scala --- @@ -0,0 +1,143

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-10-03 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 Hi @brkyvz can I get some love here please. --- - To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional

[GitHub] spark issue #18029: [SPARK-20168] [DStream] Add changes to use kinesis fetch...

2017-09-24 Thread yssharma
Github user yssharma commented on the issue: https://github.com/apache/spark/pull/18029 @brkyvz Could you please check this for the last suggestions ? --- - To unsubscribe, e-mail: reviews-unsubscr

  1   2   >