GitHub user yashs360 opened a pull request:
https://github.com/apache/spark/pull/21541
[SPARK-20168][Streaming Kinesis] Setting the timestamp directly would cause
exception on â¦
Setting the timestamp directly would cause exception on reading stream, it
can be set directly only if the mode is not AT_TIMESTAMP
## What changes were proposed in this pull request?
The last patch in the kinesis streaming receiver sets the timestamp for the
mode AT_TIMESTAMP, but this mode can only be set via the
`baseClientLibConfiguration.withTimestampAtInitialPositionInStream()
`
and can't be set directly using
`.withInitialPositionInStream()`
This patch fixes the issue.
## How was this patch tested?
Kinesis Receiver doesn't expose the internal state outside, so couldn't
find the right way to test this change. Seeking for tips from other
contributors here.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/yashs360/spark ysharma/fix_kinesis_bug
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/21541.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #21541
commit 133ba8721319efa9bfa64fbb53a690b5140c957b
Author: Yash Sharma
Date: 2018-06-12T17:33:51Z
SPARK-20168: Setting the timestamp directly would cause exception on
reading stream, it can be set directly only if the mode is not AT_TIMESTAMP
---
-
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
