[
https://issues.apache.org/jira/browse/SPARK-36520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Noam Tamim updated SPARK-36520:
-------------------------------
Description:
In `org.apache.spark.streaming.kinesis` there's a Java interface
`KinesisInitialPosition` - for some reason it's not public, so can't be used.
The reason it needs to be used is simple: it is the parent of
`KinesisInitialPositions.Latest` and `KinesisInitialPositions.TrimHorizon`.
Without using it, it's impossible to pass instances of `Latest` and
`TrimHorizon` around without calling them by name.
The workaround is to use the enum `InitialPositionInStream` instead, and a
factory method `fromKinesisInitialPosition(..)` - but it doesn't seem like the
correct pattern, because it belongs to Amazon Kinesis Client library and not
Spark (it's currently the only import in my class which is outside of
`org.apache.spark`).
The change is trivial, so it's mainly a matter of why *not* to do it.
KinesisInitialPositions:
https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/java/org/apache/spark/streaming/kinesis/KinesisInitialPositions.java
KinesisInitialPosition:
https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java
was:
In `org.apache.spark.streaming.kinesis` there's a Java interface
`KinesisInitialPosition` - for some reason it's not public, so can't be used.
The reason it needs to be used is simple: it is the parent of
`KinesisInitialPositions.Latest` and `KinesisInitialPositions.TrimHorizon`.
Without using it, it's impossible to pass instances of `Latest` and
`TrimHorizon` around without calling them by name.
The workaround is to use the enum `InitialPositionInStream` instead, and a
factory method `fromKinesisInitialPosition(..)` - but it doesn't seem like the
correct pattern.
> KinesisInitialPosition interface should be public
> -------------------------------------------------
>
> Key: SPARK-36520
> URL: https://issues.apache.org/jira/browse/SPARK-36520
> Project: Spark
> Issue Type: Improvement
> Components: DStreams
> Affects Versions: 3.1.2
> Reporter: Noam Tamim
> Priority: Minor
> Labels: kinesis
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> In `org.apache.spark.streaming.kinesis` there's a Java interface
> `KinesisInitialPosition` - for some reason it's not public, so can't be used.
> The reason it needs to be used is simple: it is the parent of
> `KinesisInitialPositions.Latest` and `KinesisInitialPositions.TrimHorizon`.
> Without using it, it's impossible to pass instances of `Latest` and
> `TrimHorizon` around without calling them by name.
> The workaround is to use the enum `InitialPositionInStream` instead, and a
> factory method `fromKinesisInitialPosition(..)` - but it doesn't seem like
> the correct pattern, because it belongs to Amazon Kinesis Client library and
> not Spark (it's currently the only import in my class which is outside of
> `org.apache.spark`).
> The change is trivial, so it's mainly a matter of why *not* to do it.
> KinesisInitialPositions:
> https://github.com/apache/spark/blob/master/external/kinesis-asl/src/main/java/org/apache/spark/streaming/kinesis/KinesisInitialPositions.java
> KinesisInitialPosition:
> https://github.com/awslabs/amazon-kinesis-client/blob/master/amazon-kinesis-client/src/main/java/software/amazon/kinesis/common/InitialPositionInStream.java
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
