[
https://issues.apache.org/jira/browse/SAMZA-2129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shanthoosh Venkataraman updated SAMZA-2129:
-------------------------------------------
Description:
Moves the offset comparison check to SystemConsumers
* Currently the offset comparator check to find a lowest for a
`SystemStreamPartition` is duplicated in some implementations of SystemConsumer
API.
* Some `SystemConsumer` implementations do not perform this offset comparator
check
in the implementation of register method.
* Moving this one level up from
`SystemConsumer.register(SystemStreamPartition, offset)` API implementation to
`SystemConsumers.register(SystemStreamPartition, offset)` API implementation
removes unnecessary duplication and ensures functional correctness.
There should be no functional breakages introduced by moving the
offset-comparator check to `SystemConsumers` layer. Here's why. Metadata and
I/O topics currently use SystemConsumer API to read data.
* Checkpoint topic: This is a stream with one partition and is log-compacted.
Samza-container reads from the beginning of this stream. There's no offset
comparisons required for this topic.
* ChangeLog topic: To read change-log topic-partition, SamzaContainer uses
`StorageManagerUtil.getStartingOffset` to read from the change-log topic and
the offset comparator check to choose the lowest offset is currently performed
within it.
* Coordinator topic: This is a stream with one partition and is log-compacted.
SamzaContainer/ApplicationMaster is currently reads from the beginning of this
stream to get the entire state.
* I/O topics: `SamzaContainer` currently reads data from input and output
topics through `SystemConsumers` API. Moving this offset comparator check
should not affect the existing functionality.
was:
* Moves the offset comparison check to SystemConsumers
* Currently the offset comparator check to find a lowest for a
`SystemStreamPartition` is duplicated in some implementations of SystemConsumer
API.
* Some `SystemConsumer` implementations do not perform this offset comparator
check
in the implementation of register method.
* Moving this one level up from
`SystemConsumer.register(SystemStreamPartition, offset)` API implementation to
`SystemConsumers.register(SystemStreamPartition, offset)` API implementation
removes unnecessary duplication and ensures functional correctness.
There should be no functional breakages introduced by moving the
offset-comparator check to `SystemConsumers` layer. Here's why. Metadata and
I/O topics currently use SystemConsumer API to read data.
* Checkpoint topic: This is a stream with one partition and is log-compacted.
Samza-container reads from the beginning of this stream. There's no offset
comparisons required for this topic.
* ChangeLog topic: To read change-log topic-partition, SamzaContainer uses
`StorageManagerUtil.getStartingOffset` to read from the change-log topic and
the offset comparator check to choose the lowest offset is currently performed
within it.
* Coordinator topic: This is a stream with one partition and is log-compacted.
SamzaContainer/ApplicationMaster is currently reads from the beginning of this
stream to get the entire state.
* I/O topics: `SamzaContainer` currently reads data from input and output
topics through `SystemConsumers` API. Moving this offset comparator check
should not affect the existing functionality.
> Move the offset comparison check to choose the lowest offset to
> SystemConsumers.
> --------------------------------------------------------------------------------
>
> Key: SAMZA-2129
> URL: https://issues.apache.org/jira/browse/SAMZA-2129
> Project: Samza
> Issue Type: Improvement
> Reporter: Shanthoosh Venkataraman
> Assignee: Shanthoosh Venkataraman
> Priority: Major
>
> Moves the offset comparison check to SystemConsumers
> * Currently the offset comparator check to find a lowest for a
> `SystemStreamPartition` is duplicated in some implementations of
> SystemConsumer API.
> * Some `SystemConsumer` implementations do not perform this offset
> comparator check
> in the implementation of register method.
> * Moving this one level up from
> `SystemConsumer.register(SystemStreamPartition, offset)` API implementation
> to `SystemConsumers.register(SystemStreamPartition, offset)` API
> implementation removes unnecessary duplication and ensures functional
> correctness.
> There should be no functional breakages introduced by moving the
> offset-comparator check to `SystemConsumers` layer. Here's why. Metadata and
> I/O topics currently use SystemConsumer API to read data.
> * Checkpoint topic: This is a stream with one partition and is
> log-compacted. Samza-container reads from the beginning of this stream.
> There's no offset comparisons required for this topic.
> * ChangeLog topic: To read change-log topic-partition, SamzaContainer uses
> `StorageManagerUtil.getStartingOffset` to read from the change-log topic and
> the offset comparator check to choose the lowest offset is currently
> performed within it.
> * Coordinator topic: This is a stream with one partition and is
> log-compacted. SamzaContainer/ApplicationMaster is currently reads from the
> beginning of this stream to get the entire state.
> * I/O topics: `SamzaContainer` currently reads data from input and output
> topics through `SystemConsumers` API. Moving this offset comparator check
> should not affect the existing functionality.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
