[
https://issues.apache.org/jira/browse/CRUNCH-611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15377332#comment-15377332
]
Micah Whitacre commented on CRUNCH-611:
---------------------------------------
Optional means it is needed to compile but that the dependency is not pushed on
to consumers transitively. If someone wants to use our Offset management they
will be responsible for including those dependencies in their assemblies.
https://maven.apache.org/guides/introduction/introduction-to-optional-and-excludes-dependencies.html
> Simplified Kafka Offset Management in HDFS
> ------------------------------------------
>
> Key: CRUNCH-611
> URL: https://issues.apache.org/jira/browse/CRUNCH-611
> Project: Crunch
> Issue Type: Improvement
> Components: IO
> Reporter: Micah Whitacre
> Assignee: Micah Whitacre
> Attachments: CRUNCH-611.patch
>
>
> With the KafkaSource the responsibility of offset management is the burden of
> the consumer. With some simple APIs it is actually trivial to support
> read/storing these offsets in an HDFS directory as checkpoints for the source.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
