Hey Florian, Can you explain a bit more how having access to the offset storage from the connector helps in your use case? I guess you are planning to use offsets to be able to tell when a task has finished a file?
Thanks, Jason On Fri, Feb 17, 2017 at 4:45 AM, Florian Hussonnois <fhussonn...@gmail.com> wrote: > Hi Kafka Team, > > I'm developping a connector which need to monitor the progress of its tasks > in order to be able to request a tasks reconfiguration in some situations. > > Our connector is pretty simple. It's used to stream a thousands of files > into Kafka. The connector scans directories then schedules each task with a > set of assigned files. > When tasks are no longer required or new files are detected the connector > requests a reconfiguration. > > In addition, files are store into a shared storage which is accessible from > each connect worker. In that way, we can distribute file streaming. > > For that prupose, it would be very convenient to have access to an > offsetStorageReader instance from either the Connector class or the > ConnectorContext class. > > I found a similar question: > https://www.mail-archive.com/dev@kafka.apache.org/msg50579.html > > Do you think this improvement could be considered ? I can contribute to it. > > Thanks, > > -- > Florian HUSSONNOIS >
