you need to partition on something, the bolt grouping should do your partitioning and eliminate the need for locking. Remove synchronised as it will slow down processes.
Jacob Johansen On Tue, Nov 15, 2016 at 3:59 PM, Milind Vaidya <[email protected]> wrote: > Hi > > I am having a use case where few files in a directory are needed to be > processed by a certain bolt x written in Java. > > I am setting number of executers and tasks same which is > 1. Say I have 4 > executers and tasks. > > As I understand, these are essentially threads in the worker process. Now > I want to make sure that each of the executer / task should process a file > uniquely. How to ensure that ? > > Should I put a synchronised block inside execute method and make sure > processing is done in thread safe manner ? > > > This is actually to be done when a topology is launched. As the worker > starts, the corresponding bolts will scan a specific directory and process > the files previously generated but not processed. > > In normal scenario, another bolt will pass on file name and path to be > processed to this bolt. > > > > >
