"If I understand it correctly the only viable solution at the moment is to create a new stream for the rules messages with as many partitions as the data stream and write each rules update message to all partitions of the new rules stream."
If the data is constantly changing, yes, AFAIK, this is the only viable solution before we provide "shared store". Cheers, Fang, Yan yanfang...@gmail.com On Tue, May 5, 2015 at 12:34 PM, Ueli Gallizzi <ueli.galli...@gmail.com> wrote: > Hi Yan, > > Thanks for your quick response. > > After I read the discussion on SAMZA-353 I think the best solution for my > use case is a "shared state" store among StreamTasks described in > SAMZA-402. To give you some background I have a stream with rules which are > constantly changing and a data stream on which I apply the rules. The rules > set is very small if you compare it with the data stream. > > If I understand it correctly the only viable solution at the moment is to > create a new stream for the rules messages with as many partitions as the > data stream and write each rules update message to all partitions of the > new rules stream. > > Cheers, > - ueli > > On Tue, May 5, 2015 at 12:06 PM, Yan Fang <yanfang...@gmail.com> wrote: > > > Hi Ueli, > > > > This feature currently is not supported by Samza. There was some > > discussions in the JIRA - SAMZA-353 > > <https://issues.apache.org/jira/browse/SAMZA-353>. > > > > But there are some workaround for this, depends on what you want to > > achieve. If you can specify what your requirement is, we can help think > of > > the solution. > > > > In another thread > > < > > > http://mail-archives.apache.org/mod_mbox/samza-dev/201504.mbox/%3c552410e2.7000...@tivo.com%3E > > >, > > Tommy Becker has similar requirement and he maybe helpful as well. > > > > Cheers, > > Fang, Yan > > yanfang...@gmail.com > > > > On Tue, May 5, 2015 at 7:42 AM, Ueli Gallizzi <ueli.galli...@gmail.com> > > wrote: > > > > > Hi, > > > > > > Is it possible that multiple tasks read from the same input stream > > > partition? > > > > > > example: > > > task 0 stream A partition 0, stream B partition 0 > > > task 1 stream A partition 1, stream B partition 0 > > > task 2 stream A partition 3, stream B partition 0 > > > > > > In this example all messages in stream B partition 0 would be processed > > by > > > all 3 tasks. > > > > > > Cheers, > > > - ueli > > > > > > > >
