Hi, Nicholas, Thank your so much for pointing out the perf degradation. I synced up w/ Jiangjie Qin from our Kafka team and confirmed that we are running 0.10 Kafka brokers w/o the timestamp message format in 0.10. I also took a look at SAMZA-855 patch and it is far from a working state yet. Given that there are many other ready-to-ship features in Samza-0.11, I would suggest moving SAMZA-855 out of the scope of 0.11.
P.S. from 0.10.1, we are trying to have a regular time-driven release as well. As you see, 0.11 is out soon (planning in Sept), and we are looking forward to 2-3 month release cadence from now on. Hence, SAMZA-855 would make it to 0.12. Hope that won't delay you too much. Thanks! -Yi On Wed, Aug 24, 2016 at 12:10 PM, Nicolas Maquet <nmaq...@gmail.com> wrote: > Hi, > > We are looking at upgrading to Kafka 0.10.0 in part for the new message > format which includes a timestamp field. Kafka 0.10.0 is backwards > compatible with 0.8.x clients but we are concerned about the performance > impact, see > http://kafka.apache.org/documentation.html#upgrade_10_performance_impact. > > Cheers, > > Nicolas > > On 25 Aug 2016 6:20 a.m., "Yi Pan" <nickpa...@gmail.com> wrote: > > > Hi, Nicolas, > > > > Could you explain to me why Samza is blocking you from upgrading your > Kafka > > brokers to 0.10? At LinkedIn, we are running Samza 0.10 w/ Kafka 0.10 > > brokers. This is a valid combination since Kafka 0.10 brokers should be > > backward compatible w/ 0.8.2 clients (which is the version Samza uses). I > > would recommend that you try to upgrade the Kafka brokers w/ Samza 0.10. > If > > there is any issues you see, we will help to investigate and fix. > > > > Meanwhile, we are planning to schedule Samza releases more regularly and > > frequent in terms of time. Hence, although SAMZA-855 is not planned in > > 0.11, it will be included in the immediate version after that. > > > > Let me know if you have any further concerns. > > > > On Mon, Aug 22, 2016 at 2:48 PM, Nicolas Maquet <nmaq...@gmail.com> > wrote: > > > > > Hi all, > > > > > > Would it be possible to consider including the following in the 0.11.0 > > > target release? > > > > > > SAMZA-855: Upgrade Samza's Kafka client version to 0.10.0.0 > > > > > > We are looking at upgrading our Kafka cluster to 0.10 and Samza is > > > currently preventing us from doing so. > > > > > > Regards, > > > > > > Nicolas Maquet > > > > > > > > > > > > 2016-08-23 9:06 GMT+12:00 xinyu liu <xinyuliu...@gmail.com>: > > > > > > > Hi, All, > > > > > > > > There have been quite a lot of new features added to master since > 0.10 > > > > release to warrant a new major release. At LinkedIn, we've done > > > functional > > > > and performance testing against master in the past weeks, and > deployed > > > jobs > > > > with the latest build in production. We will continue to test for > > > stability > > > > in the next few weeks. > > > > > > > > Here are the JIRAs of main features that will be included in this > > release > > > > (sorted in chronological order): > > > > > > > > SAMZA-876: Add AvroDataFileHdfsWriter > > > > SAMZA-849: Split-deployment of samza framework > > > > SAMZA-905: Host Affinity - Continuous offset > > > > SAMZA-906: Host Affinity - Minimize task reassignment when container > > > count > > > > changes > > > > SAMZA-924: Disk Quotas: Add disk space monitoring > > > > SAMZA-956: Disk Quotas: Add throttler and disk quota enforcement > > > > SAMZA-680: Inverting JobCoordinator and SamzaAppMaster logic > > > > SAMZA-863: Multithreading support in Samza > > > > > > > > And the JIRAs that I think we should try and get done before 0.11.0 > is > > > > released: > > > > > > > > SAMZA-980: Automatically restart job containers on change in upstream > > > > partitioning > > > > SAMZA-988: Update failure testing job configs with job coordinator > > > > properties > > > > SAMZA-997: Add documentation about disk space monitoring > > > > SAMZA-977: User doc for samza multithreading > > > > SAMZA-998: Documentation updates for refactored JC > > > > SAMZA-1000: Hello-samza latest is broken after Samza 10.1 release > > > > > > > > > > > > Here's what I purpose: > > > > > > > > 1. Cut an 0.11.0 release branch. > > > > 2. Work on getting as many of the pending JIRAs done as possible. > > > > 2. Target a release vote on the week of Sep 5. > > > > > > > > Thoughts? > > > > > > > > Xinyu > > > > > > > > > >
