Hi, XiaoChuan, Great! So I assume that you were able to upgrade to Samza 0.13.1 successfully? If you need any further help, feel free to send us emails!
Thanks! -Yi On Thu, Sep 14, 2017 at 8:11 AM, XiaoChuan Yu <xiaochuan...@kik.com> wrote: > Hi Yi, > > Thanks for all the suggestions. > Our team eventually decided to move off CDH Kafka(and YARN) due to some > issues not relevant to Samza. > For what its worth, I did test the a couple basic Hello Samza (Samza > 0.13.1) jobs against a single node CDH Kafka version > 0.10.2.0+kafka2.2.0+110 with success. > The testing was done on AWS EC2 instances running RedHat 7. > We also found out that running against CDH YARN required various dependency > overrides such as the ones in Hello Samza's "cdh5" build profile > <https://github.com/apache/samza-hello-samza/blob/master/pom.xml#L282-L297 > >. > > Thanks, > Xiaochuan Yu > > On Fri, Sep 1, 2017 at 1:18 PM Yi Pan <nickpa...@gmail.com> wrote: > > > Hi, XiaoChuan, > > > > Just confirmed with Jiangjie Qin (our in-house Kafka PMC), Kafka client > > 0.10.1.1 does not work with broker 0.10.0 due to new protocol version > > introduced in 0.10.1.1. Hence, if you try to run Samza 0.13.1 against > Kafka > > broker 0.10.0 version, it won't work. The best option for that seems to > be > > swapping out the runtime Kafka client 0.10.1.1 version in Samza 0.13.1 to > > Kafka client 0.10.0.1 (which was used in Samza 0.12). Since there is no > > incompatible API calls to Kafka client lib between Samza 0.12.0 and Samza > > 0.13.1, that should work and is worth trying out. > > > > Let us know if we can be further assistance here. > > > > Thanks! > > > > -Yi > > > > On Thu, Aug 31, 2017 at 4:14 PM, Yi Pan <nickpa...@gmail.com> wrote: > > > > > Hi, Xiaochuan, > > > > > > I am relaying your question to our in-house Kafka expert to confirm. > But > > > as far as I can tell from the description, the main breaking changes > are > > in > > > the KStreams API. Samza only uses the standard Kafka consumer/producer > > APIs > > > now and should not be affected. > > > > > > I would recommend to give it a try in your question 1. > > > > > > As for your question 2, are you referring to a) binding a Kafka > 0.10.0.0 > > > client library with Samza 0.13.1? Or b) running Samza 0.13.1 against > > Kafka > > > broker version 0.10.0.0? > > > > > > If you are referring to a), from the commit history, there is no source > > > code level changes when we upgrade the dependency from Kafka 0.10.0.1 > to > > > 0.10.1.1 in Samza 0.13.1. If you want to make sure that works, you can > > try > > > to downgrade the Kafka version in your build environment to 0.10.0.0 > and > > > build your app to see whether it breaks the build. If the build passes, > > the > > > client should work w/ Samza 0.13.1, unless Kafka client has an API that > > > changes runtime behavior between 0.10.0.1 and 0.10.1.1. > > > > > > If you are referring to b), it is not guaranteed that the higher > version > > > of Kafka client library will work with an older version of Kafka > broker. > > > However, Samza has been very conservative in the usage of new features > in > > > Kafka client library. Hence, you would have a higher chance of success > to > > > run Samza 0.13.1 against Kafka 0.10.0 brokers. > > > > > > Unfortunately, we have not test all the above combinations internally. > I > > > would wait for the confirmation from our Kafka expert before I can give > > you > > > a definite answer. > > > > > > Thanks! > > > > > > -Yi > > > > > >
