Re: Steps to Upgrading Samza (0.9 to 0.12)

FYI, I created https://issues.apache.org/jira/browse/SAMZA-1177 to track this. We will ensure that we have explicit upgrade documentation for upcoming releases. On Thu, Mar 30, 2017 at 2:21 PM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Hi Thomas and Maxim, > > Thank you for

Re: Steps to Upgrading Samza (0.9 to 0.12)

Hi Thomas and Maxim, Thank you for bringing up this concern and pointing out the gap in the documentation. We will ensure that this is documented in a separate upgrade page going forward. Best, Jagadish On Thu, Mar 30, 2017 at 12:02 PM, Navina Ramesh < nram...@linkedin.com.invalid> wrote: > Hi

Re: Steps to Upgrading Samza (0.9 to 0.12)

Hi everyone, Apologize for re-chiming in late on this issue. > I'm not sure I agree with the policy (removing migration code and wanting people to upgrade seem at odds to me), but minimally I think we should not assume people are upgrading to each new Samza version. I agree that we should not

Re: Steps to Upgrading Samza (0.9 to 0.12)

Is there some sort of document on how to upgrade Samza through various versions like the page here for Kafka: https://kafka.apache.org/documentation/#upgrade ? Having something like this would be ideal. On Thu, Mar 30, 2017 at 1:51 PM Thomas Becker wrote: > Thanks for the

Re: Steps to Upgrading Samza (0.9 to 0.12)

Thanks for the reply Yi, and I apologize if I came off a bit snarky. I'm not sure I agree with the policy (removing migration code and wanting people to upgrade seem at odds to me), but minimally I think we should not assume people are upgrading to each new Samza version. We have done so when

Re: Steps to Upgrading Samza (0.9 to 0.12)

Hi, Thomas, Sorry to hear that you were hit by the removal of migration in Samza 0.11. The reason we removed it is following a deprecate-removal policy in two versions. We are not aware that people still using 0.9 after we released 0.11 and were not expecting a direct upgrade from 0.9 to 0.12.

Re: Steps to Upgrading Samza (0.9 to 0.12)

Yes, we were burned by this. The changelog mapping will be regenerated instead of migrated and the result will completely hose the job (because the mapping was not generated deterministically in previous versions of Samza). I don't understand why the migration code was removed but it was, and to

Re: Steps to Upgrading Samza (0.9 to 0.12)

Hi colleagues, If I understand samza source code correctly, without migration code we would not lose offsets but lose TaskName-to-ChangelogPartition mapping. State restore for TaskStorage starts from ChangelogSystemStreamPartition beginning (if we don't use data locality, offsets to start from

Re: Steps to Upgrading Samza (0.9 to 0.12)

@Jake: Yes. We removed the migration code (for 0.9 to 0.10) in the 0.11 release, I believe. @XiaoChuan: As per Jagadish's recommendation, if you have changelog backed stores, you should upgrade from 0.9.1 to 0.10.0 before upgrading to samza 0.12.0. I checked with LinkedIn's internal release

Re: Steps to Upgrading Samza (0.9 to 0.12)

Good observation Jake! The code for migration was removed in Samza 11. The migration would read change-log offsets from the checkpoint topic and write them to the coordinator stream. If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to 0.10.0 first. Otherwise, you will

Re: Steps to Upgrading Samza (0.9 to 0.12)

Good observation Jake! The code for migration was removed in Samza 11. The migration would read change-log offsets from the checkpoint topic and write them to the coordinator stream. If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to 0.10.0 first. Otherwise, you will

Re: Steps to Upgrading Samza (0.9 to 0.12)

As I recall, samza 0.10 introduced the coordinator stream and there was code to do an automatic migration to use that feature. @navina, @yi, do you know if that migration code is still in samza 12? If not, then it's probably better to update from 0.9.1 to 0.10.0 and then to 0.12.0. I don't think

Re: Steps to Upgrading Samza (0.9 to 0.12)

Hi Xiaochuan, >> Do I need to upgrade Kafka and/or YARN? *Yarn version:* - Samza 0.12 supports Yarn 2.6.1 and 2.7.1. - If you already have 2.6.0 installed (as you have said), I believe you will be fine. (but I'm not sure) *Kafka version: * - Samza 0.12 upgraded the version of

Steps to Upgrading Samza (0.9 to 0.12)

Hi, What are the general steps for upgrading Samza from 0.9 to 0.12? Do I need to upgrade Kafka and/or YARN? I don't know how Samza was setup initially but we currently have the following setup: Samza version: 0.9.1 YARN version: Hadoop 2.6.0-cdh5.4.8 Kafka version: 0.9.0.1 I think