[GitHub] samza pull request #98: SAMZA-1171: Rewrite config in ApplicationRunnerMain ...

2017-03-27 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/samza/pull/98 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] samza pull request #98: SAMZA-1171: Rewrite config in ApplicationRunnerMain ...

2017-03-27 Thread xinyuiscool
GitHub user xinyuiscool opened a pull request: https://github.com/apache/samza/pull/98 SAMZA-1171: Rewrite config in ApplicationRunnerMain when creating ApplicationRunner The config needs to be rewritten before passing down to the ApplicationRunner. This is a bug that was

[GitHub] samza pull request #97: SAMZA-1143 Include fs..impl.* subkeys to Yar...

2017-03-27 Thread fredji97
GitHub user fredji97 opened a pull request: https://github.com/apache/samza/pull/97 SAMZA-1143 Include fs..impl.* subkeys to YarnConfiguration used in YarnJobFactory and YarnClusterResourceManager SAMZA-1143 Include fs..impl.* subkeys, in addition to fs..impl, to YarnConfiguration

Re: Steps to Upgrading Samza (0.9 to 0.12)

2017-03-27 Thread Maksim Logvinenko
Hi colleagues, If I understand samza source code correctly, without migration code we would not lose offsets but lose TaskName-to-ChangelogPartition mapping. State restore for TaskStorage starts from ChangelogSystemStreamPartition beginning (if we don't use data locality, offsets to start from

[GitHub] samza pull request #92: SAMZA-1094: Remove MessageEnvelope from public opera...

2017-03-27 Thread prateekm
Github user prateekm closed the pull request at: https://github.com/apache/samza/pull/92 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] samza pull request #92: SAMZA-1094: Remove MessageEnvelope from public opera...

2017-03-27 Thread prateekm
GitHub user prateekm reopened a pull request: https://github.com/apache/samza/pull/92 SAMZA-1094: Remove MessageEnvelope from public operator APIs. SAMZA-1101: Delay the creation of SinkFunction for output streams. SAMZA-1159: Move StreamSpec from a public API to an internal class.

Re: Steps to Upgrading Samza (0.9 to 0.12)

2017-03-27 Thread Navina Ramesh (Apache)
@Jake: Yes. We removed the migration code (for 0.9 to 0.10) in the 0.11 release, I believe. @XiaoChuan: As per Jagadish's recommendation, if you have changelog backed stores, you should upgrade from 0.9.1 to 0.10.0 before upgrading to samza 0.12.0. I checked with LinkedIn's internal release

Re: Steps to Upgrading Samza (0.9 to 0.12)

2017-03-27 Thread Jagadish Venkatraman
Good observation Jake! The code for migration was removed in Samza 11. The migration would read change-log offsets from the checkpoint topic and write them to the coordinator stream. If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to 0.10.0 first. Otherwise, you will

Re: Steps to Upgrading Samza (0.9 to 0.12)

2017-03-27 Thread Jagadish Venkatraman
Good observation Jake! The code for migration was removed in Samza 11. The migration would read change-log offsets from the checkpoint topic and write them to the coordinator stream. If you're using change-logged stores, I'd recommend upgrading from 0.9.1 to 0.10.0 first. Otherwise, you will

Re: Steps to Upgrading Samza (0.9 to 0.12)

2017-03-27 Thread Jacob Maes
As I recall, samza 0.10 introduced the coordinator stream and there was code to do an automatic migration to use that feature. @navina, @yi, do you know if that migration code is still in samza 12? If not, then it's probably better to update from 0.9.1 to 0.10.0 and then to 0.12.0. I don't think