Hi Hai & All,
Yes, that is the expected behaviour, these configs if set by user code need to
match (job.id = app.id) and (job.name = app.name) and if for some reason if
they are not, Samza will fail with the same exception in
JobNodeConfigurationGenerator as mentioned in the thread.
Thanks
Sanil
________________________________
From: Hai Lu
Sent: Wednesday, April 10, 2019 9:57:24 PM
To: Xinyu Liu; Benenson, Mikhail; Ho, Tom; Prateek Maheshwari; Samarth Shetty;
Audo, Nicholas; Sanil Jain
Cc: dev@samza.apache.org; LeVeck, Matt; Deshpande, Omkar
Subject: Re: Seems, Samza 1.1 confused with jobId WAS Re: Beam with SamzaRunner
It is the same stack trace. I have hit the same issue with Beam Python.
Basically we need to set the exact same values for (job.id, app.id) and
(job.name, app.name). The new patch should fix it
Thanks,
Hai
From: Xinyu Liu <xi...@linkedin.com>
Date: Wednesday, April 10, 2019 at 9:24 PM
To: "Benenson, Mikhail" <mikhail_benen...@intuit.com>, "Ho, Tom"
<tom...@intuit.com>, Prateek Maheshwari <pmaheshw...@linkedin.com>, Samarth
Shetty <sshe...@linkedin.com>, "Audo, Nicholas" <nicholas_a...@intuit.com>, Hai
Lu <h...@linkedin.com>, Sanil Jain <snj...@linkedin.com>
Cc: "dev@samza.apache.org" <dev@samza.apache.org>, "LeVeck, Matt"
<matt_lev...@intuit.com>, "Deshpande, Omkar" <omkar_deshpa...@intuit.com>
Subject: Re: Seems, Samza 1.1 confused with jobId WAS Re: Beam with SamzaRunner
+Sanil for this too. The stack trace seems a bit different from what I saw in
Python. Could you please also take a look?
Thanks,
Xinyu
________________________________
From: Xinyu Liu
Sent: Wednesday, April 10, 2019 9:17 PM
To: Benenson, Mikhail; Ho, Tom; Prateek Maheshwari; Samarth Shetty; Audo,
Nicholas; Hai Lu
Cc: dev@samza.apache.org; LeVeck, Matt; Deshpande, Omkar
Subject: Re: Seems, Samza 1.1 confused with jobId WAS Re: Beam with SamzaRunner
Hi, Benenson,
Sorry about this! There seems to be some new logic on the Samza side (1.1.0)
for job.id and app.id. We are hit by the same issue in our Python flow today as
Hai found it. I am working on a fix to set both job.id and app.id in
SamzaRunner, which Hai will verify tomorrow to see whether that works. If it
works I will merge the fix to Beam so you can pull it.
Thanks,
Xinyu
________________________________
From: Benenson, Mikhail <mikhail_benen...@intuit.com>
Sent: Wednesday, April 10, 2019 8:36 PM
To: Xinyu Liu; Ho, Tom; Prateek Maheshwari; Samarth Shetty; Audo, Nicholas; Hai
Lu
Cc: dev@samza.apache.org; LeVeck, Matt; Deshpande, Omkar
Subject: Seems, Samza 1.1 confused with jobId WAS Re: Beam with SamzaRunner
Hi, Xinyu
I run latest Beam 2.13 w Samza runner 1.1.0.
In my samza.config I have:
app.name=spp-kafka-02
app.id=123
job.id=123
Execution failed, here is an extract from the log:
INFO main - 2019-04-10 20:27:27.696; -
org.apache.samza.execution.LocalJobPlanner - Execution Plan:
{"jobs":[{"jobName":"spp-kafka-02","jobId":"123", ...
...
org.apache.samza.SamzaException: Failed to start application:
app-spp-kafka-02-123
...
Caused by: org.apache.samza.SamzaException: Generated job.id = 123 from app.id
= 123 does not match user configured job.id = 1, please configure job.id same
as app.id
at
org.apache.samza.execution.JobNodeConfigurationGenerator.validateJobConfigs(JobNodeConfigurationGenerator.java:106)
at
org.apache.samza.execution.JobNodeConfigurationGenerator.mergeConfig(JobNodeConfigurationGenerator.java:74)
Seems, Smaza understand, that jobId is 123, but then something goes wrong.
The only way we can run an application is to set jobId=appId=1, and we cannot
change it.
Could you, please, look at the issue? Full log is attached:
------------------
Michael Benenson
From: Xinyu Liu <xi...@linkedin.com>
Date: Monday, March 25, 2019 at 3:34 PM
To: "Ho, Tom" <tom...@intuit.com>, "Deshpande, Omkar"
<omkar_deshpa...@intuit.com>
Cc: Prateek Maheshwari <pmaheshw...@linkedin.com>, Samarth Shetty
<sshe...@linkedin.com>, "Audo, Nicholas" <nicholas_a...@intuit.com>, "Benenson,
Mikhail" <mikhail_benen...@intuit.com>, Hai Lu <h...@linkedin.com>
Subject: Re: Beam with SamzaRunner
This email is from an external sender.
Hi, Tom,
I am still working on the merging all the PRs of the migration of Samza 1.x to
open source Beam. Due to some bugs found in portability runner (for python),
the merge was delayed for a few days. I am expecting to finish merging all the
changes at the end of this week.
There is a nightly snapshot build published by Beam that you should be able to
pick up:
https://repository.apache.org/content/repositories/snapshots/org/apache/beam/<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Fsnapshots%2Forg%2Fapache%2Fbeam%2F&data=02%7C01%7Csnjain%40linkedin.com%7Ce4d6b74f580644a77d3508d6be3a3063%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636905554460938359&sdata=l6whz3dOHZhYXoneHHU0NRv6LvU5hpsgvcEda9gmVUU%3D&reserved=0>.
I will ping you guys once all the changes are made to open source so you can
pick them up. Sorry about the delay.
Thanks,
Xinyu
________________________________
From: Ho, Tom <tom...@intuit.com>
Sent: Monday, March 25, 2019 3:22 PM
To: Xinyu Liu; Deshpande, Omkar
Cc: Prateek Maheshwari; Samarth Shetty; Audo, Nicholas; Benenson, Mikhail; Hai
Lu
Subject: Re: Beam with SamzaRunner
Hello Xinyu and Samarth,
Our streaming platform team is currently blocked on SSL issue related to Samza
0.14/Kafka client 0.9 version and would like to move forward by upgrade to
latest unreleased versions of Beam and Samza.
Is it possible for the team to get early access to the latest unreleased
versions of Beam 2.x/Samza 1.x?
Thank you very much for your help,
Tom
From: "Ho, Tom" <tom...@intuit.com>
Date: Monday, March 25, 2019 at 7:53 AM
To: Xinyu Liu <xi...@linkedin.com>, "Deshpande, Omkar"
<omkar_deshpa...@intuit.com>
Cc: Prateek Maheshwari <pmaheshw...@linkedin.com>, Samarth Shetty
<sshe...@linkedin.com>, "Audo, Nicholas" <nicholas_a...@intuit.com>, "Benenson,
Mikhail" <mikhail_benen...@intuit.com>, Hai Lu <h...@linkedin.com>
Subject: Re: Beam with SamzaRunner
Hello Xinyu,
The team ran into a secure connection issue with kafka client v0.9, which is
used by samza version 0.14. Matt Leveck sent an email (attached) to Samza
mailing list regarding this issue.
Is it possible for us to get early access to the binaries of tested Beam
2.12/Samza 1.0? We would like to take advantages of the feature sets, APIs,
and new SamzaRunner as we build out our streaming platform.
Thank you,
Tom
From: Xinyu Liu <xi...@linkedin.com>
Date: Tuesday, March 12, 2019 at 7:56 PM
To: "Ho, Tom" <tom...@intuit.com>, "Deshpande, Omkar"
<omkar_deshpa...@intuit.com>
Cc: Prateek Maheshwari <pmaheshw...@linkedin.com>, Samarth Shetty
<sshe...@linkedin.com>, "Audo, Nicholas" <nicholas_a...@intuit.com>, "Benenson,
Mikhail" <mikhail_benen...@intuit.com>, Hai Lu <h...@linkedin.com>
Subject: Re: Beam with SamzaRunner
This email is from an external sender.
Hi, Tom,
Beam 2.11.0 release has our portability changes for python. In LinkedIn we
already have python testing jobs running based on this framework, consuming
kakfa and doing some windowing and aggregations. So far Kafka input is using
our customized Beam python transform that translates to Java KafkaIO. We are in
the middle of figuring out how to make this available to you guys, and also add
docs and examples to open source to show how to run Beam python on SamzaRunner.
Samza 1.0 migration didn't make to the 2.11.0 Beam (sorry!). The migration is
done and we are still in the middle of testing it internally (it's a pretty big
change and we don't want you guys to hit any issues before us ?? ). All the
changes will be tested and committed to Beam open source this month. So please
bear with us for a couple of weeks.
Thanks,
Xinyu
________________________________
From: Ho, Tom <tom...@intuit.com>
Sent: Tuesday, March 12, 2019 3:04 PM
To: Xinyu Liu; Deshpande, Omkar
Cc: Prateek Maheshwari; Samarth Shetty; Audo, Nicholas; Benenson, Mikhail; Hai
Lu
Subject: Re: Beam with SamzaRunner
Hello Xinyu,
It’s been a while. How are you doing? I noticed that apache beam
2.11<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fbeam.apache.org%2Fblog%2F2019%2F03%2F05%2Fbeam-2.11.0.html&data=02%7C01%7Csnjain%40linkedin.com%7Ce4d6b74f580644a77d3508d6be3a3063%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636905554460938359&sdata=F7T19fG9gaFdilnY0cIbV9mnfUvLZdpuFaoQlNFGNK4%3D&reserved=0>
was released on March 5th, 2019. Does it include the Samza 1.0 runner and
portability changes mentioned in your previous email?
Thank you,
Tom
From: Xinyu Liu <xi...@linkedin.com>
Date: Friday, January 18, 2019 at 11:21 AM
To: "Ho, Tom" <tom...@intuit.com>, "Deshpande, Omkar"
<omkar_deshpa...@intuit.com>
Cc: Prateek Maheshwari <pmaheshw...@linkedin.com>, Samarth Shetty
<sshe...@linkedin.com>, "Audo, Nicholas" <nicholas_a...@intuit.com>, "Benenson,
Mikhail" <mikhail_benen...@intuit.com>, Hai Lu <h...@linkedin.com>
Subject: Re: Beam with SamzaRunner
This email is from an external sender.
+Hai.
Xinyu
________________________________
From: Xinyu Liu
Sent: Friday, January 18, 2019 9:59 AM
To: Ho, Tom; Deshpande, Omkar
Cc: Prateek Maheshwari; Samarth Shetty; Audo, Nicholas; Benenson, Mikhail
Subject: Re: Beam with SamzaRunner
Hey, folks,
How is everything going? Please keep us updated on any issues and problems you
find so we can help.
On our side, we made quite a few progress:
* A new repo is created for examples of running Beam on Samza
(https://github.com/apache/samza-beam-examples). The examples show both local,
standalone, and yarn deployment. For local and standalone, SamzaRunner should
be fully working. For yarn, the changes are already committed and will be in
the Beam 2.10.0 release (Jan - Feb).
* SamzaRunner portability changes are in review and will go out for the
2.11.0 Beam release (March-April). This will enable running the python sdk.
* We are also working on more tutorials and docs about how to run beam on
Samza. Any feedback and contributions from you guys will be highly appreciated.
We're also wondering whether it's helpful to have a bi-weekly quick sync-up
between us. We can set up video conference so we can work together more closely.
Thanks,
Xinyuu
________________________________
From: Ho, Tom <tom...@intuit.com>
Sent: Monday, January 7, 2019 11:35 AM
To: Xinyu Liu; Deshpande, Omkar
Cc: Prateek Maheshwari; Samarth Shetty; Audo, Nicholas; Benenson, Mikhail
Subject: Re: Beam with SamzaRunner
++Nick and Michael.
Thank you Xinyu.
Tom
From: Xinyu Liu <xi...@linkedin.com>
Date: Monday, January 7, 2019 at 11:32 AM
To: "Ho, Tom" <tom...@intuit.com>, "Deshpande, Omkar"
<omkar_deshpa...@intuit.com>
Cc: Prateek Maheshwari <pmaheshw...@linkedin.com>, Samarth Shetty
<sshe...@linkedin.com>
Subject: Beam with SamzaRunner
This email is from an external sender.
Hi, Tom and Omkar,
It's great to hear Intuit are using Beam API on top of Samza for your streaming
use cases. We will be very happy to support you guys as needed. In the past few
days it seems Omkar has sent emails to dev@samza for questions, but due to some
weird recipient setup on my gmail, I am not able to see the emails and respond
to it. While I am figuring out the problem, could you please use a gmail
account to send the questions? That way I can get to it as soon as possible.
Meanwhile, I am working on the examples to show running Beam on Samza in local,
standalone and yarn deployment. Once it's complete, I will share the github
repo with you guys.
Thanks,
Xinyu
