Re: [VOTE] SEP-24: Cluster-based Job Coordinator Dependency Isolation

[Yi Pan]

OK. If I understand correctly, your answer is the following:
yarn.resources.* configuration variables are used by YARN localizer to make
API and infrastructure classpath available, together with the application's
own classpath, which is also determined by the YARN localizer.
The question here is: how do we let the container JVM know the
API/infrastructure classpaths when launching the container processes? If
the API and infrastructure classpaths (i.e. installation path determined by
the localizer) are customizable, then we would need to tell the container
JVM those API/infra classpaths via some configuration variables as well,
right? Hence, those configuration variable names need to be understood by
the Samza application's code (which is run within the container) as well.
If not, what's the mechanism that we will use to let the container JVM
process to know where the YARN localizer has put API/infra classpaths?

Thanks!

-Yi



On Wed, Mar 11, 2020 at 8:09 PM Cameron Lee  wrote:

> The configuration variables are only used by the YARN localizer. The Samza
> application will look for the framework resources in certain places in the
> application's working directory when it needs to access them. My aim is to
> do something similar to how "yarn.package.path" works. In other execution
> environments, it is my understanding that "yarn.package.path" would get
> replaced by a different environment-specific configuration key/value.
> I agree that we should not use "yarn.resources.*" if the configurations are
> not YARN-specific. Do you think that these resource localization configs
> are generalizable to arbitrary environments? If so, does that mean
> "yarn.package.path" is also generalizable? For example, what if some
> execution environment does not use URLs to specify resource locations
> (although maybe this isn't a reasonable concern to worry about?)?
>
> Thanks,
> Cameron
>
> On Wed, Mar 11, 2020 at 4:43 PM Yi Pan  wrote:
>
> > Hi, Cameron,
> >
> > Thanks for the quick responses! Appreciate it.
> >
> > I am still having a concern on a): are those configuration variables used
> > by YARN localizer or by Samza applications? If those are used only by the
> > YARN localizer, I agree that we should keep those as yarn specific.
> > Otherwise, I think that would still be better to name those as
> > cluster.based.resources.*. The reason being: Samza applications are
> > supposed to be able to run on different execution environments. Ideally,
> > when we are deploying the same Samza application on YARN vs Mesos or
> > managed K8s clusters, we should only need to change the configure values,
> > not the configuration variable names and values. Does it make sense?
> > Otherwise, we can schedule a conf call to clarify that.
> >
> > Thanks!
> >
> > -Yi
> >
> > On Tue, Mar 10, 2020 at 3:25 PM Cameron Lee 
> > wrote:
> >
> > > a) The "yarn.resources.*" configs are for localizing the necessary
> > > resources into the working directory for the process. I felt that the
> > > specific configuration format to specify these resources might be
> > > YARN-specific (e.g. YARN has type and visibility configs for each of
> its
> > > resources), so a generic format might not apply. In a non-YARN case,
> the
> > > localization configs would need to be specified according to the
> > technology
> > > being used.
> > > b) It is correct that the Avro version will need to be compatible with
> > the
> > > version that is used by the infrastructure, if infrastructure needs to
> > use
> > > Avro and pass the Avro object to the application. This is the case with
> > any
> > > serde technology that needs to be used. For the job coordinator, it is
> > not
> > > much of a concern anyways, since it is not doing serde of Avro
> messages.
> > > This may be more of a concern for general split deployment, which will
> > > impact the processing containers, and will be a separate SEP.
> > > c) It should work to leave infrastructure serdes in the infrastructure
> > > classpath. The infrastructure serdes just see generic types (which are
> > > java.lang.Object at runtime) for the messages, and they don't do
> anything
> > > with the concrete types, so in the infrastructure classes, the messages
> > get
> > > passed around as Object, but their concrete classes can still be loaded
> > > from the application. As with (b), this is more of a concern for
> general
> > > split deployment, since the job coordinator doesn't do message serde. I
> > > have run some tests regarding this classloading pattern, but we will do
> > > further verification for general split deployment.
> > > d) Yes, you are correct. Good catch. It should be "described above at
> > > Application classloader".
> > >
> > > Thanks for all of your questions. I will clarify some details in the
> doc
> > > regarding your questions.
> > >
> > > Cameron
> > >
> > > On Mon, Mar 9, 2020 at 12:07 PM Yi Pan  wrote:
> > >
> > > > Hi, Cameron,
> > > >
> > > > Sorry to chime in late. Overall, looks great! 

Re: [VOTE] SEP-24: Cluster-based Job Coordinator Dependency Isolation

[Cameron Lee]

The configuration variables are only used by the YARN localizer. The Samza
application will look for the framework resources in certain places in the
application's working directory when it needs to access them. My aim is to
do something similar to how "yarn.package.path" works. In other execution
environments, it is my understanding that "yarn.package.path" would get
replaced by a different environment-specific configuration key/value.
I agree that we should not use "yarn.resources.*" if the configurations are
not YARN-specific. Do you think that these resource localization configs
are generalizable to arbitrary environments? If so, does that mean
"yarn.package.path" is also generalizable? For example, what if some
execution environment does not use URLs to specify resource locations
(although maybe this isn't a reasonable concern to worry about?)?

Thanks,
Cameron

On Wed, Mar 11, 2020 at 4:43 PM Yi Pan  wrote:

> Hi, Cameron,
>
> Thanks for the quick responses! Appreciate it.
>
> I am still having a concern on a): are those configuration variables used
> by YARN localizer or by Samza applications? If those are used only by the
> YARN localizer, I agree that we should keep those as yarn specific.
> Otherwise, I think that would still be better to name those as
> cluster.based.resources.*. The reason being: Samza applications are
> supposed to be able to run on different execution environments. Ideally,
> when we are deploying the same Samza application on YARN vs Mesos or
> managed K8s clusters, we should only need to change the configure values,
> not the configuration variable names and values. Does it make sense?
> Otherwise, we can schedule a conf call to clarify that.
>
> Thanks!
>
> -Yi
>
> On Tue, Mar 10, 2020 at 3:25 PM Cameron Lee 
> wrote:
>
> > a) The "yarn.resources.*" configs are for localizing the necessary
> > resources into the working directory for the process. I felt that the
> > specific configuration format to specify these resources might be
> > YARN-specific (e.g. YARN has type and visibility configs for each of its
> > resources), so a generic format might not apply. In a non-YARN case, the
> > localization configs would need to be specified according to the
> technology
> > being used.
> > b) It is correct that the Avro version will need to be compatible with
> the
> > version that is used by the infrastructure, if infrastructure needs to
> use
> > Avro and pass the Avro object to the application. This is the case with
> any
> > serde technology that needs to be used. For the job coordinator, it is
> not
> > much of a concern anyways, since it is not doing serde of Avro messages.
> > This may be more of a concern for general split deployment, which will
> > impact the processing containers, and will be a separate SEP.
> > c) It should work to leave infrastructure serdes in the infrastructure
> > classpath. The infrastructure serdes just see generic types (which are
> > java.lang.Object at runtime) for the messages, and they don't do anything
> > with the concrete types, so in the infrastructure classes, the messages
> get
> > passed around as Object, but their concrete classes can still be loaded
> > from the application. As with (b), this is more of a concern for general
> > split deployment, since the job coordinator doesn't do message serde. I
> > have run some tests regarding this classloading pattern, but we will do
> > further verification for general split deployment.
> > d) Yes, you are correct. Good catch. It should be "described above at
> > Application classloader".
> >
> > Thanks for all of your questions. I will clarify some details in the doc
> > regarding your questions.
> >
> > Cameron
> >
> > On Mon, Mar 9, 2020 at 12:07 PM Yi Pan  wrote:
> >
> > > Hi, Cameron,
> > >
> > > Sorry to chime in late. Overall, looks great! I do have a few
> > > suggestions/questions before I can cast my vote here:
> > > a) for the configuration variable names, why are we limiting ourselves
> to
> > > yarn.resource.*? We have changed some of the configuration variables
> from
> > > yarn specific to non-yarn specific. I would love to keep that
> consistent
> > > (i.e. gradually moving all our yarn-specific configuration variables to
> > > non-yarn-specifc names)
> > > b) for the avro case as referred to in the delegation case in the
> > > Infrastructure classloader, if we delegate the object deserialization
> > class
> > > to the application classloader, would it be possible that the
> application
> > > provides an non-compatible version of avro class than the ones used
> > within
> > > the "infrastructure plugins" and hence causing runtime exception in the
> > > infrastructure plugin? Or is the solution being: do not directly use
> > serde
> > > classes in the infrastructure code?
> > > c) following the description of infrastructure classloader flow, where
> > > should we expect the serde classes? In the application classpath, I
> > guess?
> > > So, does that mean that we should 

[RESULT] [VOTE] Apache Samza 1.4.0 RC1

[Cameron Lee]

The vote has passed. Thanks!
+1 (binding) x 3: Yi, Xinyu, Bharath

Cameron

On Wed, Mar 11, 2020 at 3:28 PM Bharath Kumara Subramanian <
codin.mart...@gmail.com> wrote:

> +1(binding).
>
> Check-all.sh and the integration tests passed.
>
> Thanks,
> Bharath
>
> On Tue, Mar 10, 2020 at 3:35 PM Xinyu Liu  wrote:
>
> > +1 (binding).
> >
> > Run check-all.sh and integration tests for both yarn and standalone. All
> > passed.
> >
> > Thanks,
> > Xinyu
> >
> >
> > On Fri, Mar 6, 2020 at 6:46 PM Yi Pan  wrote:
> >
> > > Have downloaded the files, build with check-all.sh, and ran both YARN
> and
> > > standalone integration tests. All passed.
> > >
> > > +1 (binding).
> > >
> > > Thanks!
> > >
> > > -Yi
> > >
> > > On Tue, Mar 3, 2020 at 3:03 PM Cameron Lee 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > This is a call for a vote on a release of Apache Samza 1.4.0. Thanks
> to
> > > > everyone who has contributed to this release.
> > > >
> > > > The release candidate can be downloaded from here:
> > > > https://home.apache.org/~cameronlee/samza-1.4.0-rc1/
> > > >
> > > > The release candidate is signed with pgp key 0x54CB3CE3, which can be
> > > found
> > > > here:
> > > >
> > > >
> > >
> >
> https://keyserver.ubuntu.com/pks/lookup?search=0x54CB3CE3=on=index
> > > > or to directly see the public key here:
> > > >
> > > >
> > >
> >
> https://keyserver.ubuntu.com/pks/lookup?op=get=0x71b0145290ecdbfa5caea6dbd786a7ba54cb3ce3
> > > >
> > > > The git tag is release-1.4.0-rc1, signed by the same pgp key above:
> > > >
> > > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=samza.git;a=commit;h=5327fafb8502b126482ec0c4efc8d1aa9b96ba44
> > > >
> > > > Test binaries have been published to Maven's staging repository, and
> > are
> > > > available here:
> > > >
> https://repository.apache.org/content/repositories/orgapachesamza-1077
> > > >
> > > > The vote will be open for 72 hours (until Friday, March 6, 2020 at
> 3pm
> > > > PST).
> > > >
> > > > Please download the release candidate, check the hashes/signature,
> > build
> > > it
> > > > and test it, and then please vote:
> > > > [ ] +1 approve
> > > > [ ] +0 no opinion
> > > > [ ] -1 disapprove (and reason why)
> > > >
> > > > I ran check-all.sh and integration tests.
> > > >
> > > > +1 (non-binding) from my side.
> > > >
> > > > Thank you,
> > > > Cameron
> > > >
> > >
> >
>

Re: Got Error Produce Respons with Correlation Id.

[Yi Pan]

Hi, Jeremiah,

Sorry to reply late. This WARN message indicates that producer failed to
flush to checkpoint topic and would retry. Do you see the Samza job hanging
after that? Is the checkpointed offset metrics incrementing in this case?
Not clear on your claiming: "logs stop at that point". No logs are written
after the WARN lines? What's your Samza configuration? Is the Samza
container still running after you see those WARN logs?

Thanks!

-Yi

On Wed, Mar 11, 2020 at 2:39 PM Jeremiah Adams
 wrote:

> Can anyone take a look at the message below? We are trying to gauge our
> risk before moving forward.
>
>
> Jeremiah Adams
> Software Engineer
> www.helixeducation.com
> Blog | Twitter | Facebook | LinkedIn
>
> 
> From: Jeremiah Adams 
> Sent: Wednesday, March 4, 2020 2:28 PM
> To: dev@samza.apache.org
> Subject: Got Error Produce Response iwth Correlation Id.
>
> Hello devs,
>
>
> I've got a warning showing up in the logs while testing our new Confluent
> Cloud config.  Can anyone tell me how concerned I should be about this
> warning? Is there a setting to control timeouts?
>
>
> Also, logs stop at that point, so I can't tell if the "metatdata update"
> was complete.
>
>
>
> 2020-03-04 21:17:51 Sender [WARN] [Producer
> clientId=kafka_producer-application_submission-1] Got error produce
> response with correlation id 144 on topic-partition
> __samza_checkpoint_ver_1_for_application-submission_1-0, retrying
> (2147483646 attempts left). Error: NETWORK_EXCEPTION
> 2020-03-04 21:17:51 Sender [WARN] [Producer
> clientId=kafka_producer-application_submission-1] Received invalid metadata
> error in produce request on partition
> __samza_checkpoint_ver_1_for_application-submission_1-0 due to
> org.apache.kafka.common.errors.NetworkException: The server disconnected
> before a response was received.. Going to request metadata update now
>
>
> Jeremiah Adams
> Software Engineer
>
> https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~
> <
> https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df8YVt3SYmcjmLWdKMWzAAINWlUUA33ebGI7pSoTl9cg1g~~
> >
> Blog<
> https://url.emailprotection.link/?basKr9vk92a8vVw0XMnK5bmaSKuBc0AuEZ7YasYc7Df-lAcqG1fqHPpNw-wd9z7HtUJeCG5_8UjCf2mHtn6C_zQ~~>
> | Twitter<
> https://url.emailprotection.link/?bVO2q0UXR235wN_yOnM0FjqITPdBYMD3reLGNddq-zPV5ChMQK9JwV4Be-QnrbRoXpJl8IcknAqKzYtA3RABKww~~>
> | Facebook<
> https://url.emailprotection.link/?bUU7m4NfMS_EWGtH1yojBHX9sWZ6uxVdT1eQUkmU5vWY01WFZiS2KJ-c9iLIncdHB7Uw1lRYCprEEpPPQCdiK6Q~~>
> | LinkedIn<
> https://url.emailprotection.link/?b0ZQfJ1pZYnASyoShs9MJI46-r1lxPhA-JS5VSkR7so-DFP0_HxbOo2LsajGOaoYXxb1ZCOMAu7hZscPCnIKWpXz0cpgQ386SnNHjPcwsu4z90mzBkuwoZc6YxOCzMGA0
> >
>

Re: [VOTE] SEP-24: Cluster-based Job Coordinator Dependency Isolation

[Yi Pan]

Hi, Cameron,

Thanks for the quick responses! Appreciate it.

I am still having a concern on a): are those configuration variables used
by YARN localizer or by Samza applications? If those are used only by the
YARN localizer, I agree that we should keep those as yarn specific.
Otherwise, I think that would still be better to name those as
cluster.based.resources.*. The reason being: Samza applications are
supposed to be able to run on different execution environments. Ideally,
when we are deploying the same Samza application on YARN vs Mesos or
managed K8s clusters, we should only need to change the configure values,
not the configuration variable names and values. Does it make sense?
Otherwise, we can schedule a conf call to clarify that.

Thanks!

-Yi

On Tue, Mar 10, 2020 at 3:25 PM Cameron Lee  wrote:

> a) The "yarn.resources.*" configs are for localizing the necessary
> resources into the working directory for the process. I felt that the
> specific configuration format to specify these resources might be
> YARN-specific (e.g. YARN has type and visibility configs for each of its
> resources), so a generic format might not apply. In a non-YARN case, the
> localization configs would need to be specified according to the technology
> being used.
> b) It is correct that the Avro version will need to be compatible with the
> version that is used by the infrastructure, if infrastructure needs to use
> Avro and pass the Avro object to the application. This is the case with any
> serde technology that needs to be used. For the job coordinator, it is not
> much of a concern anyways, since it is not doing serde of Avro messages.
> This may be more of a concern for general split deployment, which will
> impact the processing containers, and will be a separate SEP.
> c) It should work to leave infrastructure serdes in the infrastructure
> classpath. The infrastructure serdes just see generic types (which are
> java.lang.Object at runtime) for the messages, and they don't do anything
> with the concrete types, so in the infrastructure classes, the messages get
> passed around as Object, but their concrete classes can still be loaded
> from the application. As with (b), this is more of a concern for general
> split deployment, since the job coordinator doesn't do message serde. I
> have run some tests regarding this classloading pattern, but we will do
> further verification for general split deployment.
> d) Yes, you are correct. Good catch. It should be "described above at
> Application classloader".
>
> Thanks for all of your questions. I will clarify some details in the doc
> regarding your questions.
>
> Cameron
>
> On Mon, Mar 9, 2020 at 12:07 PM Yi Pan  wrote:
>
> > Hi, Cameron,
> >
> > Sorry to chime in late. Overall, looks great! I do have a few
> > suggestions/questions before I can cast my vote here:
> > a) for the configuration variable names, why are we limiting ourselves to
> > yarn.resource.*? We have changed some of the configuration variables from
> > yarn specific to non-yarn specific. I would love to keep that consistent
> > (i.e. gradually moving all our yarn-specific configuration variables to
> > non-yarn-specifc names)
> > b) for the avro case as referred to in the delegation case in the
> > Infrastructure classloader, if we delegate the object deserialization
> class
> > to the application classloader, would it be possible that the application
> > provides an non-compatible version of avro class than the ones used
> within
> > the "infrastructure plugins" and hence causing runtime exception in the
> > infrastructure plugin? Or is the solution being: do not directly use
> serde
> > classes in the infrastructure code?
> > c) following the description of infrastructure classloader flow, where
> > should we expect the serde classes? In the application classpath, I
> guess?
> > So, does that mean that we should exclude serde classes (including
> > SerializableSerde and JsonSerdeV2) in the Samza infrastructure package,
> and
> > tell the users to package them in application package?
> > d) I am a bit confused about the description on "multiple" application
> > classloaders on the job coordinator: one is for the describe flow and the
> > other is in the "Application" classloader, instead of "API" classloader,
> > right?
> >
> > Best,
> >
> > -Yi
> >
> >
> > On Wed, Mar 4, 2020 at 11:32 AM Ke Wu  wrote:
> >
> > > +1.
> > >
> > > Thanks for driving this effort.
> > >
> > > Best,
> > > Ke
> > >
> > > > On Mar 3, 2020, at 6:28 PM, Jagadish Venkatraman <
> > jagadish1...@gmail.com>
> > > wrote:
> > > >
> > > > +1 binding.
> > > >
> > > > Thanks Cameron. I look forward to this feature taking our "Stream
> > > > Processing as a service" offering to the next level.
> > > >
> > > > Cheers
> > > >
> > > > On Tuesday, March 3, 2020, Prateek Maheshwari 
> > > wrote:
> > > >
> > > >> +1 (binding) from me. Thanks for contributing this feature. Looking
> > > forward
> > > >> to having dependency isolation and 

Re: [VOTE] Apache Samza 1.4.0 RC1

[Bharath Kumara Subramanian]

+1(binding).

Check-all.sh and the integration tests passed.

Thanks,
Bharath

On Tue, Mar 10, 2020 at 3:35 PM Xinyu Liu  wrote:

> +1 (binding).
>
> Run check-all.sh and integration tests for both yarn and standalone. All
> passed.
>
> Thanks,
> Xinyu
>
>
> On Fri, Mar 6, 2020 at 6:46 PM Yi Pan  wrote:
>
> > Have downloaded the files, build with check-all.sh, and ran both YARN and
> > standalone integration tests. All passed.
> >
> > +1 (binding).
> >
> > Thanks!
> >
> > -Yi
> >
> > On Tue, Mar 3, 2020 at 3:03 PM Cameron Lee 
> > wrote:
> >
> > > Hi all,
> > >
> > > This is a call for a vote on a release of Apache Samza 1.4.0. Thanks to
> > > everyone who has contributed to this release.
> > >
> > > The release candidate can be downloaded from here:
> > > https://home.apache.org/~cameronlee/samza-1.4.0-rc1/
> > >
> > > The release candidate is signed with pgp key 0x54CB3CE3, which can be
> > found
> > > here:
> > >
> > >
> >
> https://keyserver.ubuntu.com/pks/lookup?search=0x54CB3CE3=on=index
> > > or to directly see the public key here:
> > >
> > >
> >
> https://keyserver.ubuntu.com/pks/lookup?op=get=0x71b0145290ecdbfa5caea6dbd786a7ba54cb3ce3
> > >
> > > The git tag is release-1.4.0-rc1, signed by the same pgp key above:
> > >
> > >
> >
> https://gitbox.apache.org/repos/asf?p=samza.git;a=commit;h=5327fafb8502b126482ec0c4efc8d1aa9b96ba44
> > >
> > > Test binaries have been published to Maven's staging repository, and
> are
> > > available here:
> > > https://repository.apache.org/content/repositories/orgapachesamza-1077
> > >
> > > The vote will be open for 72 hours (until Friday, March 6, 2020 at 3pm
> > > PST).
> > >
> > > Please download the release candidate, check the hashes/signature,
> build
> > it
> > > and test it, and then please vote:
> > > [ ] +1 approve
> > > [ ] +0 no opinion
> > > [ ] -1 disapprove (and reason why)
> > >
> > > I ran check-all.sh and integration tests.
> > >
> > > +1 (non-binding) from my side.
> > >
> > > Thank you,
> > > Cameron
> > >
> >
>

Re: Got Error Produce Respons with Correlation Id.

[Jeremiah Adams]

Can anyone take a look at the message below? We are trying to gauge our risk 
before moving forward.


Jeremiah Adams
Software Engineer
www.helixeducation.com
Blog | Twitter | Facebook | LinkedIn


From: Jeremiah Adams 
Sent: Wednesday, March 4, 2020 2:28 PM
To: dev@samza.apache.org
Subject: Got Error Produce Response iwth Correlation Id.

Hello devs,


I've got a warning showing up in the logs while testing our new Confluent Cloud 
config.  Can anyone tell me how concerned I should be about this warning? Is 
there a setting to control timeouts?


Also, logs stop at that point, so I can't tell if the "metatdata update" was 
complete.



2020-03-04 21:17:51 Sender [WARN] [Producer 
clientId=kafka_producer-application_submission-1] Got error produce response 
with correlation id 144 on topic-partition 
__samza_checkpoint_ver_1_for_application-submission_1-0, retrying 
(2147483646 attempts left). Error: NETWORK_EXCEPTION
2020-03-04 21:17:51 Sender [WARN] [Producer 
clientId=kafka_producer-application_submission-1] Received invalid metadata 
error in produce request on partition 
__samza_checkpoint_ver_1_for_application-submission_1-0 due to 
org.apache.kafka.common.errors.NetworkException: The server disconnected before 
a response was received.. Going to request metadata update now


Jeremiah Adams
Software Engineer
https://url.emailprotection.link/?bM9S-3pRw1lv8pYfwa-TwdjElP4W2K6b9vP5Crz22L_YcgsRJ-13h-OgPZSwFtU7GSNTDi1z-jdaRvWESRhtTVA~~
Blog
 | 
Twitter
 | 
Facebook
 | 
LinkedIn