[DISCUSS] SEP-8: Add in-memory system consumer & producer

Hi all, I have created SEP-8 for in-memory system support. Please find the design doc here Let me know your thoughts and feedback. Thanks, Bharath

[CANCEL][VOTE] Apache Samza 0.13.0 release

Hi all, This is an official CANCEL for the RC0 vote, and a status update for the new RC. Resending this since previous emails did not get delivered to the list. We found a few issues in RC0 during further testing. The following issues are already fixed and will be available in new RC:

[VOTE] SEP-8: Add in-memory system consumer & producer

Hi all, Can you please vote for SEP-8? You can find the design document here . Thanks, Bharath

[DISCUSS] Samza 0.14.0 release

Hi all, We have added couple of major features to master since 0.13.1 that warrants a major release. Within LinkedIn, some of these features have already been tested as part of our test suites. We plan to continue our testing in coming weeks to validate the stability prior to release. We

[VOTE] Apache Samza 0.14.0 RC2

Hi, This is a call for vote for the release of Apache Samza 0.14.0. We are excited to see new features and contributors in this release. The release candidate is signed with the pgp key, which can be found on keyservers: https://pgp.mit.edu/pks/lookup?op=get=0xE0A84E1C9C19AA37 The git

[CANCEL][VOTE] Apache Samza 0.14.0 RC1

Hi, This is an official CANCEL for the RC1 vote and a status update. We are waiting on the following items - SAMZA-1524 - Create Eventhub example in hello-samza - SAMZA-1525 - Add

[VOTE] Apache Samza 0.14.0 RC1

Hi, This is a call for vote for the release of Apache Samza 0.14.0. We are excited to see new features and contributors in this release. The release candidate is signed with the pgp key, which can be found on keyservers: https://pgp.mit.edu/pks/lookup?op=get=0xE0A84E1C9C19AA37 The git

[CANCEL][VOTE] Apache Samza 0.14.0 RC0

Hi, This is an official CANCEL for the RC0 vote and status update on the new RC. We have few last minute cleanup and javadoc changes related to SerDe support through APIs. Keep an eye out for the new RC. We are also working on updating documentation and tutorials for Samza-SQL. It will be

[VOTE] Apache Samza 0.14.0 RC0

Hi, This is a call for vote for the release of Apache Samza 0.14.0. We are excited to see new features and contributors in this release. The release candidate is signed with the pgp key, which can be found on keyservers: https://pgp.mit.edu/pks/lookup?op=get=0xE0A84E1C9C19AA37 The git

[VOTE] SEP-21: Samza Async API for High Level

Hi all, This is a call for a vote for SEP-21: Samza Async API for High Level SEP-21 has been discussed and implemented using SAMZA-2055. For reference, the design document can be found - https://cwiki.apache.org/confluence/display/SAMZA/SEP-21%3A+Samza+Async+API+for+High+Level Thanks,

Re: [DISCUSS] Change Apache Samza git comments/merge email recipient to commits@samza

+1 to separate discussion and commit threads. On Fri, Apr 5, 2019 at 5:23 PM Jagadish Venkatraman wrote: > +1 to the proposal Xinyu. > > Thanks, > > On Fri, Apr 5, 2019 at 5:01 PM Prateek Maheshwari > wrote: > > > Thanks Xinyu. Separating discussions and commit messages sounds good to > me. >

Re: Supported Versions Question

Hi Darin, The recommended dependency version in release notes are the ones we validate our releases. Typically, the patch and minor versions of the recommended dependency version works. Here is the supported matrix for Kafka clients Dependency Supported versions Not supported version Samza

Re: Supported Kafka version by Samza

HI Qi, You are right. I forgot to update the other thread on this. I suspected Kafka 1.0 was the reason for NPE since it had trouble instantiating the logger. Here is the Kafka client support matrix for our releases Dependency Supported versions Not supported version Samza version Kafka 0.11.*

Re: REMINDER. [VOTE] Apache Samza 1.2.0 RC4

+1 (non-binding) Verified build and test on Linux. I too noticed some intermittent failures on mac for Scala 2.12. Thanks, Bharath On Tue, Jun 4, 2019 at 2:00 PM Hai Lu wrote: > +1 (non-binding) > > Verified build and test on Linux box. On mac the test is failing but seems > like flakiness

Re: Can't start app in Yarn

AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471) > at > java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708) > at > java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) > at > java.util.stream.Reference

Re: [ANNOUNCE] Please welcome Boris Shkolnik to the Samza PMC

Congratulations Boris! On Fri, Jun 7, 2019 at 3:19 PM Jagadish Venkatraman wrote: > Congratulations Boris! > > On Fri, Jun 7, 2019 at 3:15 PM Xinyu Liu wrote: > > > Congrats, Boris! > > > > Xinyu > > > > On Fri, Jun 7, 2019 at 3:13 PM Jakob Homan wrote: > > > > > Howdy all- > > >I'm very

Re: Can't start app in Yarn

Hi Qi, Can you share your application configuration? Especially the systems your application consumes and produces to and its related configuration. Also, would it be possible for to attach the entire log? Thanks, Bharath On Tue, May 28, 2019 at 7:07 PM QiShu wrote: > Hi, > > Below is

Re: [VOTE] Apache Samza 1.2.0 RC0

-1 disapprove. We have a bug fix for Async API. The fix has been committed to master and we need to include it in 1.2. The commit hash c087f8 to cherry pick.

Re: [DISCUSS] 1.2 release

We have a critical bug fix for Async high level API. The changes have been committed to master. Can we include that as part of 1.2? Thanks, Bharath On Mon, May 20, 2019 at 3:38 PM Jake Maes

[DISCUSS] SAMZA-2305: Standalone rebalance bug

We recently encountered a bug in standalone that results in processors running with inconsistent job model. Here is a document that describes the problem and proposes few approaches to fix the

Re: [PROPOSAL] Async ParDo API

Please ignore the thread. It was intended for a different community. On Fri, Sep 13, 2019 at 9:37 AM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > Hi all, > > I have put a proposal > <https://docs.google.com/document/d/1t--UYXgaij0ULEoXUnhG3r8OZPBljN9

[PROPOSAL] Async ParDo API

Hi all, I have put a proposal to add async ParDo API. For context, please refer to the archive conversation: https://www.mail-archive.com/dev@beam.apache.org/msg12101.html Thanks, Bharath

Re: [VOTE] SEP-18: Startpoints - Manipulating Starting Offsets for Input Streams

+1 (non-binding) Looks good to me. On Fri, Sep 6, 2019 at 8:11 AM Jagadish Venkatraman wrote: > +1 (binding) > > Excellent work Dan! LGTM; > > On Fri, Sep 6, 2019 at 8:02 AM Daniel Nishimura > wrote: > > > Please vote for SEP-18 > > < > > >

Re: Occasional checkpoint mismatch on Samza task reload

Hi Malcolm, The warning is not particularly related to your restarts. It happens when the offset requested by the consumer no longer exists on the broker. It can typically happen if you are consuming from a topic with TTL enabled and the time retention kicked in and broker purged older offsets.

Re: [DISCUSS] SEP 25: PR Title and Description Guidelines

+1. Template looks good to me. It will be really helpful to sift through, categorize and prepare release notes from notable PRs during releases. Thanks, Bharath On Fri, Dec 13, 2019 at 11:24 PM Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > +1, thanks for the write-up Prateek. > >

Re: Samza App Deployment Topology - Best Practices

Hi Eric, Answers to your questions are as follows > > > > *Can I, or is it recommended to, package multiple jobs as 1 deploymentwith > 1 properties file or keep each app separated? Based on thedocumentation, > it appears to support 1 app/job within a singleconfiguration as there is no >

Re: Samza App Deployment Topology - Best Practices

ollowing: > > RESTful web services (or other none Samza services/applications) as Kafka > producer -> Input stream B (Kafka Topic B) -> Elasticsearch (or write to > HDFS) > > Regards, > > Eric > > On Mon, Oct 14, 2019 at 8:35 PM Bharath Kumara Subramanian < > codin.ma

Re: MessageStream with multiple concurrent operation stacks

Hi Eric, Based on the source code, it appears that each job designates a unique > group id when subscribing to kafka topic, is my understanding correct? > Yes. Samza uses a combination of job name and job id to generate the group id. > is it possible to have 2 independent stack of operations

Re: [VOTE] Apache Samza 1.3.0 RC0

-1 Unfortunately, we discovered a bug in side inputs for standby containers. Here is the ticket that tracks the issue - https://issues.apache.org/jira/browse/SAMZA-2382 Can we pull https://github.com/apache/samza/pull/1218 into the release as well? Thanks, Bharath On Tue, Nov 12, 2019 at 3:41

Re: [VOTE] SEP 25: PR Title and Description Guidelines

+1 (non-binding). Thanks, Bharath On Wed, Dec 18, 2019 at 10:42 AM Prateek Maheshwari wrote: > Hi folks, > > This is a call for a vote on SEP 25: PR Title and Description Guidelines. > Thanks to everyone who helped review the proposal and provided feedback. > > Feedback from the discussion is

Re: Running Samza with YARN Node label support

Hi Debraj, To get the node label working, set the label configurations[1] pointed out by Yang in your application config. Samza will take care of embedding the node label in the resource request automatically if it notices the label configuration inside your application. Samza framework respects

Re: Running Samza with YARN Node label support

ur custom YARN distribution if applicable. Thanks, Bharath On Wed, Dec 18, 2019 at 1:15 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > Hi Debraj, > > To get the node label working, set the label configurations[1] pointed out > by Yang in your application config. Samza

Re: Samza - New Relic integration

Hi, I am not sure I fully understand the ask. IIUC, Samza doesn't have native integration with New relic. However, you should still be able to integrate your application with New relic on your end without native support. If you are particularly looking to integrate native Samza metrics w/ New

Re: [VOTE] Apache Samza 1.3.1 RC0

Ran check-all and integration tests passed with one exception. The following test was flaky and passed on the second attempt for a specific combination (Scala - 2.12, YARN - 2.7.1 & JDK 8) > Task :samza-test_2.12:test testRepartitionedSessionWindowCounter FAILED java.lang.AssertionError:

Re: Samza throwing exception: Duplicate key org.apache.beam.runners.samza.runtime.KeyedTimerData@5a5eb850 registration for the same timer

Hi Sandeep, Thank you for reporting this issue. We noticed this problem during our testing for upcoming Samza release. You can find more details about the bug - https://issues.apache.org/jira/browse/SAMZA-2463 We have a fix and should be included in our upcoming Samza 1.4 release that is slotted

Re: [DISCUSS] Samza 1.3.1 release

+1. Cheers, Bharath On Thu, Feb 13, 2020 at 12:54 PM Yang Zhang wrote: > +1. Thanks! > > Best, > Yang > > On Thu, Feb 13, 2020 at 12:06 PM Xinyu Liu wrote: > > > Cool! Thanks for driving this release. > > > > Thanks, > > Xinyu > > > > On Thu, Feb 13, 2020 at 10:28 AM Hai Lu wrote: > > > > >

Re: Reduce kafka partition for a topic samza is using

m temp topic to new topic > > What I have do with the __samza_checkpoint and __samza_coordinator_ topics? > Should I also delete them? > > Can you explain what do you mean by reroute? > > On Mon, Jan 6, 2020 at 7:40 PM Bharath Kumara Subramanian < > codin.mart...@g

Re: [VOTE] Apache Samza 1.4.0 RC1

+1(binding). Check-all.sh and the integration tests passed. Thanks, Bharath On Tue, Mar 10, 2020 at 3:35 PM Xinyu Liu wrote: > +1 (binding). > > Run check-all.sh and integration tests for both yarn and standalone. All > passed. > > Thanks, > Xinyu > > > On Fri, Mar 6, 2020 at 6:46 PM Yi Pan

Re: Execution of action: JobModelVersionChange failed.

Hi Omkar, The errors are related to timeouts during shutdown which gets triggered during a rebalance. Whenever a new processor joins the quorum or leaves the quorum, a rebalance is triggered which requires all the existing processors to shutdown its container before agreeing on the new job model.

Re: Execution of action: JobModelVersionChange failed.

to handle this? > > And does rebalancing always include stopping all the containers? > We are running on K8S and the pods are often moved around. And every time > a pod is moved, rebalance will be triggered. > And the rebalance in turn will restart all other pods. > > > On 3/22/20,

[DISCUSS] Samza 1.5.1 release

Hi all, In 1.5 release, we enabled transactional state by default for all samza jobs. We identified a critical bug related to trimming the state which requires a minor release. I wanted to kick off the discussion on the open source forum as the bug fix has been validated internally at LinkedIn.

[RESULT][VOTE] Apache Samza 1.5.1 RC0

Hi all, The vote for 1.5.1 release has been out for more than 72 hours and we got +1(binding) x3 (Yi, Boris, Bharath) Samza 1.5.1 officially passed the VOTE! Thanks for your contribution and help with validation. -- Bharath

Re: [DISCUSS] Samza 1.5.1 release

n wrote: > > > +1 to this release as well! > > > > On Wed, Aug 19, 2020 at 9:21 AM Prateek Maheshwari > > wrote: > > > > > +1, this is a critical bug and we should release the fix ASAP. > > > > > > Thanks, > > > Prateek &g

[VOTE] Apache Samza 1.5.1 RC0

Hi all, This is a call for a vote on a release of Apache Samza 1.5.1. We are releasing 1.5.1 to address a critical bug related transaction state feature. More details on the bug can be found here: https://issues.apache.org/jira/browse/SAMZA-2578 The release candidate can be downloaded from

Re: [VOTE] SEP-22: Container Placements in Samza

+1. Looking forward to use the feature. -- Bharath On Fri, May 29, 2020 at 12:16 PM Prateek Maheshwari wrote: > +1 from me too. Thanks for adding this feature! > > - Prateek > > On Fri, May 29, 2020 at 11:56 AM Boris S wrote: > > > Hi, > > LGTM. > > +1. > > > > On Wed, May 27, 2020 at 11:28

[DISCUSS] Samza 1.5 release

Hi all, We have accumulated few features/improvements since the last release and would like to make Samza 1.5 release. I wanted to kick off the discussion on the open source forum as some of these changes have already been tested internally at LinkedIn. Some of the features/improvements include

Re: [VOTE] Apache Samza 1.5.0 RC0

) at org.junit.Assert.assertTrue(Assert.java:52) at org.apache.samza.table.batching.TestBatchProcessor$TestBatchTriggered.testBatchOperationTriggeredByBatchSize(TestBatchProcessor.java:114) We are working on fixing the issue. Thanks, Bharath On Thu, Jun 4, 2020 at 9:59 PM Bharath Kumara Subramanian

Re: [DISCUSS] Samza 1.5 release

Since there is no objection. I am moving on with a vote for the release. Thanks, Bharath On Mon, Jun 1, 2020 at 8:37 PM Yi Pan wrote: > lgtm. Thanks for kicking off the discussion! > > -Yi > > On Tue, May 26, 2020 at 8:55 PM Bharath Kumara Subramanian < > codin.mar

[VOTE] Apache Samza 1.5.0 RC0

Hi all, This is a call for a vote on a release of Apache Samza 1.5.0. We are excited to see some new features and improvements in this release. The release candidate can be downloaded from here: http://home.apache.org/~bharathkk/samza-1.5.0-rc0/ The release candidate is signed with pgp key

[RESULT][VOTE] Apache Samza 1.5.0 RC1

The vote for 1.5.0 release has been out for more than 72 hours and we got +1 (binding) x3 (Yi, Boris, Bharath) +1(non-binding) x1 (Ke) Samza 1.5.0 officially passed the VOTE! Thanks for your contribution to the release and help with validation. -- Bharath

[VOTE] Apache Samza 1.5.0 RC1

Hi all, This is a call for a vote on a release of Apache Samza 1.5.0. We are excited to see some new features and improvements in this release. The release candidate can be downloaded from here: http://home.apache.org/~bharathkk/samza-1.5.0-rc1/ The release candidate is signed with pgp key

Re: [VOTE] Apache Samza 1.6.0 RC1

Thanks for driving this release Boris! Verified signatures and ran all the tests. check-all and integration tests passed. +1 (binding) -- Bharath On Mon, Jan 11, 2021 at 1:28 PM Boris S wrote: > Quick reminder, > Please take some time to validate the release and vote on it. > > Thanks, >

Re: [DISCUSS] Apache Samza 1.6.0 RC0

+1. Can you please tag all the JIRAs with fix version = 1.6? The link in the email points to JIRAs across versions. Looking forward to the new release. On Wed, Nov 18, 2020 at 10:36 AM Boris S wrote: > ** with correct subject this time** > > Hi all, > > We have added a number of major features

Re: Upgrading from 0.14 -> 1.5.1

Hi Malcolm, Based on the following log INFO [org.apache.samza.zk.ZkJobCoordinator] New JobModel does not contain > pid=a3e86ddf-8d18-40c9-8063-1efd588cec56. Stopping this processor. New > JobModel: JobModel [..] > I'd have to guess that the processor isn't part of the quorum (list of

Re: Welcome Sanil Jain as Apache Samza committer

Congratulations Sanil! On Tue, Feb 2, 2021 at 3:43 PM Miguel Sanchez Schwarz wrote: > Congratulations Sanil!! > > From: Wei Song > Sent: Tuesday, February 2, 2021 2:27 PM > To: dev@samza.apache.org > Subject: Re: Welcome Sanil Jain as Apache Samza committer >

Re: [VOTE] SEP-28: Samza State Backend Interface and Checkpointing Improvement

+1 (binding). Looking forward to seeing this running in production. -- Bharath On Tue, Jun 22, 2021 at 1:41 PM Sanil Jain wrote: > +1 (non-binding) Thanks for the contribution > > On Tue, 22 Jun 2021 at 13:09, Prateek Maheshwari > wrote: > > > +1 (binding). Thanks for the contribution! > > >

Re: [VOTE] SEP-29: Blob Store Based State Backup And Restore

+1 (binding). Thanks for the contribution! -- Bharath On Tue, Jun 22, 2021 at 8:19 PM Yi Pan wrote: > +1 (binding). Thanks for rolling out this big feature! > > -Yi > > On Tue, Jun 22, 2021 at 1:42 PM Sanil Jain wrote: > > > +1 (non-binding) Thanks for this contribution! > > > > -Sanil > > >

Draft board report for Samza

Hi all, Here is a draft report for Samza. Please let me know if I missed anything. ## Description: The mission of Samza is the creation and maintenance of software related to distributed stream processing

Re: Kafka Offset Commit

Hi Talat, It is expected behavior since Samza uses low level Kafka consumer instead of the high level consumer. As a result, offset management is done by Samza and doesn't leverage the offset management that Kafka consumer has by default. Additionally, kafka consumer group doesn't apply to samza

Re: [ANNOUNCE] Welcome Daniel Chen as Samza Committer

Congratulations Daniel! Looking forward to more contributions. On Mon, Sep 20, 2021 at 8:03 AM Hai Lu wrote: > Congrats!!! > > On Fri, Sep 17, 2021 at 2:06 PM Sanil Jain wrote: > > > Congrats Daniel ! > > > > On Fri, 17 Sept 2021 at 11:40, Jagadish Venkatraman < > >

Re: [DISCUSS] Apache Samza 1.7.0 RC0

+1 from my end. On Wed, Jan 26, 2022 at 3:07 PM Yi Pan wrote: > Huge +1! Can't wait to see this list of features coming out! > > -Yi > > On Wed, Jan 26, 2022 at 2:36 PM Daniel Chen wrote: > > > Hi folks, > > > > We have added a number of major features and changes to master since > > > > 1.6,

Re: [VOTE] SEP-30: Support Updates in Table API

+1 (binding). Looking forward to seeing this in production. Thanks, Bharath On Mon, Jan 24, 2022 at 4:58 PM Yi Pan wrote: > Discussed and resolved the minor concerns offline. +1 (binding) for this > one. > > Thanks! > > -Yi > > On Tue, Dec 21, 2021 at 1:28 PM Xinyu Liu wrote: > > > +1 on my

Re: [DISCUSS] SEP-32: Elasticity for Samza

+1 on my end. Looks good to me. Thanks for putting this together, Manasa! Cheers, Bharath On Mon, Feb 6, 2023 at 11:51 PM Jagadish Venkatraman wrote: > Thank you Manasa for the proposal. I reviewed it and it looks good to me. > nice work! > > +1 (approve) from my end. > > > > On Mon, Feb 6,

Re: [VOTE] SEP-32: Elasticity for Samza

+1 (binding) Cheers, Bharath On Tue, Feb 7, 2023 at 12:56 PM Lakshmi Manasa wrote: > Hi folks, > > This is a call for vote on SEP-32: Elasticity for Samza. > Thank you for reviewing the SEP and giving feedback. > > I have addressed the comments on the SEP and since there were three +1 on >