Re: [VOTE] SEP-32: Elasticity for Samza

Thank you Manasa. I reviewed it and it looks good to me. +1 (approve) this SEP. On Tue, Feb 7, 2023 at 2:14 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > +1 (binding) > > Cheers, > Bharath > > On Tue, Feb 7, 2023 at 12:56 PM Lakshmi Manasa > wrote: > > > Hi folks, > > > >

Re: [DISCUSS] SEP-32: Elasticity for Samza

Thank you Manasa for the proposal. I reviewed it and it looks good to me. nice work! +1 (approve) from my end. On Mon, Feb 6, 2023 at 11:41 PM Yi Pan wrote: > Hi, Manasa, > > Sorry for the late reply. The revision lgtm. Thanks for the great work! > > Best, > > -Yi > > On Mon, Jan 30, 2023 at

Re: [VOTE] SEP-31: Pipeline Drain- Support the ability to drain pipelines to allow incompatible intermediate schema changes

Hi Ajo, I reviewed your work and the overall design. Looks great! Thank you for contributing this much needed feature. [+1] I approve. On Thu, Dec 8, 2022 at 4:45 PM Yi Pan wrote: > +1. Long awaited feature! Thanks! > > -Yi > > On Tue, Nov 29, 2022 at 11:46 AM Xinyu Liu wrote: > > > +1. >

Re: [ANNOUNCE] Welcome Daniel Chen as Samza Committer

Congrats Daniel on this well deserved recognition. Look forward to more contributions! On Fri, Sep 17, 2021 at 11:25 AM Yi Pan wrote: > Congrats, Daniel, well deserved!!! > > -Yi > > On Fri, Sep 17, 2021 at 11:23 AM Xinyu Liu wrote: > > > Hi, all, > > > > I am glad to announce that Daniel

Re: How are samza.container.id generated in yarn?

Hi Debraj, Clarifying a bit on Yi’s response, since it was referring to the physical Yarn container id.. If there are N Yarn containers, samza.container.ids are generated sequentially from 0 to N-1. This ID is meant to be durable - ie., if a particular container fails, the Samza AM will restart

Re: Welcome Sanil Jain as Apache Samza committer

Congrats Sanil! Looking forward to more contributions! On Tue, Feb 2, 2021 at 7:31 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > Congratulations Sanil! > > On Tue, Feb 2, 2021 at 3:43 PM Miguel Sanchez Schwarz > wrote: > > > Congratulations Sanil!! > >

Re: [DISCUSS] Samza 1.5.1 release

+1. On Saturday, August 22, 2020, Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > It has been more than 72+ hours and the response seems positive. > Let me proceed with a voting thread. > > Thanks, > Bharath > > On Wed, Aug 19, 2020 at 10:31 AM Xinyu Liu wrote: > > > +1.

Re: [VOTE] SEP-22: Container Placements in Samza

+1 (binding) Thanks Sanil for driving this proposal! On Fri, May 29, 2020 at 5:24 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > +1. > Looking forward to use the feature. > > -- > Bharath > > On Fri, May 29, 2020 at 12:16 PM Prateek Maheshwari > wrote: > > > +1 from me too.

Re: [VOTE] SEP-24: Cluster-based Job Coordinator Dependency Isolation

+1 binding. Thanks Cameron. I look forward to this feature taking our "Stream Processing as a service" offering to the next level. Cheers On Tuesday, March 3, 2020, Prateek Maheshwari wrote: > +1 (binding) from me. Thanks for contributing this feature. Looking forward > to having dependency

Re: Samza - New Relic integration

-9505020924 > > vaibhavgar...@gmail.com > > LinkedIn <https://www.linkedin.com/in/vaibhavgarg90/> > > > > > > On Sun, Mar 1, 2020 at 12:31 PM Jagadish Venkatraman < > > jagadish1...@gmail.com> wrote: > > > >> Hi Vaibhav, > >> > >>

Re: Samza - New Relic integration

Hi Vaibhav, Can you try ./__package/newrelic/newrelic.jar? If that doesn't work, I'd recommend doing the following: 1. Print the current working directory of your Samza container JVM 2. Use (1) to determine a relative reference to the directory path within your package Cheers Jagadish On

Re: [VOTE] Apache Samza 1.3.1 RC0

+1 (binding) On Wednesday, February 19, 2020, Prateek Maheshwari wrote: > Integration tests and check-all passed successfully. +1 (binding) from me. > > Thanks, > Prateek > > On Tue, Feb 18, 2020 at 12:47 PM Bharath Kumara Subramanian < > codin.mart...@gmail.com> wrote: > > > Ran check-all and

Re: [ANNOUNCE] Please welcome Bharath Kumarasubramanian to the Samza PMC

Congrats Bharath. Great work! Looking forward to continued contributions! On Thursday, February 13, 2020, Yang Zhang wrote: > Congratulations, Bharath! Nice work and thanks for the contributions! > > Best, > Yang > > On Thu, Feb 13, 2020 at 4:27 PM Xinyu Liu wrote: > > > Hi all, > > > > I'm

Re: [DISCUSS] Samza 1.3.1 release

+1 thank you Hai for shepherding this release. On Thu, Feb 13, 2020 at 3:53 PM Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > +1. > > Cheers, > Bharath > > On Thu, Feb 13, 2020 at 12:54 PM Yang Zhang wrote: > > > +1. Thanks! > > > > Best, > > Yang > > > > On Thu, Feb 13, 2020 at

Re: [Draft] Samza quarterly report

lgtm, thanks Yi. On Thu, Jan 9, 2020 at 6:46 PM Prateek Maheshwari wrote: > Looks good to me, thanks! > > - Prateek > > On Thu, Jan 9, 2020 at 1:43 PM Yi Pan wrote: > > > ## Description: > > - Apache Samza is a distributed stream processing engine that are highly > > configurable to process

Re: [VOTE] SEP-26: Add SystemProducer for Azure Blob Storage

+1 (binding), looking forward to Samza's integration with Azure blobs On Wednesday, January 8, 2020, Lakshmi Manasa wrote: > Hi, > > This is a call for a vote on SEP-26: Add SystemProducer for Azure Blob > Storage. > Thanks for taking a look and giving feedback. > > I have addressed the

Re: [VOTE] SEP 25: PR Title and Description Guidelines

+1 (binding) On Thursday, December 19, 2019, Daniel Nishimura wrote: > +1 non-binding > > > On Dec 18, 2019, at 7:09 PM, Yi Pan wrote: > > > > +1 (binding) > > > > On Wed, Dec 18, 2019 at 10:49 AM Bharath Kumara Subramanian < > > codin.mart...@gmail.com> wrote: > > > >> +1 (non-binding). > >>

[DISCUSS] SEP 25: PR Title and Description Guidelines

+1, thanks for the write-up Prateek. Let's also update the contributor's guidelines at: https://samza.apache.org/contribute/contributors-corner.html On Friday, December 13, 2019, Prateek Maheshwari wrote: > Hi folks, > > In order to make Samza PR descriptions and commit messages more

Re: [DISCUSS] SEP-23: Simplify Job Runner

+1 Lgtm, thanks Ke for the proposal. On Tuesday, December 10, 2019, Prateek Maheshwari wrote: > Looks good to me as well. +1 for the overall proposal, and thanks for > putting it together. > > - Prateek > > On Tue, Dec 10, 2019 at 1:26 PM Xinyu Liu wrote: > > > Thanks for updating the SEP

Re: [VOTE] Apache Samza 1.3.0 RC2

Hai, Do you mind sending a VOTE RESULT email to the mailing list to officially conclude the vote? Thanks Jagadish On Wednesday, December 4, 2019, Hai Lu wrote: > Thanks everyone. The vote passed. > > > On Wed, Dec 4, 2019 at 3:25 PM Prateek Maheshwari > wrote: > > > + 1 (binding) > > > >

Re: [VOTE] SEP-20: Samza on Kubernetes

+1 binding. Thanks Weiqing for driving this! On Thursday, November 7, 2019, Xinyu Liu wrote: > +1 (binding). > > Thanks, > Xinyu > > On Thu, Nov 7, 2019 at 10:50 AM Weiqing Yang > wrote: > > > Hi All, > > > > The feedback from the discussion thread: > >

Re: [DISCUSS] SEP-20: Samza on Kubernetes

+1, look forward to Samza K8s integration :) On Monday, November 4, 2019, Xinyu Liu wrote: > +1 on the design. This is a great feature to allow Samza to expand its > deployment to Kubernetes clusters. Nice job! > > Thanks, > Xinyu > > On Mon, Nov 4, 2019 at 10:10 AM Weiqing Yang > wrote: > > >

Re: [VOTE] SEP-18: Startpoints - Manipulating Starting Offsets for Input Streams

+1 (binding) Excellent work Dan! LGTM; On Fri, Sep 6, 2019 at 8:02 AM Daniel Nishimura wrote: > Please vote for SEP-18 > < > https://cwiki.apache.org/confluence/display/SAMZA/SEP-18%3A+Startpoints+-+Manipulating+Starting+Offsets+for+Input+Streams > >. > Thanks to the committers and

Re: At-least once processing guarantee

ish, > Is the rejected alternatives document meant to be private (link at the > bottom of the document)? > > On Wed, Jul 24, 2019 at 11:46 AM Jagadish Venkatraman > > wrote: > > > Dear Samza users, > > > > We recently discovered an issue with the way we handle state in Samza

At-least once processing guarantee

Dear Samza users, We recently discovered an issue with the way we handle state in Samza Beam and Samza High-Level API Window operators. Under certain situations, at least once processing guarantee is violated. *Details on the issue* The Samza high-level API includes operators such as windows

[ANNOUNCE] Welcoming Rayman and Bharath as new Samza committers!

I'm pleased to announce that the Samza PMC has voted to invite Rayman & Bharath as committers, and they have accepted. Please join us in congratulating them on this recognition! A quick summary of their accomplishments.. *Rayman * Ray has been driving multiple improvements to Samza for stateful

Re: [ANNOUNCE] Please welcome Boris Shkolnik to the Samza PMC

Congratulations Boris! On Fri, Jun 7, 2019 at 3:15 PM Xinyu Liu wrote: > Congrats, Boris! > > Xinyu > > On Fri, Jun 7, 2019 at 3:13 PM Jakob Homan wrote: > > > Howdy all- > >I'm very pleased to announce that the Samza PMC has voted Boris > > Shkolnik to be a Project Management Committee

Re: Auto-sizing subsystem for Samza

Thanks Ray. This proposal that takes Samza's capabilities to the next level. On Thu, Jun 6, 2019 at 11:38 AM rayman preet wrote: > I'm trying to put together a Samza subsystem that is able to auto-size a > Samza job > by looking at metrics. Here are some initial thoughts: > >

Re: REMINDER. [VOTE] Apache Samza 1.2.0 RC4

+1 (binding) Verified signatures, built the RC successfully. On Wed, Jun 5, 2019 at 5:17 PM Prateek Maheshwari wrote: > +1 (binding) > > Verified build + check-all + integration tests + signatures. > Thanks for help with the release, Boris and Pawas. > > - Prateek > > On Wed, Jun 5, 2019

Re: AM resource needs

use of the > recipient(s) named above. If you have received this message in error, > please notify the sender immediately and delete the original message. > > > On Thu, May 23, 2019 at 10:39 PM Jagadish Venkatraman < > jagadish1...@gmail.com> wrote: > > > No, the def

Re: AM resource needs

No, the default setup should be sufficient - the number of tasks should have no significant impact on AM memory/resources. If you run out-of-memory, you can of course increase yarn.am.memory.mb. On Thu, May 23, 2019 at 10:45 AM Malcolm McFarland wrote: > Hey folks, > > Are there any guidelines

Re: Samza tasks aren't starting in YARN containers

t.ConnectException. Retrying... > > There's no other information in the log about what is going on. Does > anybody have ideas on this? > > Btw, how do I pull a thread-dump of the stuck container? > > Cheers, > Malcolm > > > On Tue, May 7, 2019 at 10:48 PM Jagadish Venka

Re: Samza tasks aren't starting in YARN containers

Malcolm, Did the AM-process come up? If so, can you attach its entire log-file? "> everything will launch fine one time, and then it will do this RUNNING-but-no-Samza thing the next." IIUC, you believe your container is not making progress. If the issue is recurs, can you attach a thread-dump &

Re: Samza for text processing

more appropriate? > > Also for files over 1mb would you increase the default kafka limit? Break > the document into chunks or pass a reference in the message? > > Thanks again > > > > On Sun, 28 Apr 2019, 16:20 Jagadish Venkatraman, > wrote: > > > Hi Rob, >

Re: Running Apache Samza on Kubernetes cluster with Zookeeper

+Weiqing Yang, who gave a community talk on this at KubeCon this year. Hi Stefano, Running Samza-standalone + Zk on Kubernetes should be no different than running any other application on Kubernetes. At a high-level, you would: 1. Package your application as a container image - just as you

Re: Samza for text processing

Hi Rob, Yes, your use-case is a good fit. You can use Samza for fault-tolerant stream processing. We have document (eg: member profiles, articles/blogs) standardization use-cases at LinkedIn powered by Samza. Please let us know should you have further questions! On Sun, Apr 28, 2019 at 7:09 AM

Re: [ANNOUNCE] New committer announcement: Cameron Lee

Awesome addition! Congrats Cameron. well deserved. On Tuesday, April 16, 2019, Xinyu Liu wrote: > Hi, all, > > Please join me and the rest of the Samza PMC in welcoming a new committer: > Cameron Lee. > > Cameron has been contributing to Samza since early 2018. He worked on > multiple areas:

Re: [DISCUSS] Change Apache Samza git comments/merge email recipient to commits@samza

+1 to the proposal Xinyu. Thanks, On Fri, Apr 5, 2019 at 5:01 PM Prateek Maheshwari wrote: > Thanks Xinyu. Separating discussions and commit messages sounds good to me. > I'm +1, but happy to keep it as-is if others find the commit emails useful. > > - Prateek > > On Thu, Apr 4, 2019 at 3:14

Re: Empty Kafka topic partition Warning

itions empty. One of our upstream jobs is likely configured > incorrectly. > > > > > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > ________ > From: Jagadish Venkatr

Re: SSL with Samza 0.14.1?

Hi Matt, >> We’re trying to connect to a cluster that has SSL enabled, and we’re getting some errors that are indicative of SSL connectivity failing. Can you share some more details on the errors - eg: log files, stack-traces? Do you get the error when Samza attempts to create the topic? >> Is

Re: Empty Kafka topic partition Warning

res.redelivery-store.changelog=kafka.delivery-changelog > stores.default.changelog.replication.factor=3 > > > > > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > >

Re: Empty Kafka topic partition Warning

Hi Jeremiah, >> why is the offset 0? This likely means that the change-log is empty and does not have any messages. Can you try consuming from partition-number: 0 using a KafkaConsumer? Best, Jagadish On Fri, Mar 22, 2019 at 11:45 AM Jeremiah Adams wrote: > I'm seeing these in our log

Re: [VOTE] Apache Samza 1.1.0 RC2

1. Verified check-sum and signatures for the release binaries. 2. Ran ./check-all.sh successfully 3. Ran YARN integration tests successfully 4. Encountered an error on the standalone integration test, but it succeeded after setting Kafka's replication factor config to 1. +1(binding) from my side.

Announcing Samza 101: Video tutorials on Stream processing

Hello Samza users, We are really excited to share with you a short video course for getting started with Stream Processing and Samza: https://www.youtube.com/playlist?list=PLZDyxA22zzGyNgtBMUIXAgIaO5Ok3PR-x A few topics

Re: Looking for YARN Alternative

The standalone mode was introduced for this exact reason for customers who don’t want to run YARN. Have you considered running Samza in stand-alone mode? In this mode, Samza is an embedded library - very similar to Kafka Streams.

Re: [VOTE] Migration of Samza git repo to gitbox.apache.org

+1 (binding). Thank you Pawas for driving this! On Wed, Jan 23, 2019 at 2:40 PM Xinyu Liu wrote: > +1 (binding). > > On Wed, Jan 23, 2019 at 2:39 PM Prateek Maheshwari > wrote: > > > +1 (binding) again > > > > - Prateek > > > > On Wed, Jan 23, 2019 at 11:50 AM Pawas Chhokra > > wrote: > > > >

Re: Question about too much log on resource manager and node manager

Hi Qi, >> I want to know is this normal and how to avoid such log. >From Samza's perspective, I would treat these messages as red-herrings as long as your containers are up and running. The INFO messages in the ResourceManager log correspond to resources requested by your application.

Re: Alerting and Monitoring Samza Checkpointing?

Hi Jeremiah, +1 to what Tom said. Samza currently does not rely on Kafka consumer's checkpointing behavior and exposes its own notion of a "lag". This is reported as a per-partition metric under KafkaSystemConsumerMetrics#messagesBehindHighWatermark

Re: [VOTE] [RESULT] Apache Samza 1.0.0 RC4

gt; > Cheers! > > -Yi > > On Tue, Nov 6, 2018 at 9:33 PM Jagadish Venkatraman < > jagadish1...@gmail.com> > wrote: > > > The vote of 1.0.0 has been more than 72 hours. We got +1 (binding)x 3 and > > +1 (non-binding) x 3 and no vetos. > > > > *Bindi

Release artifacts published for Samza 1.0.0

Hey everyone, Now that the vote has concluded, binaries for Samza 1.0.0 have been published to Maven.. I'll follow up with a blogpost on the release early next week. Note that Samza 1.0.0 has a dependency on Kafka 0.11.2, which does not support

[VOTE] [RESULT] Apache Samza 1.0.0 RC4

t; samza-hello-samza (High level and low level jobs). > > +1 > > > > On Wed, Oct 31, 2018 at 7:15 PM Jagadish Venkatraman < > jagad...@apache.org> > > wrote: > > > > > Hi all, > > > > > > This is a call for a vote on a release of Apache Samza

[VOTE] Apache Samza 1.0.0 RC4

Hi all, This is a call for a vote on a release of Apache Samza 1.0.0. Thanks to everyone who has contributed to this release. The release candidate can be downloaded from here: http://home.apache.org/~jagadish/samza-1.0.0-rc4/ The release candidate is signed with pgp key AF81FFBF, which can be

Re: [VOTE] Apache Samza 1.0.0 RC2

Boris, Do users have the option to switch to use the "old" Kafka consumer if they encounter any issue with the "new" consumer?. If not, should we pull in https://github.com/apache/samza/pull/755? It is my understanding that PR-755 adds support for this. Thanks, Jagadish On Tue, Oct 23, 2018 at

Re: [VOTE] Apache Samza 1.0.0 RC0

I ran the integration test and encountered this failure: 2018-10-22 15:58:43,687 zopkio.test_runner [INFO] test_samza_jobfailed 2018-10-22 15:58:43,688 zopkio.test_runner [INFO] ['AssertionError: Job (negate_number) appears not to have started. Expected to see a log line matching regex:

Re: [VOTE] SEP-15: New Runtime Context API

+1 (binding) from my side. LGTM On Mon, Oct 15, 2018 at 10:44 AM Prateek Maheshwari wrote: > +1 (non-binding) for these changes. > > (Resending from a non-LI email due to email delivery issues) > > - Prateek > On Fri, Oct 12, 2018 at 3:28 PM Cameron Lee wrote: > > > > Hi all, > > > > SEP-15

Re: [VOTE] SEP-14: System and Stream Descriptors

+1 binding from my side, thanks! This is a great addition to Samza-1.0 On Fri, Oct 12, 2018 at 12:30 PM Prateek Maheshwari wrote: > Hi folks, > > Now that SAMZA-1804 has been implemented and reviewed, we've updated > SEP-14 with the latest APIs and design decisions. > > Please vote for

Re: [VOTE] SEP-13: unified ApplicationDescriptor and ApplicationRunner APIs for high and low- level APIs in YARN and standalone deployment

+1(binding) from my side. This is a great overhaul of our APIs. Thanks On Monday, October 15, 2018, Xinyu Liu wrote: > +1 (binding). Thanks for the effort to making our API consistent. > > Thanks, > Xinyu > > On Fri, Oct 12, 2018 at 12:32 PM Prateek Maheshwari > wrote: > > > +1 (non-binding)

Re: Path to Java 11

; on-the-java-se-release-cadence > > > > > Jeremiah Adams > Software Engineer > www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > > From: Jagadish Venkatraman > Sent: Tuesday, September 11, 2018 1:09

Re: Path to Java 11

Hi Jeremiah, Most of the users in the Samza community currently use Java 8. We have not investigated/tested Java-11 support just yet since it's fairly bleeding edge. Are you aware of any known incompatibilities / pitfalls? Thanks, Jagadish On Mon, Sep 10, 2018 at 2:52 PM, Jeremiah Adams

Re: Samza/Yarn cluster having issue with OffsetOutOfRangeException

Hi Will, Is the topic in question your change-log topic or the checkpoint-topic or one of your inputs? (My understanding from reading this is its your checkpoint) Can you please attach some more surrounding logs? Thanks, Jagadish On Mon, Aug 20, 2018 at 6:16 AM, Will Schneider wrote: >

Re: Using dynamodb as source stream

Hi Anoop, Currently, Samza has built-in support for consuming data from AWS Kinesis. As a short-term solution, if you want to consume from "Dynamo DB Streams", you can certainly pipe it to Kinesis and write a Samza job that consume from it. Please see: Consuming from Kinesis

Re: Question regarding Samza's Kafka consumer

Hi Thomas, >> Has Samza been tested against newer broker versions using the new message format, and if so does it have a significant performance impact? We have not benchmarked Kafka broker performance with the new message format. Any benchmarking may not be reliably reproducible since there are

Re: MessageCollector and WindowableTask

> www.helixeducation.com > Blog | Twitter | Facebook | LinkedIn > > > From: Jagadish Venkatraman > Sent: Tuesday, June 5, 2018 10:22 AM > To: dev@samza.apache.org > Subject: Re: MessageCollector and WindowableTask > > Hey Jeremiah, > > >> Please only use t

Re: MessageCollector and WindowableTask

Hey Jeremiah, >> Please only use that MessageCollector object for sending messages, and don't use it outside of the call to window(). Why MessageCollector should not be used outside the call to window() in a WindowableTask? The collector and coordinator instances are expected to be used within a

Re: [VOTE] SEP-12: Integration Test Framework

Thanks Sanil for the proposal. This will go a long way in simplifying testing of Samza applications. +1 (binding) On Thu, May 17, 2018 at 9:45 AM, Daniel Nishimura wrote: > +1 > > Looks great! > > On Thu, May 17, 2018 at 9:08 AM, Xinyu Liu wrote:

Re: [Discuss] Samza 0.14.1 release

Xinyu, This comes up with an exciting lineup of features. +1 (binding) from my side. Tom, >> These all look like excellent changes to me, I'd be happy to have more stability in Standalone! We are actively investing in standalone and have fixed some edge-cases that surfaced when running it at

Re: [VOTE] SEP-11: Host affinity in standalone.

, Jagadish Venkatraman <jagadish1...@gmail.com > wrote: > Let's extend the vote to Monday 11:59PM PST. > > On Thu, Apr 5, 2018 at 5:05 PM, xinyu liu <xinyuliu...@gmail.com> wrote: > >> +1 (binding). Look forward to the implementation. >> >> Xinyu >> &

Re: [VOTE] SEP-11: Host affinity in standalone.

Let's extend the vote to Monday 11:59PM PST. On Thu, Apr 5, 2018 at 5:05 PM, xinyu liu wrote: > +1 (binding). Look forward to the implementation. > > Xinyu > > On Wed, Apr 4, 2018 at 2:43 PM, Yi Pan wrote: > > > +1 (binding). Thanks for the

Re: Old style "low level" Tasks with alternative deployment model(s)

our web-page: https://samza.apache.org/learn/documentation/0.14/aws/kinesis.html Best, Jagdish On Tue, Mar 20, 2018 at 11:24 AM, Tom Davis <t...@recursivedream.com> wrote: > What format would that be best in? Happy to put something together this > weekend as well. > > J

Re: Old style "low level" Tasks with alternative deployment model(s)

d. >> -Thanks, >> Thunder >> >> >> -Original Message- >> From: Thunder Stumpges >> Sent: Monday, March 19, 2018 20:58 >> To: dev@samza.apache.org >> Cc: Jagadish Venkatraman <jagadish1...@gmail.com>; t...@recursivedream.com; >

Re: Stream Processing Meetup @LinkedIn tomorrow (3/21)

Hey Tom, No sign-ups are necessary to watch the live-stream. The sign-ups are for in-person attendance. Hoping you'll join us virtually :-) Thanks, Jagdish On Tue, Mar 20, 2018 at 10:20 AM, Tom Davis wrote: > To access the live stream do we need to sign up? I don't

Re: Old style "low level" Tasks with alternative deployment model(s)

hat I can pass, (or pull from an env variable and > add to the config) ? I am assuming it is my responsibility to ensure that > each instance is started with a unique container ID..? > > > > I am getting started on the above (Approach 2.), and looking closer at the > code so

Re: Streams Meetup at LinkedIn Sunnyvale 3/21/2018

Hey folks, We have an exciting line-up of tech-talks on use-cases for Apache Samza and Kafka! A friendly reminder to RSVP at https://www.meetup.com/Stream- Processing-Meetup-LinkedIn/events/248309045/ Best, Jagdish On Thu, Mar 1, 2018 at 3:16 PM, Jake Maes wrote: > Hi

Re: Old style "low level" Tasks with alternative deployment model(s)

s) > > Hey there! > > You are correct that this is focused on the higher-level API but doesn't > preclude using the lower-level API. I was at the same point you were not > long ago, in fact, and had a very productive conversation on the list: > you should look for "Question a

Re: Question about custom StreamJob/Factory

+Yi Hi Tom, Thank you for your feedback on Samza's architecture. Pluggability has been a differentiator that has enabled us to support a wide range of use-cases - from stand-alone deployments to managed services, from streaming to batch inputs and integrations with various systems from Kafka,

Re: No Metrics Emitted in new High-Level API

Can you turn on DEBUG logging for the MetricsSnapshotReporter class and re-attach the logs? I cannot observe any obvious red-flag from your logs. On Fri, Jan 26, 2018 at 7:13 PM, Srujun Thanmay Gupta wrote: > Hello Yi, > > The logs are less than 100KB each, I'm not sure

Re: No Metrics Emitted in new High-Level API

Hi Srujun, Metrics reporting should be supported independent of the deployment model. Can you please attach your container log file? Are you seeing log lines from MetricsSnapshotReporter:

Re: Welcome Xinyu as new Samza PMC!

Big Congrats Xinyu. Thanks for your continued contributions to all aspects of the project! On Wed, Jan 17, 2018 at 10:36 AM, Wei Song wrote: > Congrats, Xinyu! > > -- > Thanks > -Wei > > > On 1/17/18, 10:35 AM, "Navina Ramesh" wrote: > >

Re: Samza Freestyle Build

Thanks Chris for making these changes for us. Appreciate it. Best, Jagdish On Thu, Jan 11, 2018 at 5:05 PM, Chris Thistlethwaite wrote: > Greetings! > > I was working on cleaning up some builds on the Jenkins Windows nodes > and noticed your build failing. When I inspected

[RESULT][VOTE] Apache Samza 0.14 RC5

The vote of 0.14.0 has been more than 72 hours and we got +1 (binding) x 3 and +1 (non-binding) x 4. Samza 0.14 officially passed the VOTE! Thanks all for helping validate this release! -- Jagadish V, Graduate Student, Department of Computer Science, Stanford University

Re: [VOTE] Apache Samza 0.14.0 RC5

+1 (binding) Verified the RC. Ran *check-all.sh* and integration tests successfully on OS X. Thanks Xinyu, and everyone for driving Samza-0.14! On Thu, Dec 28, 2017 at 3:22 AM, Yi Pan wrote: > +1 (binding). > > Verified the signature and MD5 > Ran ./bin/check-all.sh on OSX

Re: [RESULT][VOTE] Apache Samza 0.14.0 RC3

Thank you everyone for the effort, and to Xinyu for shepherding Samza 0.14! This includes an exciting line-up of features including - SQL support, Kinesis integration and EventHub support. On Thu, Dec 21, 2017 at 11:03 AM, xinyu liu wrote: > The vote of 0.14.0 RC3 has

Re: [VOTE] Apache Samza 0.14.0 RC3

LGTM. I verified signatures and ran all tests on OsX. Additionally, *check-all.sh* succeeded. +1 (binding) On Thu, Dec 21, 2017 at 1:07 AM, Yi Pan wrote: > +1 binding > > Verified git tag and source signatures > > Ran check-all.sh on OSX > Ran integration tests from

Re: [DISCUSS] Samza 0.14.0 release

+1 from my side. Thank you Bharath for driving the release! On Mon, Nov 27, 2017 at 9:50 AM, Bharath Kumara Subramanian < codin.mart...@gmail.com> wrote: > Hi all, > > > > We have added couple of major features to master since 0.13.1 that warrants > a major release. > > Within LinkedIn, some

Re: Comparison between Samza and Kafka Streams

Thanks for the feedback Giridhar! We'll add a comparison with KStreams there as well. Roughly, the two are similar - The design of Samza certainly influenced what went into Kafka Streams. However, here are some key differences: - Support for non-Kafka source and sink natively: Samza has native

Re: Exit code 248 from YARN and InterruptedException

Hi Xiaochuan, Is there any exception in your custom SystemConsumer? Or, are you saying that your consumer is being interrupted externally? The entire log will be helpful. Thanks, Jagdish On Mon, Oct 30, 2017 at 9:14 AM, XiaoChuan Yu wrote: > Hi, > > I'm trying to debug

Re: Per task/topic checkpoint?

In Samza, the logical unit of processing (and hence, checkpointing) is a task. Hence, you cannot selectively checkpoint SSPs within a task. However, you can configure how you group your SSPs into tasks by choosing a Grouper. If you want to control checkpointing at the granularity of an SSP, then

Re: permission to create a SEP page

Hi Aditya, We usually create a SEP page first, and allow contributors to edit it later (instead of provisioning create permissions). I just created https://cwiki.apache.org/confluence/display/SAMZA/SEP-9+Add+a+Kinesis+SystemConsumer+and+SystemProducer Can you please let me know if you are able

Re: [VOTE] SEP-8: Add in-memory system consumer & producer

LGTM , +1 on the overall design. This will drastically improve testing of Samza applications! -- Jagdish On Thu, Sep 14, 2017 at 1:23 AM, Yi Pan wrote: > Hi, Bharath, > > Overall looks good! I have the following comments: > > i) Question on the Type of IME + data

Re: Deploying Samza Jobs Using S3 and YARN on AWS

> > . > However, due to a dependency conflict on the AWS SDK between our code and > "hadoop-aws", we can't actually include it. > We are now planning to make use of HTTP FS instead. > > On Fri, Sep 15, 2017 at 2:45 PM Jagadish Venkatraman < > jagadish1...@gm

Re: Connection timed out error while installing "Hello Samza"

Waiting for zookeeper to start... > ^C > [root@csadatlak-wc-a1t hello-samza]# bin/grid start yarn > EXECUTING: start yarn > starting resourcemanager, logging to /home/sanant003c/hello-samza/ > deploy/yarn/logs/yarn-root-resourcemanager-csadatlak-wc- > a1t.sys.comcast.net.out > Waiting f

Re: Samza Job Slow to Restart

Hi Xiaochuan, >> What does that loop do exactly? Most of what the run-loop does is documented in https://samza.apache.org/learn/documentation/0.9/container/event-loop.html >> We are running into a problem where it seems to take a very long time to restart a Samza job. Some follow-up questions,

Re: Connection timed out error while installing "Hello Samza"

Hi Anantharaman, It's likely that your server blocks the GIT protocol. If so, I'd recommend cloning via HTTPS instead of git. *First, determine if the git protocol is blocked on the server as follows:* Please run: # connects to github default port ➜ ~ nc github.com 9418 < /dev/null; echo

Re: Deploying Samza Jobs Using S3 and YARN on AWS

Thank you Xiaochuan for your question! You should ensure that *every machine in your cluster* has the S3 jar file in its YARN class-path. From your error, it looks like the machine you are running on does not have the JAR file corresponding to *S3AFileSystem*. >> Whats the right way to set this

Re: Samza 0.13.1 officially released

Thank you Fred for shepherding this release and to everyone for your contributions! On Fri, Aug 25, 2017 at 5:40 PM Fred Haifeng Ji wrote: > Hi, all, > > I am pleased to let you know that we have officially released Samza 0.13.1! > > Huge thanks to everyone for working on

Fwd: Question about stop in SystemProducer

d everything works fine. Thanks! Qi Shu > 在 2017年8月24日，15:06，舒琦 <sh...@eefung.com> 写道： > > Yes, in the same thread. > > File list is not empty, because after samza job shutdown, all files in writing are not in complete state. > > ———— > Qi Shu &g

Re: Question about stop in SystemProducer

File(statusFile.getFilePath()); > LOGGER.info("Successfully close file: {}", > statusFile.getFilePath()); > } > > files.clear(); > } > > } > > > > Qi Shu > > > 在 2017年8月24日，14:21，Jagadish Venkatraman <jagadish1...@g

Re: Question about stop in SystemProducer

Hi Qi, >> the stop method in SystemProducer is called, but the close files operation(may need some time, there may be cache data to be flushed) in stop method is not executed completely Are you seeing the *close()* method hang? SystemProducer.*close* is a synchronous operation, and will block

Re: [VOTE] Apache Samza 0.13.1 RC0

Ran check-all.sh, and it succeeded! +1 (non binding) On Mon, Aug 21, 2017 at 4:34 PM, xinyu liu wrote: > Built the src, and ran the tests using check-all.sh. Most of the tests ran > fine. There was an transient test failure ( >

Re: Custom ordering when using async

Hi XiaoChuan, Are you setting task.max.concurrency > 1 that allows multiple messages in-flight? (The "keyed executor pool" is only meaningful with that scenario) Also, Have you tried increasing your *job.container.thread.pool.size *config and setting it to the number of tasks in the container?

  1   2   3   4   >