Re: "send to" ordering is inconsistent

ble in the Samza 1.2 release: > https://samza.apache.org/releases/1.2.0 > > Thanks, > Prateek > > On Tue, Mar 12, 2019 at 9:11 AM Tom Davis wrote: > > > > Bummer! Yes, that works. The phase from "pending" to "running" is a > > nice-to-have at the m

Re: Error handling

k On Mon, Mar 25, 2019 at 7:51 AM Tom Davis wrote: I am using Samza 1.0, yes. The stacktrace is: > 19:24:49.326 [Samza StreamProcessor Container Thread-0] ERROR > org.apache.samza.processor.StreamProcessor - Container: > org.apache.samza.container.SamzaContainer@3e923d9e faile

Re: Error handling

, though, so we just silently stop processing events. How do you determine the number of expected containers? Or are you speaking of containers in terms of YARN and not Samza processors? - Prateek On Fri, Mar 15, 2019 at 7:26 AM Tom Davis wrote: I'm using the LocalApplicationRunner and had added

Error handling

I'm using the LocalApplicationRunner and had added a liveness check around the `status` method. The app is running in Kubernetes so, in theory, it could be restarted if exceptions happened during processing. However, it seems that "container failure" is divorced from "app failure" because the app

Re: "send to" ordering is inconsistent

o include SAMZA-2116 in the upcoming 1.1 release due to time constraints. It'll have to go in to the 1.2 release, which will tentatively be in June. Does that still work for you? Thanks, Prateek On Thu, Feb 28, 2019 at 2:16 PM Tom Davis wrote: Thanks, Prateek! Yes, the workaround will be fin

Re: "send to" ordering is inconsistent

nclude this in the next Samza release, which should be mid to late next month. In the mean time, the workaround would be to keep all of this functionality in a sink function. Does this work for you? Thanks, Prateek On Wed, Feb 27, 2019 at 2:54 PM Tom Davis wrote: Prateek Maheshwari writes: > Hi Tom

Re: "send to" ordering is inconsistent

Since both sendTo and sink are terminal operators (void return type), I don't think you'll be able to easily get around this. Let me discuss this with the team and get back to you with a workaround / fix. Thanks a lot! <3 Thanks, Prateek On Tue, Feb 26, 2019 at 7:08 PM Tom Davis wrot

Re: "send to" ordering is inconsistent

nents? Cheers, Tom Tom Davis writes: Hey folks! We have noticed some inconsistencies in message ordering when running a StreamApplication that calls two separate `map` functions over an input and sends results to the same output. I have attached my Execution Plan, but the gist is that the first

"send to" ordering is inconsistent

Hey folks! We have noticed some inconsistencies in message ordering when running a StreamApplication that calls two separate `map` functions over an input and sends results to the same output. I have attached my Execution Plan, but the gist is that the first `map` function marks a thing as

Re: Looking for YARN Alternative

I'll chime in to say I am running the standalone mode successfully in Kubernetes. The ZK coordinator is very useful in this context as you can partition a topic for max *desired* parallelism without continually running that many nodes. You could also use the "operator" pattern in Kube to create a

Re: InMemorySystemDescriptor ignores serde

d I create the ticket > https://issues.apache.org/jira/browse/SAMZA-2075 to track this. We > will see> whether we can get it there in the next release. > > Thanks, > Xinyu > > > > > > On Thu, Jan 17, 2019 at 6:11 AM Tom Davis > wrote:> >> Hey

Re: InMemorySystemDescriptor ignores serde

? Thanks Sanil On Tue, 8 Jan 2019 at 17:28, Tom Davis wrote: I am in the process of updating a project to 1.0 and spent today debugging a rather odd test failure. When using input/output streams with IntegerSerde, things worked fine -- however, using LongSerde, every message value was 0! I

InMemorySystemDescriptor ignores serde

I am in the process of updating a project to 1.0 and spent today debugging a rather odd test failure. When using input/output streams with IntegerSerde, things worked fine -- however, using LongSerde, every message value was 0! I eventually found that InMemorySystemDescriptor#getInputDescriptor

Re: Alerting and Monitoring Samza Checkpointing?

Have you looked into KafkaSystemConsumerMetrics? Is the meaning of "lag" there different from what you mean? Jeremiah Adams writes: We are replacing a node.js app that consumed topics on a Kafka cluster with Samza jobs. We use kafka-offsets to trigger alerts based on message lag. e.g.,

Re: Samza 1.0 Release

Hey Rayman, I can't speak for anybody else but I'd love a Samza 1.0 release! Ideally it would coincide with Kafka 1.0 support but I suppose major features and a ton of bug fixes will have to do ;) rayman preet writes: Hi all, We have added a number of major features and changes to

ClassNotFoundException when loading classes via Util#getObj

I am having one hell of a head-scratching issue here. I am trying to write a test that uses LocalApplicationRunner to run an app configured to use the InMemorySystemFactory. But when I run my test, org.apache.samza.system.inmemory.InMemorySystemFactory is not found! Theweird thing is, I see the

Re: [Discuss] Samza 0.14.1 release

s much for the feedback! >>> >>>>> Any timeline on getting Kafka 0.11 upgrade in? >>> >>> We have historically been doing Kafka upgrades on major releases >>> (instead>>> of >>> minor ones). So, it will likely be in 0.15.0. I'll provide

Re: [Discuss] Samza 0.14.1 release

These all look like excellent changes to me, I'd be happy to have more stability in Standalone! Any timeline on getting Kafka 0.11 upgrade in? It seems to have been done a while, but waiting for 0.15 I presume. Best, Tom On Mon, Apr 30, 2018, at 3:10 PM, Xinyu Liu wrote: > Hi, All, > > We have

Re: Old style "low level" Tasks with alternative deployment model(s)

r -Original Message- From: Jagadish Venkatraman [mailto:jagadish1...@gmail.com] Sent: Tuesday, March 20, 2018 11:36 To: Tom Davis <t...@recursivedream.com> Cc: Prateek Maheshwari <prateek...@gmail.com>; dev@samza.apache.org; yi...@linkedin.com; Yi Pan <nickpa...@gmail.com>

Re: Old style "low level" Tasks with alternative deployment model(s)

fka High-level consumer comes with other nice tools for > >> monitoring offsets, lag, etc > > Samza exposes<https://github.com/apache/samza/blob/master/ > samza-kafka/src/main/scala/org/apache/samza/system/kafka/ > KafkaSystemConsumerMetrics.scala> the b

Re: Stream Processing Meetup @LinkedIn tomorrow (3/21)

To access the live stream do we need to sign up? I don't want to take a spot from someone able to be physically present. Yi Pan writes: Hi, all, Just a reminder that we will host another stream processing meetup @LinkedIn tomorrow at 6pm:

Re: Old style "low level" Tasks with alternative deployment model(s)

Hey there! You are correct that this is focused on the higher-level API but doesn't preclude using the lower-level API. I was at the same point you were not long ago, in fact, and had a very productive conversation on the list: you should look for "Question about custom StreamJob/Factory" in the

Re: Question about custom StreamJob/Factory

know should you need any further help. Here's an example to get you started: https://github.com/apache/samza-hello-samza/tree/master/src/main/java/samza/examples/wikipedia/application Best, Jagdish On Sat, Jan 27, 2018 at 8:54 AM, Tom Davis <t...@recursivedream.com> wrote: Hi there! Fi

Re: Question about custom StreamJob/Factory

r/src/main/java/samza/examples/wikipedia/application Best, Jagdish On Sat, Jan 27, 2018 at 8:54 AM, Tom Davis <t...@recursivedream.com> wrote: Hi there! First off, thanks for the continued work on Samza -- I looked into many DC/stream processors and Samza was a real standout with its smart a

Question about custom StreamJob/Factory

Hi there! First off, thanks for the continued work on Samza -- I looked into many DC/stream processors and Samza was a real standout with its smart architecture and pluggable design. I'm working on a custom StreamJob/Factory for running Samza jobs on Kubernetes. Those two classes are pretty