Re: Samza yarn job - cannot bind to local host

ory > > > > > > # Systems > > > > systems.kafka.samza.factory=org.apache.samza.system.kafka.Ka > > fkaSystemFactory > > > > systems.kafka.samza.msg.serde=json > > > > > > systems.kafka.consumer.zookeeper.connect=host1:2181,

Re: [DISCUSS] [VOTE] Apache Samza 0.10.1 RC0

It has been more than 5 days and we have got 3 +1 (binding) and 5 +1 (non-binding) already. Can we conclude this vote? Thanks! On Tue, Aug 2, 2016 at 1:10 PM, Boris Shkolnik wrote: > +1 (non-binding). > > Boris. > > On Mon, Aug 1, 2016 at 11:39 AM, Navina Ramesh > > wrote: > > > Hey all, > > >

Re: [DISCUSS] [VOTE] Apache Samza 0.10.1 RC0

Navina > > > On Sun, Aug 7, 2016 at 7:53 PM, Yi Pan wrote: > > > It has been more than 5 days and we have got 3 +1 (binding) and 5 +1 > > (non-binding) already. Can we conclude this vote? Thanks! > > > > On Tue, Aug 2, 2016 at 1:10 PM, Boris Shkolnik wrote: &g

Upcoming Streams Meetup @LinkedIn

Hi, all, I am pleased to announce that LinkedIn invites you to attend a Streams Processing meetup on Tuesday, August 23 at our Mountain View campus. There will be speakers from LinkedIn, Confluent, and TripAdvisor. Plea

Re: Question on changelog partition mapping

Hi, Tommy, Which version of Samza are you using? Since 0.10, the changelog partition mapping has been moved to the coordinator stream, not in the checkpoint topic any more. That said, I want to ask a few more questions to understand what you referred to as "non-deterministic" behavior. So, betwee

Re: Question on changelog partition mapping

ch time it runs, even if the number of tasks is the > same. Does that make sense? The code has changed some since 0.9.1 but > seems to have the same issue even in 0.10.1. > > -Tommy > > On 08/11/2016 06:12 PM, Yi Pan wrote: > > Hi, Tommy, > > Which version of Samza are y

Re: Samza container hang on exception

Hi, Sining, This is a known bug that is fixed in 0.10.1 (SAMZA-911). Please try to upgrade to 0.10.1. Thanks! -Yi On Sun, Aug 21, 2016 at 5:55 AM, 李斯宁 wrote: > I have tried restart every kafka server. The container did not recover. > > log have something below: > > 2016-08-21 20:08:21 [WARN

Re: kafka dependency version

Hi, Gaurav, There is already an effort going on for this one: SAMZA-855 . It would be good if you can try out the patch. Thanks! -Yi On Mon, Aug 22, 2016 at 1:11 AM, Gaurav Agarwal wrote: > My initial attempt to build against kafka 0.9.0 or 0.1

Re: [DISCUSS] Samza 0.11.0 release

Hi, Nicolas, Could you explain to me why Samza is blocking you from upgrading your Kafka brokers to 0.10? At LinkedIn, we are running Samza 0.10 w/ Kafka 0.10 brokers. This is a valid combination since Kafka 0.10 brokers should be backward compatible w/ 0.8.2 clients (which is the version Samza us

Re: [DISCUSS] Samza 0.11.0 release

a timestamp field. Kafka 0.10.0 is backwards > compatible with 0.8.x clients but we are concerned about the performance > impact, see > http://kafka.apache.org/documentation.html#upgrade_10_performance_impact. > > Cheers, > > Nicolas > > On 25 Aug 2016 6:20 a.m., "Yi

Re: Question on changelog partition mapping

you consider accepting a PR that makes this > change to the standard groupers? It's just strange that the generated > partition mappings can vary like this, even for identical inputs. > > -Tommy > > > On 08/16/2016 03:04 PM, Yi Pan wrote: > > Hi, Tommy, > >

Re: Samza Mesos

Hi, Sriram, The story behind delaying the integration of SAMZA-375 is that there are tons of repeated code in SamzaAppMaster that exist in both samza-yarn and Mesos. W/o the change we recently made in SAMZA-680, we are going to copy the SamzaAppMaster code for every distributed execution system th

Re: Samza Mesos

16 at 10:45 AM, Sriram Ramachandrasekaran < > > sri.ram...@gmail.com> wrote: > > > > > Yi, > > > That's a good amount of history to know. I will take a look at 680 and > > then > > > see if I can implement something as well. If there's so

Re: Samza container hang on exception

ading does not solve the problem. All task hang in > today's > >> > "rush hour". > >> > I attached log and jstack. > >> > > >> > The SAMZA-911 want to fix by stopping the process if failed too much > >> > times. But the process

Re: checkpoint on flush of system producer

Hi, Jarrad, Yes! You have found your answer! Looking forward to your implementation of SystemProducer. Just curious, what's the target output system that you are writing to? -Yi On Tue, Sep 6, 2016 at 9:01 AM, Jarrad, Ken wrote: > I think I have discovered the answer to my question. Hopefully

Re: Samza container hang on exception

Fri, Sep 2, 2016 at 2:17 AM, 李斯宁 wrote: > yes, upgraded to 0.10.1 > > jstack: > https://drive.google.com/open?id=0B19olQZ1dUO8VjltQmtxLTJ4SVdFZ > WhYWHZ3Y2hMOVhCMWNn > task log: > https://drive.google.com/open?id=0B19olQZ1dUO8eVRLWmJCVl9nRlg2U > UM4c21udUViWW8tSUVV > >

Re: SIGSEGV in RocksDB when killing jobs

Hi, Tommy, Thanks for reporting this. Definitely we can be more defensive in coding here. I just wonder what's the specific reason for you to call RocksDB store close() explicitly? As you see that SamzaContainer#shutdownStores already calling flush() and close() automatically. Does it work for you

Re: Issue with consuming non-existent topics in 0.10.1

Hi, Tommy, Could you open a JIRA for this one? Also, could you include the Kafka broker version in this test? Thanks! -Yi On Wed, Sep 14, 2016 at 6:06 AM, Tommy Becker wrote: > We are testing an upgrade to 0.10.1 from 0.9.1 and noticed a regression. > When starting a stream job that consumes

Re: [Discuss] Moving Samza to Java 1.8 source compatibility.

hould we wait until we are at 1.0. > > > > > > I think the users in the community need to provide feedback so we can > > make > > > progress accordingly. > > > > > > Thanks! > > > Navina > > > > > > > > > >

Re: [DISCUSS] [VOTE] Apache Samza 0.11.0 RC0

Hi, guys, I found a few issues w/ the current release candidate: - integration test (./bin/integration-tests.sh) is broken due to reference to "0.11.0-SNAPSHOT" version of tgz files in the build directory. samza-test/src/main/python/configs/tests.json needs to be updated w/ official 0.11.0 version

Re: [VOTE] Apache Samza 0.11.0 RC2

Build, validated MD5, test w/ integration tests and passed. Thanks! +1 (binding) On Mon, Oct 10, 2016 at 4:07 PM, xinyu liu wrote: > Hey all, > > This is a call for a vote on a release of Apache Samza 0.11.0. Thanks to > everyone who has contributed to this release. We are very glad to see some

Re: [DISCUSS] Samza 0.12.0 release

lgtm, +1 On Fri, Dec 23, 2016 at 10:44 AM, santhosh venkat < santhoshvenkat1...@gmail.com> wrote: > Hi All, > > There have been quite a lot of new features added to master since 0.11 > release to warrant a new major release. At LinkedIn, we've done functional > and performance testing against mas

Re: [DISCUSS] Samza 0.12.0 release

from Jetty 8 > > Thanks, > > Fred > > On Fri, Dec 23, 2016 at 2:38 PM, Yi Pan wrote: > > > lgtm, +1 > > > > On Fri, Dec 23, 2016 at 10:44 AM, santhosh venkat < > > santhoshvenkat1...@gmail.com> wrote: > > > > > Hi All, > > > > &

[REPORT] Samza - Jan 2017

## Description: - Apache Samza is a stream processing framework built on top of Apache Hadoop YARN and Apache Kafka. ## Issues: - there are no issues requiring board attention at this time ## Activity: - Streaming and SQL analytics on Samza in QCon SF'16 - Streaming meetup at LinkedIn in No

Re: Materialized views as in "Turning the database inside out with Apache Samza"

Hi, Yevhenii, Could you clarify how you want to consume the "materialized views"? Are you planning to access it just in realtime analytic pipeline (i.e. Samza)? Or are you planning to serve it s.t. it can be accessed by some online application outside Samza? If it is the later case, usually the "m

Re: How to gracefully stop samza job

Hi, Qi, Sorry to reply late. I am curious on your comment that the close and stop methods are not called. When user initiated a kill request, the graceful shutdown sequence is triggered by the shutdown hook added to SamzaContainer. The shutdown sequence is the following in the code: {code} in

Re: How to gracefully stop samza job

the log of am > can be seen. > > > > > ShuQi > > 在 2017年1月16日，10:39，Liu Bo 写道： > > Hi, > > *container log will be removed automatically,* > > you can turn on yarn log aggregation, so that terminated yarn jobs' log > will be dumped to HDFS >

Re: How to gracefully stop samza job

地址：长沙市岳麓区文轩路27号麓谷企业广场A4栋1单元6F > 网址：http://www.eefung.com > 微博：http://weibo.com/eefung > 邮编：410013 > 电话：400-677-0986 > 传真：0731-88519609 > > > 在 2017年1月17日，17:18，Yi Pan 写道： > > > > Hi, Qi, > > > > In your log, the log line stops at "closing

Join us in the first Streams meetup @LinkedIn in 2017!

Hi, all, We have an exciting agenda setup in the Streams meetup @LinkedIn, to welcome the year 2017! It will be on Thursday, February 16, 2017, at 6pm in LinkedIn office in Sunnyvale. There will be talks from Uber, LinkedIn, and Optimizly on Kafka and Samza, talking about Kafka on SSD, async proce

Re: [VOTE] Apache Samza 0.12.0 RC2

Ran check-all and integration tests on Mac. Passed and verified the pgp key. P.S. I do see a non-consistent test hanging issue on my Mac in TestStreamProcessor test. Drilled in a bit and found out that the Kafka broker is not started serving correctly during the job initialization. It passed in Li

Streams meetup @LinkedIn on 2/16

Hi, all, Just a kind reminder that our first meetup in 2017 will be held tomorrow. Details here: https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/237171557/ Looking forward to seeing you all! -Yi

Re: [VOTE] SEP-1: Semantics of ProcessorId in Samza

@Navina, Sorry to chime in late. One question: 1. Why is it in JobCoordinator, and why not in StreamProcessor class? Because JobCoordinator provides coordination service across many processors, an interface getProcessorId() in JobCoordinator is confusing regarding to which processorId we are getti

Re: Steps to Upgrading Samza (0.9 to 0.12)

Hi, Thomas, Sorry to hear that you were hit by the removal of migration in Samza 0.11. The reason we removed it is following a deprecate-removal policy in two versions. We are not aware that people still using 0.9 after we released 0.11 and were not expecting a direct upgrade from 0.9 to 0.12. Doc

Re: [VOTE] SEP-1: Semantics of ProcessorId in Samza

ution environment. > > > > Let me know if you have a different perspective on this. > > > > Cheers! > > Navina > > > > On Thu, Mar 30, 2017 at 9:42 AM, Yi Pan wrote: > > > > > @Navina, > > > > > > Sorry to chime i

[DISCUSS] SEP-2: ApplicationRunner Design

Hi, everyone, In light of new features such as fluent API and standalone that introduce new deployment / application launch models in Samza, I created a new SEP-2 to address the new use cases. SEP-2 link: https://cwiki.apache.org/confluence/display/SAMZA/SEP-2%3A+ApplicationRunner+Design Please t

[REPORT] Apache Samza - April 2017

## Description: Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data sources, including real-time messaging system (e.g. Kafka) and distributed file systems (e.g. HDFS). ## Issues: - there are no issues requiring board attent

Re: [VOTE] Samza Logo

Really? The one with the maze on the left currently is top one? I can't relate to that either. My favorite was the logo w/ Taiji symbol. Since that did not make the top 4, I am voting for the red bufferfly one, same as Navina. -Yi On Fri, Apr 14, 2017 at 3:33 PM, Navina Ramesh wrote: > I prefer

Re: [DISCUSS] SEP-2: ApplicationRunner Design

Made some updates to clarify the role and functions of RuntimeEnvironment in SEP-2. On Fri, Apr 14, 2017 at 9:30 AM, Yi Pan wrote: > Hi, everyone, > > In light of new features such as fluent API and standalone that introduce > new deployment / application launch models in Samza, I c

Re: [VOTE] Apache Samza 0.13.0 RC0

Hi, all, Based on the conversation above, can we officially cancel this vote? Thanks! -Yi On Mon, May 15, 2017 at 9:31 AM, Ignacio Solis wrote: > Thanks! > > On Mon, May 15, 2017 at 8:00 AM, Navina Ramesh > wrote: > > I will try to get the patch out today. Work doesn't look trivial. I am on

Re: A question about checkpointing

Hi, Qi, This would depend on the following two factors: # whether the send() is async or sync # how do you handle the send failure If the send() is sync, you will always receive an exception in your process() method when MessageCollector.send() is invoked. Hence, if your code does not handle the

Re: [DISCUSS] SEP-6: Support Watermark Across Intermediate Streams for Batch Processing

Hi, Xinyu, Thanks for the proposal. I took a quick pass and had the following questions/comments: - message shuffling ==> data shuffling??? - the proposal is for all types of control messages, not just for end-of-stream, right? Better to define the scope and layout the comment requirements of co

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

Hi, Don, Thanks for the detailed design doc for a long-waited feature in Samza! Really appreciate it! I did a quick pass and have the following comments: - minor: "limit the maximum size of partition" ==> "limit the maximum size of each partition" - "However, Samza currently is not able to handle

Re: [DISCUSS] SEP-6: Support Watermark Across Intermediate Streams for Batch Processing

; > > InputWatermark(stage) = min { OutputWatermark(stage') for stage' in > > Upstream(stage) } > > OutputWatermark(stage) = min { InputWatermark(stage), OldestWork(stage) } > > > > OldestWork(stage) is the oldest message that has been received by the > sta

Re: [DISCUSS] SEP-2: ApplicationRunner Design

gt; > of > > > > >> use > > > > >> >> > when launched from main(). I'd prefer making it consitently > > > > >> non-blocking > > > > >> >> > instead, esp. since in

Re: [VOTE] Apache Samza 0.13.0 RC6

+1 (binding) build and ran all local integration tests on Linux. On Tue, Jun 6, 2017 at 4:01 PM, Boris S wrote: > +1 (non-binding) > build and tested on Linux (with python 2.7; 2.4 and 3.5 - didn't work) > > On Tue, Jun 6, 2017 at 2:49 PM, Jacob Maes wrote: > > > +1 (non-binding) > > > > Built

Samza 0.13.0 officially released

Hi, all, We have finally announced the official release of Samza 0.13.0! Thanks all for push this out of the door! Check it out for high-level API and standalone deployment! https://blogs.apache.org/samza/ Cheers! -Yi

Re: [DISCUSS] SEP-2: ApplicationRunner Design

/nickpan47/samza/pull/2/files Please feel free to comment and provide feedbacks! Thanks! -Yi On Tue, Jun 6, 2017 at 11:16 AM, Yi Pan wrote: > Hi, all, > > Thanks for all the inputs! Finally I got some time to go through the > discussion thread and digest most of the points made above

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

Hi, Dong and everyone, Thanks for the detailed discussion on SEP-5! Really appreciate the thorough consideration on this issue. I also noticed that Dong has updated the SEP-5 wiki to clarify: 1) SEP-5 provides a solution to retain the same number of task/state w/o re-partitioning (as illustrated i

Re: [DISCUSS] SEP-5: Enable partition expansion of input streams

trivial change to the wiki to illustrate the use of new > config and interface for user to specify new-partition-to-old-partition > mapping. Can you please help review it? > > Thanks, > Dong > > > On Thu, Jun 22, 2017 at 2:25 AM, Yi Pan wrote: > > > Hi, Dong and everyon

[REPORT] Samza - July 2017

## Description: - Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data sources, including real-time messaging system (e.g. Kafka) and distributed file systems (e.g. HDFS). ## Issues: - there are no issues requiring board attentio

Re: Using Amazon Kinesis as Samza Backend

Hi, Christopher, I am glad that you are interested in the Kinesis integration to Samza! Yes, Renato's implementation is so far the most complete one. However, I think that it might not handle dynamic shard split/merge cases. Aditya Toomula in our team is currently working on a complete proposal to

Re: [Discuss] Samza 0.13.1 release

+1 for the list! Let's proceed! On Fri, Aug 11, 2017 at 6:13 PM, Ignacio Solis wrote: > +1 > > On Fri, Aug 11, 2017 at 3:52 PM, Jacob Maes wrote: > > Looks good! > > > > +1 > > > > On Thu, Aug 10, 2017 at 6:53 PM, Jagadish Venkatraman < > > jagadish1...@gmail.com> wrote: > > > >> +1 for the rel

Re: [VOTE] Apache Samza 0.13.1 RC0

Downloaded the source, compiled and ran the integration tests. All passed. +1 (binding) w/ the following minor comments: # Please make a note in the release note that this version requires JDK 1.8.0.112+ (I have test w/ JDK 1.8.0.121) # Please make sure that we publish artifacts compiled w/ Scala

Upcoming Streams meetup @ LinkedIn

Hi, all, We have the upcoming Streams meetup @LinkedIn planned on 9/12. We will talk about realtime search index and Redfin notification using Samza and Kafka controller updates: come to learn the details in our next streams meet up @ LinkedIn! https://www.meetup.com/Stream-Processing-Meetup-Linke

Re: [RESULT][VOTE] Apache Samza 0.13.1 RC0

Thanks, Fred, for driving this release! Thanks for everyone voted! Cheers! -Yi On Thu, Aug 24, 2017 at 2:44 PM, Fred Haifeng Ji wrote: > The vote of 0.13.1 RC0 has been more than 72 hours and we got +1 (binding) > x 3 and +1 (non-binding) x 3. > > Samza 0.13.1 officially passed the VOTE! > > T

Re: Custom ordering when using async

Hi, Xiaochuan, Please refer to the document here: https://samza.apache.org/learn/tutorials/0.13/samza-async-user-guide.html As stated, by default, when task.max.concurrency=1, in-order processing within a task is guaranteed, no matter whether you implement StreamTask or AsyncStreamTask. The engin

Re: Kafka 0.10.2 compatibility with Samza 0.13.1

Hi, Xiaochuan, I am relaying your question to our in-house Kafka expert to confirm. But as far as I can tell from the description, the main breaking changes are in the KStreams API. Samza only uses the standard Kafka consumer/producer APIs now and should not be affected. I would recommend to give

Re: Kafka 0.10.2 compatibility with Samza 0.13.1

if we can be further assistance here. Thanks! -Yi On Thu, Aug 31, 2017 at 4:14 PM, Yi Pan wrote: > Hi, Xiaochuan, > > I am relaying your question to our in-house Kafka expert to confirm. But > as far as I can tell from the description, the main breaking changes are in > the KStr

Reminder for Streams meetup @LinkedIn

Hi, all, Just a reminder on the coming meetup on 9/12: Realtime search index and Redfin notification using Samza in our next streams meet up @ LinkedIn! https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/242656767/ Thanks! -Yi

Re: Connection timed out error while installing "Hello Samza"

Hi, Anantharaman, Could you post your zookeeper startup logs here? On Tue, Sep 12, 2017 at 10:21 AM, Anantharaman, Srinatha (Contractor) < srinatha_ananthara...@comcast.com> wrote: > It hangs while bringing up the service > > [root@codehdplak-po-r19p hello-samza]# pwd > /app/home/eventsvc/samza-

Re: Connection timed out error while installing "Hello Samza"

t:-1):: > PrepRequestProcessor@617] - Got user-level KeeperException when > processing sessionid:0x15e770d6192 type:create cxid:0x3f zxid:0x18 > txntype:-1 reqpath:n/a Error Path:/brokers Error:KeeperErrorCode = > NodeExists for /brokers > 2017-09-12 17:18:07,290 [myid:] - INFO [ProcessThread

Re: [VOTE] SEP-8: Add in-memory system consumer & producer

Hi, Bharath, Overall looks good! I have the following comments: i) Question on the Type of IME + data partition: How do we enforce that user adds IME w/ the expected partition id to the corresponding sub-collection? ii) In the architecture graph, what's the difference between SSP queues and D

Re: Kafka 0.10.2 compatibility with Samza 0.13.1

n AWS EC2 instances running RedHat 7. > We also found out that running against CDH YARN required various dependency > overrides such as the ones in Hello Samza's "cdh5" build profile > <https://github.com/apache/samza-hello-samza/blob/master/pom.xml#L282-L297 > >. > >

Re: Connection timed out error while installing "Hello Samza"

gt; > > > mkdir -p deploy/samza > > > > tar -xvf ./target/hello-samza-*-SNAPSHOT-dist.tar.gz -C deploy/samza > > > > > > NOTE : BTW From the above steps I could not execute " gradle -b > bootstrap.gradle" since that command does not exists > > &g

Re: Connection timed out error while installing "Hello Samza"

:21 AM, Yi Pan wrote: > Hi, Ananarath, > > It is very strange that you are seeing this timeout exception that we do > not see. I am trying to follow the exact steps you did to see whether there > is anything broken. I will update you this afternoon. > > Meanwhile, could yo

Re: Connection timed out error while installing "Hello Samza"

ct issues with your local laptop setup - it > could either related resolving "localhost" or firewall that prevents > communication between ports. > > > Navina > > > From: Yi Pan > Sent: Thursday, September 14, 2017 11:37:46 AM > To:

Re: Connection timed out error while installing "Hello Samza"

P.S., the attachment you post w/ the previous email does not show up (Apache mailing list disallow attachments). If would be better if you send the command line outputs in pure text or html format. -Yi On Thu, Sep 14, 2017 at 3:34 PM, Yi Pan wrote: > Hi, Anantharaman, > > A lot of

Re: Connection timed out error while installing "Hello Samza"

d. Run: bin/grid install yarn > > > EXECUTING: stop zookeeper > > > Zookeeper is not installed. Run: bin/grid install zookeeper > > > EXECUTING: install samza > > > ~/.samza/download /home/sanant003c/hello-samza Cloning into 'samza'... > > > fatal: unable

Re: Connection timed out error while installing "Hello Samza"

ob Package" { was > referring to .7 and this step is not part of it) > > ./gradlew publishToMavenLocal > > When I run the above command I am getting below error > > [root@csadatlak-wc-a1t hello-samza]# ./gradlew publishToMavenLocal > > FAILURE: Build failed with an exception

Re: Historical container logs in YARN UI

Hi, XiaoChuan, Our SRE team have been using timeline server in YARN at LinkedIn to get the historical container logs in our admin dashboard. @Jon Bringburst, can you share some experience regarding to how to configure timeline server in YARN? Thanks a lot! -Yi On Sat, Sep 30, 2017 at 1:08 PM, X

Re: [VOTE] SEP-8: Add in-memory system consumer & producer

committed). -Yi On Tue, Sep 26, 2017 at 10:58 AM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > LGTM , +1 on the overall design. This will drastically improve testing of > Samza applications! > > -- > Jagdish > > On Thu, Sep 14, 2017 at 1:23 AM, Yi Pa

Re: buffering records for join

Hi, Jef, I would recommend that you use a KV store to buffer the messages for join. The logic would be more predictable and state is also durable. In StreamTask.process(), you can do some pseudo code like below: {code} public void process(IncomingMessageEnvelope msg, MessageCollector collector, Ta

[REPORT] Apache Samza - Oct 2017

## Description: Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data sources, including real-time messaging system (e.g. Kafka) and distributed file systems (e.g. HDFS). ## Issues: - there are no issues requiring board attention a

Re: buffering records for join

ed, so many chained joins might be very inefficient. If > so we would prefer to use the "classic" API. > > -Jef > > On Wed, Oct 11, 2017 at 7:37 PM, Yi Pan wrote: > > > Hi, Jef, > > > > I would recommend that you use a KV store to buffer the messages f

Re: buffering records for join

12, 2017 at 11:40 AM, Yi Pan wrote: > Hi, Jef, > > What I suggest is exactly in-process, in-memory KV store. Samza has two > types of such built-in KVstores: in-memory and RocksDB. Both can be backed > by a changelog topic in Kafka as the failure recovery mechanism (i.e. if a > c

Re: Samza Job Slow to Restart

Awesome that you have figured it out! Just a general notice: any logcompact topic used in Samza may see this slow-down if the Kafka log cleaner thread dies, which include checkpoint, coordinator stream, and changelog topics. Best! -Yi On Thu, Oct 19, 2017 at 12:14 PM, XiaoChuan Yu wrote: > Hi,

Re: Samza 13.1 and Kafka 1.0

Hi, XiaoChuan, Sorry to reply late. Samza 13.1 is built w/ Kafka 0.10.1.1. We have not test against Kafka broker 1.0 yet. We usually release Samza with the Kafka version that is running as a stable one in LinkedIn. If you have specific feature requirements that is only in Kafka 1.0, please let us

Dec Streams meetup @LinkedIn

Hi, all, We have finalized the date for Stream Processing meetup @LinkedIn. It will happen on Dec 4th: https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/244889719/ There will be important updates on Samza-SQL, Slack on Samza, and Kafka @Uber. Come join us! -Yi

Re: Running apache samza with Kafka Client 1.0 - JIRA - SAMZA - 1418

Hi, Debraj, First, yes, this is the correct forum for user questions as well. Second, we are actively working on a patch to bring in Kafka 0.11 client in Samza. It would require a lot of testing on Samza side to make sure: 1) back-compat w/ older version of brokers; 2) performance test. We alread

Re: Running apache samza with Kafka Client 1.0 - JIRA - SAMZA - 1418

to higher version of Kafka broker, including 1.0, if the concern is on client/broker compatibility. -Yi On Wed, Nov 22, 2017 at 9:52 AM, Yi Pan wrote: > Hi, Debraj, > > First, yes, this is the correct forum for user questions as well. > > Second, we are actively working on a pa

Re: Running apache samza with Kafka Client 1.0 - JIRA - SAMZA - 1418

o 1.0 so was trying to have the same > client version. > > Sent from GMail on Android > > On Nov 22, 2017 11:32 PM, "Yi Pan" wrote: > > > And just one more question on your requirement for Kafka Client 1.0: is > > this requirement because of your broker v

Re: Dec Streams meetup @LinkedIn

Hey, all, Just a reminder, we will be talking about Samza-SQL, Slack on Samza, and Kafka @Uber on Dec 4th: https://www.meetup.com/Stream-Processing-Meetup- LinkedIn/events/244889719/. Please sign up! -Yi On Wed, Nov 8, 2017 at 10:30 AM, Yi Pan wrote: > Hi, all, > > We have fina

Re: [DISCUSS] Samza 0.14.0 release

Thanks for driving this! +1 A few minor things that are pending that I think we should pull in: 1) https://issues.apache.org/jira/browse/SAMZA-1459 2) https://github.com/apache/samza/pull/302 3) https://github.com/apache/samza/pull/301 4) https://github.com/apache/samza/pull/286 5) https://issues.

Re: Running apache samza with Kafka Client 1.0 - JIRA - SAMZA - 1418

RLClassLoader.findClass(URLClassLoader.java:381) > at java.lang.ClassLoader.loadClass(ClassLoader.java:424) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335) > at java.lang.ClassLoader.loadClass(ClassLoader.java:357) > ... 9 more > > On Mon, Nov

Re: [VOTE] Apache Samza 0.14.0 RC3

+1 binding Verified git tag and source signatures Ran check-all.sh on OSX Ran integration tests from OSX Thanks for push it forward! -Yi On Wed, Dec 20, 2017 at 3:19 PM, Jake Maes wrote: > +1 binding > > Verified git tag `release-0.14.0-rc3` > Verified source signature > > Ran check-all.sh o

Re: [VOTE] Apache Samza 0.14.0 RC5

+1 (binding). Verified the signature and MD5 Ran ./bin/check-all.sh on OSX Ran integration tests on OSX Verified ./gradlew releaseToolsTarGz generated samza-tools-0.14.0.tgz in build directory Thanks for all! -Yi On Fri, Dec 22, 2017 at 6:10 PM, Boris S wrote: > Verified the signature. > Ran

Re: [RESULT][VOTE] Apache Samza 0.14 RC5

Great! Thanks Jagadish for closing the VOTE. Let's wrap the rest of the release process in the first week of 2018! Happy new years to all! -Yi On Thu, Dec 28, 2017 at 3:22 PM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > The vote of 0.14.0 has been more than 72 hours and we got +1 (b

Re: Samza 0.14.0 officially released!

Awesome! Thanks for the great effort to push it all the way through! Cheers! -Yi On Fri, Jan 5, 2018 at 9:50 AM, xinyu liu wrote: > Hi, all, > > I am pleased to let you know that we have officially released Samza 0.14.0! > > Huge thanks to everyone for working on the features and bugs in this

[REPORT] Apache Samza - Jan 2018

Submitted via Whimsy. Re-posting to dev list as well. ## Description: - Apache Samza is a distributed stream processing engine that are highly configurable to process events from various data sources, including real-time messaging system (e.g. Kafka) and distributed file systems (e.g. H

Welcome Xinyu as new Samza PMC!

Finally all the documentation procedure is completed and Xinyu Liu has been officially promoted to Samza PMC member! This is well deserved due to his continued contribution to the Samza project. Please join me to welcome Xinyu as our newest PMC member! Cheers! -Yi Pan

Re: No Metrics Emitted in new High-Level API

Hi, Srujun, I did not see the container logs in the attachment. Is it too big? Can you send it to my personal email yi...@linkedin.com? Thanks! -Yi On Fri, Jan 26, 2018 at 5:23 PM, Srujun Thanmay Gupta wrote: > Hello, > > I have attached the container log and the job coordinator logs for the

Re: No Metrics Emitted in new High-Level API

Hi, Srujun, I noticed that you did not configure the metrics serde as stated here: http://samza.apache.org/learn/documentation/0.14/container/metrics.html. Could that be the reason? -Yi On Fri, Jan 26, 2018 at 7:58 PM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Can you turn on DEBU

Re: No Metrics Emitted in new High-Level API

les/samza/wikipedia-application > (metrics not working unless serdes are also configured) > > From what I can see, there are metrics serialization errors in the > wikipedia-application logs, as expected. But no such issue for > wikipedia-feed, even though no serde was configured. Is ther

Re: No Metrics Emitted in new High-Level API

SAMZA-1581 is the ticket On Thu, Feb 1, 2018 at 5:20 PM, Yi Pan wrote: > Hi, Srujun, > > Sorry to reply late. I am glad that you figured it out. The issue you > reported seems to be a bug that we set the default for metrics serde in one > case and not in the other one. I will o

Re: [DISCUSS] SEP: Host affinity in standalone.

Hi, Santhoosh, Thanks for the SEP and the latest revisions. Here are some of my comments based on the latest proposal: - The basic idea for implementing state-aware task-to-physical-process assignment in JobModel is not quite clear. ContainerAllocator is solving a different problem in host-affini

Re: SEP-11. Host Affinity in standalone discussion.

Linking Boris' earlier comment in another email to this thread: http://mail-archives.apache.org/mod_mbox/samza-dev/201801.mbox/%3CCAPAaT%2BtH2H5TEvFQUn9jw6iR%3DyvVEu46rDLJsqexpwKz0CAH1g%40mail.gmail.com%3E On Fri, Jan 26, 2018 at 4:17 PM, Boris S wrote: > Shanthoosh, > Thank you for suggesting a

Re: [DISCUSS] SEP: Host affinity in standalone.

Linking Boris' earlier comments to the correct [DISCUSSION] thread: http://mail-archives.apache.org/mod_mbox/samza-dev/201801.mbox/%3CCAPAaT%2BtH2H5TEvFQUn9jw6iR%3DyvVEu46rDLJsqexpwKz0CAH1g%40mail.gmail.com%3E On Thu, Feb 1, 2018 at 5:27 PM, Yi Pan wrote: > Hi, Santhoosh, > > Than

Re: [VOTE] SEP-11: Host affinity in standalone.

Hi, Santhoosh, Overall looks good to me now. There are still some minor issues / errors in the design doc - in YARN, are you sure that we persist the JobModel to coordinator stream? I checked the code and didn't find that. The JobModel was simply generated and served from the memory in the JobCoor

Stream Processing Meetup @LinkedIn tomorrow (3/21)

Hi, all, Just a reminder that we will host another stream processing meetup @LinkedIn tomorrow at 6pm: https://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/248309045/ Expect to see you there! -Yi

Re: [VOTE] SEP-11: Host affinity in standalone.

+1 (binding). Thanks for the revisions! -Yi On Wed, Apr 4, 2018 at 2:39 PM, santhosh venkat < santhoshvenkat1...@gmail.com> wrote: > Hi, > > This is a voting thread for SEP-11: Host affinity in standalone. > > For reference, here is the wiki link: https://cwiki.apache.org > /confluence/pages/vie

<    1   2   3   4   5   6   7   8   9   10   >