Re: multilingual tuples via kafka

Perfect. Thanks, Taylor. That explains the basics. So now I'm taking the string and parsing it as json. What should be the best practice to do it directly in a scheme? Cheers, On Tue, May 26, 2015 at 6:12 PM, P. Taylor Goetz wrote: > The data coming from Kafka to the Kafka spout is just a byte

Re: Aeolus 0.1 available

Hi Matthias, The project looks interesting. Any detailed performance data compared with latest storm versions (0.9.3 / 0.9.4) ? Thanks, Manu Zhang On Tue, May 26, 2015 at 11:52 PM, Matthias J. Sax < mj...@informatik.hu-berlin.de> wrote: > Dear Storm community, > > we would like to share our pro

What will Storm do when a supervisor is manually killed?

​Hi Storm fellows, ​I've got a simple question and would like to have a quick answer. Let's say a storm topology is running on a cluster without any supervision, at the beginning it is behaving properly and have a balanced distribution. But you know, sometimes errors may occur and bring down the s

RE: java.lang.OutOfMemoryError: GC overhead limit exceeded

Dell - Internal Use - Confidential The previous reply was from March 10th so Bill may have resolved this issue but here is some info that can be useful to other folks in the future. You can set the JVM flags to do a heapdump on OOM using -XX:+HeapDumpOnOutOfMemoryError. You can then analyze th

SF / East Bay Area Stream Processing Meetup next Thursday (6/4)

http://www.meetup.com/Bay-Area-Stream-Processing/events/219086133/ Thursday, June 4, 2015 6:45 PM TubeMogul 1250 53rd St #1 Emeryville, CA 6:45PM to 7:00PM - Socializing 7:00PM to 8:00PM - Talks 8:00PM to 8

Re: Load /Cache Data inside Bolt

If it is really static thing, you can serialize it and pass to Bolt's constructor. If you don't want to spend serialization cost, you can store it to file (and include it to resources of jar) and load it from preparation of Bolt. If you want to have small jar, you can use external storage. (RDB, HD

Re: java.lang.OutOfMemoryError: GC overhead limit exceeded

Not sure if you fixed the issue but I think the problem may come from the max spout pending. You are using Trident and this value is the max number of pending "BATCHES" and not number of "tuples" so let say your topic has 10 partitions and max spout pending is set to 10 and the max fetch size is se

RE: Status of running storm on yarn (the yahoo project)

Hi Nathan, I want to do real time computation using storm, which one is best storm or trident. i need to handle huge amount of data , exactly once please help me Thanks! Date: Wed, 27 May 2015 12:40:43 -0400 Subject: Re: Status of running storm on yarn (the yahoo project) From: nat...@nathanmarz

Re: Status of running storm on yarn (the yahoo project)

Mesosphere has official support for Storm on Mesos: https://github.com/mesos/storm On Wed, May 27, 2015 at 11:14 AM, wrote: > *Dell - Internal Use - Confidential * > > Thanks Bobby, for the detailed answer. > > > > So it sounds like , it is better not to combine Storm with batch > workloads at

Load /Cache Data inside Bolt

I need to load some static data inside Bolts and cached it , any recommendation as what is the best way to do it. Thanks,

RE: Status of running storm on yarn (the yahoo project)

Dell - Internal Use - Confidential Thanks Bobby, for the detailed answer. So it sounds like , it is better not to combine Storm with batch workloads at this point (yarn, mesos or ec2), due to the network saturation and timeout threats. Is this behavior also seen in other streaming frameworks l

Re: Status of running storm on yarn (the yahoo project)

I also developed a prototype/proof-of-concept (read: duck tape and bailer twine) for running Storm on YARN. I took a slightly different approach than Yahoo’s storm-yarn and Slider which from a high level allow you to spin up a Storm cluster on top of YARN. In my PoC a topology is treated as a s

Re: Status of running storm on yarn (the yahoo project)

Mesos is very similar to YARN.  It is a resource scheduler.  Storm in the past had support for mesos, through a separate repo https://github.com/nathanmarz/storm-mesos it might still work with the latest versions of storm.  I don't know.  The concept here is that there was a special layer instal

Re: Status of running storm on yarn (the yahoo project)

I have heard Nathan Marz mention Mesos. How is yarn / storm-yarn / slider-yarn different from Mesos? These are the links I found to Mesos: https://github.com/mesos/storm https://github.com/nathanmarz/storm-mesos http://mesos.apache.org/ Thank you for your time! + Jeff Maass

Re: Status of running storm on yarn (the yahoo project)

storm-yarn was originally done as a proof of concept.  We had plans to take it further, but the amount of work required to make it production ready on a very heavily used cluster was more then we were willing to invest at the time.  Most of that work was around network scheduling, isolation and

Re: unsubscribe

Hi , If file is small you can pass them as json serialized object in storm config while submiting topology and get from config map at spout and bolt. Thanks, _Nipur On May 27, 2015 3:22 PM, "Chris Bedford" wrote: > > > On Wed, May 27, 2015 at 1:51 AM, Tousif wrote: > >> Hi , >> >> >> Is there

Status of running storm on yarn (the yahoo project)

Hello! I'm interesting in running the storm topologies on yarn. I was looking at the yahoo project https://github.com/yahoo/storm-yarn, and I could observed that there is no activity since 7 months ago. Also, the issues and requests lists are not updated. Therefore I have some questions: 1. Is ther

unsubscribe

On Wed, May 27, 2015 at 1:51 AM, Tousif wrote: > Hi , > > > Is there a way to share a resource file across all workers similar to > hdfs. That resource/config file will have to be updated run time. i'm not > looking at using hdfs for now. > > -- > > > Regards > Tousif Khazi > > -- Chris Bedfor

sharing a resource across all workers

Hi , Is there a way to share a resource file across all workers similar to hdfs. That resource/config file will have to be updated run time. i'm not looking at using hdfs for now. -- Regards Tousif Khazi