Contributing Cassandra-Spout

Hi Storm Community, We have developed a cassandra-spout that can be used to: 1. Stream data from Cassandra continuously or 2. Read all data at once for a bulk bootstrap. If there is interest in the community, we can contribute it back to the open-source and open a pull request. It contains

Re: NullPointerException seen in 1.1.0

t the other node when it > tries to read this loses the race and comes up empty handed. But if you > only have a single nimbus configured then this is not the case and > something else worse is happening. > > > - Bobby > > On Monday, May 8, 2017, 4:41:13 PM CDT, S G <sg.onl

Re: When would you have multiple tasks in an executor?

ding dev@ to include anyone who might be interested.) > > -Taylor > > On May 13, 2017, at 7:24 PM, S G <sg.online.em...@gmail.com> wrote: > > Thanks Anshu. > If you can provide an example, that would be much appreciated. > > On Sat, May 13, 2017 at 1:24 PM, anshu shuk

When would you have multiple tasks in an executor?

Hi, As per this guide: http://storm.apache.org/releases/current/Understanding-the-parallelism-of-a-Storm-topology.html , An executor is a thread containing many tasks. And each task is an instance of a spout or a bolt. I cannot imagine when would someone want to have multiple tasks in an

NullPointerException seen in 1.1.0

Hi, I am trying to upgrade from 1.0.2 to 1.1.0 version of Storm. And I see the below exception happening randomly on the Nimbus node. When it happens, Nimbus is unable to accept any new topologies. java.lang.NullPointerException: null at

Nimbus-HA and recommended value for topology.min.replication.count

Hi, This is with reference to: http://storm.apache.org/releases/2.0.0-SNAPSHOT/nimbus-ha-design.html For topology.min.replication.count, it says: Minimum number of nimbus hosts where the code must be replicated before leader nimbus can mark the topology as active and create assignments. Default

Re: Too many threads per bolt instance (as seen with jconsole images)

ning > threads in the worker. > -roshan > > > On 3/16/17, 5:07 PM, "S G" <sg.online.em...@gmail.com> wrote: > > Thanks for sharing the doc Roshan. > It is very informative. > > I think number of threads per bolt is on the high side

Re: Too many threads per bolt instance (as seen with jconsole images)

usp=sharing > > > > > Get Outlook for iOS<https://aka.ms/o0ukef> > > > > > On Thu, Mar 16, 2017 at 12:16 PM -0700, "S G" <sg.online.em...@gmail.com< > mailto:sg.online.em...@gmail.com>> wrote: > > Hi, > > I am trying to make s

Too many threads per bolt instance (as seen with jconsole images)

Hi, I am trying to make sense of number of threads seen in JConsole. It seems like a very high number of threads are launched per bolt thread. ​ *Experiment 1* topology.workers.val=16 spout.parallelism.val=1 bolt1.parallelism.val=900 bolt2.parallelism.val=160 observation: Threads seen per node

Error while communicating with Nimbus

Hi, I am running a 3-nimbus, 12-supervisor storm cluster, version: 1.0.2 Many-a-times, I see that it gets impossible to kill a topology from the UI as it gives an error: "Error while communicating with Nimbus." Only message I see in the logs is: o.a.s.t.s.AbstractNonblockingServer$FrameBuffer

Ideas for storing state in Spout

Hi, We are writing a spout that needs to store its progress and unlike kafka, the datastore here has no concept of storing offsets. I looked at IStatefulBolt but that seems to be usable only for Bolts. Any idea if an IStatefulSpout is planned for the recent future? Thinking aloud, IStatefulSpout

Re: [VOTE] Release Apache Storm 1.1.0 (RC2)

Zookeeper version being used by Storm is 3.4.6 and it has a pretty serious bug when hostnames are used instead of IPs. The fix for the same is pretty small (See the PRs in https://issues.apache.org/jira/browse/STORM-2290) It would be good to include this for 1.1 -SG On Tue, Feb 28, 2017 at

Re: storm stops recieving events suddenly

Hi Ganesh, You can try enabling http://storm.apache.org/releases/1.0.0/Eventlogging.html if you are using 1.x version of storm. You can also try using http://storm.apache.org/releases/1.0.0/dynamic-log-level-settings.html to debug the solr-bolt -SG On Thu, Feb 16, 2017 at 1:22 AM, Ganesh Kumar

Seeing some spurious entries in Storm UI

Hi, Please see the screenshot below: ​I am seeing 6 hosts in nimbus summary where as I have only 3. I have checked the storm.yaml for each host too and it has only 3 entries for nimbus.seeds with "-ha" names. Can someone suggest please how I can get rid of the spurious hosts? Note: I changed

Re: [DISCUSS] Release Storm 1.1.0

Not sure if its a little late to include for the 1.1.0 and 1.0.3 releases now, but can we consider using zookeeper 3.4.9 for the future versions as 3.4.9 brings in a lot of stability improvements ( http://zookeeper.apache.org/releases.html) and storm is still using 3.4.6 (

Re: Mark DRPC as deprecated

it out and be disappoined. We should mention the same in the above docs too that its not production ready. Thanks SG On Sat, Jan 7, 2017 at 12:13 PM, S G <sg.online.em...@gmail.com> wrote: > Hi, > > I came across https://storm.apache.org/releases/current/ > Distributed-RP

Mark DRPC as deprecated

Hi, I came across https://storm.apache.org/releases/current/Distributed-RPC.html and thought of diving into it. But some research showed me that it is a deprecated feature. http://storm.apache.org/2012/08/02/storm080-released.html Transactional topologies documentation says that explicitly

Re: Twitter4j dependency (can't release Apache Storm)

Should we consider upgrading some versions in our pom.xml also? https://github.com/apache/storm/blob/master/pom.xml#L265 Some versions we use and the latest are: zookeeper - 3.4.6 --> 3.4.9 jackson - 2.6.3 --> 2.8.5 kryo - 3.0.3 --> 4.0.0 avro - 1.7.7 --> 1.8.1 Thanks SG On Wed, Jan 4, 2017

Re: Dynamic reconfiguration for the nodes in Nimbus/Pacemaker clusters

order you want. This should just work so long as you > have at least one nimbus node still up. > If you want to file a JIRA for it that would be wonderful. If you want to > do some of the work on reloading the configs periodically that would be > great too. - Bobby > > On Monda

Re: [DISCUSS] Prioritizing works in progress

r(new PairFunction<String, String, Integer>() { > @Override > public Pair<String, Integer> apply(String w) { > return Pair.of(w, 1); > } > }) > .countByKey() > .filter(new Predicate<Pair<String, Long>>() { > @Override > public bo

Re: [DISCUSS] Prioritizing works in progress

developers. (This last point reflects the thoughts from some of my fellow developers, not representative of all the devs) Thanks SG On Mon, Jan 2, 2017 at 8:26 PM, Jungtaek Lim <kabh...@gmail.com> wrote: > Hi S G, > > I'd be happy if you could elaborate your opinion. Did you found ba

Re: [DISCUSS] Prioritizing works in progress

I have found several bad usages of Java 8 lambdas and many developers are not comfortable using them. So we should use them only if it really makes the code beautiful and easier to understand. My 2c, Thanks SG On Mon, Dec 26, 2016 at 5:59 AM, Arun Mahadevan wrote: > The

Dynamic reconfiguration for the nodes in Nimbus/Pacemaker clusters

Hi, I was reading a zookeeper article: https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html And was wondering if a similar functionality exists for Nimbus/Pacemaker Cluster. It would be really nice if there is no need for restarting servers in the Nimbus/Pacemaker cluster whenever a

Plans to release 1.0.3 ?

Hi, 1.0.2 was released in Aug and 100+ issues have been fixed since then as per JIRA Any chances that

Re: How to debug java.nio.channels.ClosedChannelException in Storm 1.0.2

ouble and is closing connections unexpectedly. > > - Bobby > > On Thursday, December 15, 2016 8:15 PM, S G <sg.online.em...@gmail.com> > wrote: > > > Hi, > > I am using Storm 1.0.2 > My configuration is quite simple: `kafka-spout` feeding to `solr-bolt` > &

Confusing "nil" value for topology.eventlogger.executors

Hi, I see the value "nil" for topology.eventlogger.executors at http://storm.apache.org/releases/1.0.0/Eventlogging.html While other values are 0 or 1, "nil" seems really confusing especially because it does not disable something, rather it enables something. Something not expected of "nil" as

How to debug java.nio.channels.ClosedChannelException in Storm 1.0.2

Hi, I am using Storm 1.0.2 My configuration is quite simple: `kafka-spout` feeding to `solr-bolt` topology.workers = 2 spout.parallelism = 1 bolt.parallelism = 1 Our messages coming from kafka are large: around 100kb per message to max of 500kb per message. But I see lots of errors: Window

Re: Are storm metrics reported through JMX too?

; Ok maybe there is some effort duplication. For the config, I have the > bare minimum to get the default reporter up. I'll focus on that since you > could use it. Will update JIRA with more. > > > > Alessandro > > > > > > - Forwarded Message - > > Fro

Re: Are storm metrics reported through JMX too?

Goetz <ptgo...@gmail.com> > *To:* "dev@storm.apache.org" <dev@storm.apache.org> > *Cc:* S G <sg.online.em...@gmail.com>; na...@narendasan.com < > na...@narendasan.com>; Austin Chung <achun...@illinois.edu> > *Sent:* Tuesday, November 29, 2016, 1:27:58 PM

Re: Are storm metrics reported through JMX too?

ois Open Source class. > > > > > > On Tuesday, October 11, 2016 11:48 PM, S G <sg.online.em...@gmail.com> > > wrote: > > > > > > Response on this important issue is pretty good. I am happily surprised > :) > > > > I want to mention our

Should nimbus.seeds always be refreshed in storm.yaml?

Hi, We are setting up a storm cluster where a nimbus node would be automatically added to the nimbus cluster if any nimbus machine fails. In such a scenario, the new nimbus node will have a different host-name/IP address. If that happens, is it absolutely mandatory for all the

Re: Are storm metrics reported through JMX too?

ic versioning...? > > -Taylor > > > On Oct 11, 2016, at 9:28 PM, Jungtaek Lim <kabh...@gmail.com> wrote: > > > > Yeah I admit that configuration flag was bad for me also, but I have no > > alternatives. Only way to avoid struggling with design limitation is >

Re: Are storm metrics reported through JMX too?

; > | || > > > > | > > > > | > > | > > | | > > Sending Metrics from Storm to Graphite - Michael G. Noll > > By Michael G. Noll Sending application-level metrics from Storm > topologies > > to the Graphite monitoring system | | > >

Re: Are storm metrics reported through JMX too?

e to support that, but it is far from a requirement. - > Bobby > > On Monday, October 10, 2016 10:47 PM, S G <sg.online.em...@gmail.com> > wrote: > > > +1 > We can probably start by opening a JIRA for this and adding a design > approach for the same? > I w

Re: Are storm metrics reported through JMX too?

> HTTP server is down the consumer crashes and continues to crash until the > server is brought back up). It was written very quickly for a test > situation and it honestly never crossed my mind that anyone would want to > use it in production. > > > > - Bobby > > >

Re: Are storm metrics reported through JMX too?

r this. We are in the process of trying to > redesign the metrics system to allow for features like this, but it is > still a ways off. > > - Bobby > > > On Saturday, October 8, 2016 11:39 AM, S G <sg.online.em...@gmail.com> > wrote: > > > Thanks Bobby, > > W

Re: Are storm metrics reported through JMX too?

ht work, but going directly to telegraph > would probably be better. - Bobby > > On Thursday, October 6, 2016 1:43 PM, S G <sg.online.em...@gmail.com> > wrote: > > > Hi, > > We want to use Telegraf ( > https://github.com/influxdata/telegraf/tree/

Are storm metrics reported through JMX too?

Hi, We want to use Telegraf ( https://github.com/influxdata/telegraf/tree/master/plugins) for getting storm's metrics. But we do not want to add a HttpForwardingMetricsServer just to get the metrics and send them to telegraf. Other option is to use Jolokia (https://jolokia.org/) that can read

Re: [DISCUSS] Provision for dead-letter topic in storm

exity to your idea > and may not be feasible. > On Tue, Sep 20, 2016 at 1:15 AM S G <sg.online.em...@gmail.com> wrote: > > > Hi, > > > > I want to gather some thoughts on a suggestion to provide a dead-letter > > functionality common to all spouts/bolts. >

[DISCUSS] Provision for dead-letter topic in storm

Hi, I want to gather some thoughts on a suggestion to provide a dead-letter functionality common to all spouts/bolts. Currently, if any spout / bolt reports a failure, it is retried by the spout. For a single bolt-failure in a large ADG, this retry logic can cause several perfectly successful

Re: Flux exception while upgrading to Storm 1.0.2

t may give you a hint. > > Thanks, > Manu > > > On Tue, Sep 13, 2016 at 6:50 AM S G <sg.online.em...@gmail.com> wrote: > > > From the > > https://github.com/apache/storm/tree/1.0.x-branch/external/storm-kafka > > docs, > > > > "This means that

Re: Flux exception while upgrading to Storm 1.0.2

2016 at 3:44 PM, S G <sg.online.em...@gmail.com> wrote: > Can anyone help with this? > > I tried relocating classes too during shading as: > > > backtype.storm > org.apache.storm >

Re: Flux exception while upgrading to Storm 1.0.2

as a transitive dependency in my project. What to do now? Thx, SG On Mon, Sep 12, 2016 at 12:24 PM, S G <sg.online.em...@gmail.com> wrote: > Hi, > > We are upgrading to Storm 1.0.2 and have done all the package name changes. > > grep -ri backtype in our code repo ret

Re: How does Storm's OutputCollector work?

Thx Bobby, Just curious, how much of the code-base is planned to be moved away from clojure to Java? When do we plan to make a release for the Java-based code? On Mon, Sep 12, 2016 at 6:21 AM, Bobby Evans wrote: > Manu Zhang is correct in his answer. The main

Flux exception while upgrading to Storm 1.0.2

Hi, We are upgrading to Storm 1.0.2 and have done all the package name changes. grep -ri backtype in our code repo returns nothing. Yet on deploying the jar to Storm, following error is seen: storm/storm_installed/apache-storm-1.0.2/bin/storm \ jar \ $jarFile \ org.apache.storm.flux.Flux \

Re: [VOTE] Accept storm-jms Code Donation

+1 On Thu, Sep 8, 2016 at 1:50 PM, P. Taylor Goetz wrote: > Following an earlier discussion thread, I’d like to start of VOTE on > whether to accept the storm-jms code donation. > > The codebase being donated can be found here [1]. > > [ ] +1 Accept the code donation. > [ ] 0

Re: Embedded Storm (Cluster)

; Thanks, > Satish. > > On Thu, Sep 8, 2016 at 5:28 AM, S G <sg.online.em...@gmail.com> wrote: > > > Hi, > > > > Some open-source tools have an embedded option that makes it very easy to > > test. > > Example: > > http://stackoverflow.com/questions/1980

Embedded Storm (Cluster)

Hi, Some open-source tools have an embedded option that makes it very easy to test. Example: http://stackoverflow.com/questions/19806750/what-is-wrong-with-embedding-solr "The embedded server is recommended when you need a simple solution that is not distributed. For example when writing small

Performance improvements in 1.0.0

Hi, On the http://storm.apache.org/2016/04/12/storm100-released.html page, I see a notice about performance improvements to the order of 16 times. Can someone please provide a little more details on the same? Is the throughput of a topology increased 16 times (that sounds a lot, hence the

Re: [DISCUSS] storm-jms Code Donation

I would love to see it part of storm/external and can volunteer to provide any help if required during the migration. -SG On Fri, Sep 2, 2016 at 12:59 PM, P. Taylor Goetz wrote: > I’d like to start a discussion around adding storm-jms as an external > module. I’ve gotten a

Re: Package name changes for 3rd party storm dependencies

ade-plugin/ > examples/class-relocation.html > > ) > > > > You can add the patterns in this class - > > > > https://github.com/apache/storm/blob/master/storm- > rename-hack/src/main/java/org/apache/storm/hack/StormShadeRequest.java > > > > > &g

Re: storm-jms

Got it. Thanks a lot Taylor ! On Thu, Aug 25, 2016 at 4:34 PM, P. Taylor Goetz <ptgo...@gmail.com> wrote: > It's on central now: > > http://search.maven.org/#search%7Cga%7C1%7Ca%3A%22storm-jms%22 > > -Taylor > > > On Aug 25, 2016, at 7:04 PM, S G <sg.online.em...

Re: storm-jms

com.github.ptgoetz > storm-jms > 1.0.2 > > > -Taylor > > > On Aug 25, 2016, at 2:32 PM, S G <sg.online.em...@gmail.com> wrote: > > > > Thanks a lot Taylor ! > > Waiting to get the next storm-jms > > > > On Thu, Aug 25, 2016 at 10:52

Package name changes for 3rd party storm dependencies

Hi, Our project is planning to upgrade storm from 0.10.0 to 1.0.2 We use a couple of 3rd party storm dependencies like storm-jms, https://github.com/HolmesNL/kafka-spout etc. Some of these projects have plans to upgrade soon to 1.0.2 but some do not have it on their roadmap. What is the best

Re: storm-jms

re that can happen. > > In the meantime, I will update the project to use the latest version of > Storm and release it to maven central. > > -Taylor > > > On Aug 25, 2016, at 1:03 PM, S G <sg.online.em...@gmail.com> wrote: > > > > Hi, > > > > It se

storm-jms

Hi, It seems that https://github.com/ptgoetz/storm-jms is running 0.9.3 version of storm. Any chance that it can be added to storm/external and kept up-to-date with latest releases of storm? Thanks SG

STORM-971: KafkaSpout - Improve "_lostMessageCount" metric for failed tuples

Hi, I have a small pull request for review. Its a one-line functionality change with some small changes for making 3 warnings more helpful. Would appreciate if someone can help me get this in. Here is the PR: https://github.com/apache/storm/pull/1288 Thanks Sachin

Re: Code review help for a small pull request

Sending a gentle reminder for review help. Anybody familiar with Kafka Spout should be able to review this small patch. On Thu, Mar 31, 2016 at 6:08 PM, S G <sg.online.em...@gmail.com> wrote: > Hi, > > I have a small pull request for review. > > Its a one-line functiona

Code review help for a small pull request

Hi, I have a small pull request for review. Its a one-line functionality change with some small changes for making 3 warnings more helpful. Would appreciate if someone can help me get this in. Here is the PR: https://github.com/apache/storm/pull/1288 Thanks Sachin