Re: Question about scheduler.display.resource option.

Any help with this? Thanks. Ethan Li Tech Yahoo, Software Dev Eng, Interm1908 S. First St. Champaign, IL 61820etha...@yahoo-inc.com Mobile:  812-361-1879 On Thursday, July 13, 2017 3:44 PM, Ethan Li <etha...@yahoo-inc.com> wrote: Hello, I am facing with a really strange issue

Re: how to keep storm running when console ends

Hi, Storm docs recommend to use daemontools or monit which provides error recovery and etc..  I use linux "nohup" or "screen" command for simplicity.  Ethan  On Friday, August 4, 2017 7:29 AM, J.R. Pauley wrote: this seems silly but I have not figured out how to

Re: how to keep storm running when console ends

ss...@gmail.com> wrote: Try adding nohup at the beginning of each of those commands.  Nohup prevents a process from terminating when it's parent shell is closed. nohup /opt/storm/apache*/bin/ storm nimbus & Sent from my iPhone On Aug 4, 2017, at 09:08, Ethan Li <etha...@yahoo-inc.com

Re: Storm Kerberos starting topology fails with "The TGT found is not renewable"

Hi Prakash, It might sound silly but did you check if the ticket you think you are using is the one that’s actually being used. I fixed the “The TGT found is not renewable” problem in my use case before but sorry I couldn’t remember the details. Best, Ethan > On Jan 11, 2018, at 3:10 PM,

Re: Display ackers executors

localhost:8080/api/v1/topology/my-topology-id/component/__acker > <http://localhost:8080/api/v1/topology/my-topology-id/component/__acker> > doesn’t give me the same output as other components. > > Cheers, > > Alessio > > >> On 19 Jul 2018, at 18:20, Ethan

Re: Display ackers executors

At the bottom of the topology page, you can find “Show System Stats” button. Click on it and you will see __acker. Best, Ethan > On Jul 19, 2018, at 11:04 AM, Alessio Pagliari wrote: > > Hello everybody, > > I cannot find a way to show in the UI or retrieve through REST APIs the > placement

Re: Running multiple topologies

Hi Daniel, I am not sure if I understand your questions correctly. But will the resource aware scheduler help? https://github.com/apache/storm/blob/master/docs/Resource_Aware_Scheduler_overview.md Thanks

Re: worker.yaml and worker.pid in workers.artifacts director

TS_DIR}//logs, but > worker.yaml and worker.pid will still be sent to > {STORM_WORKERS_ARTIFACTS_DIR}/{worker id}/{worker port}. Is there anyway to > specify the path of worker.yaml and worker.pid as the same as the other log > files? > > Sent from Bloomberg Professional for Android &g

Re: worker.yaml and worker.pid in workers.artifacts directory

Hi Mitchell, worker.yaml and worker.pid are used by storm itself. They will be put into {STORM_WORKERS_ARTIFACTS_DIR}/{worker id}/{worker port} (see https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/utils/ConfigUtils.java#L259-L261

Re: Question about Evenscheduler in Storm 2.0

Hi Junguk, Please set "storm.scheduler” (https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/DaemonConfig.java#L96 ) to change scheduler -

Re: Start command not propagated by Nimbus

Hi Mitchell, Does the UI show that this topology is fully scheduled? Are you using ResourceAwareScheduler? Because it’s possible that the scheduler cannot find enough resources to schedule this topology. Also after starting the topology, you could login to zookeeper and check if there is a

Re: [ANNOUNCE] Apache Storm 2.0.0 Released

It’s great news! Thank you very much for the release Best, Ethan > On May 30, 2019, at 3:50 PM, Alexandre Vermeerbergen > wrote: > > Thanks you very much Taylor for releasing this Storm 2.0 major release ! > > Kind regards, > Alexandre Vermeerbergen > > Le jeu. 30 mai 2019 à 21:56, P.

Re: Doubt about ShuffleGrouping and LocalOrShuffleGrouping

Hi Mohit, Your understanding about ShuffleGrouping is correct. For LocalOrShuffleGrouping, the upstream spout/bolt will only send tuples to the downstream bolt on the same worker (not machine) if it can. Otherwise, it will be just like ShuffleGrouping. Code:

Re: Apache Storm version 2.0 problem running local topology in maven

Hi Damjan, Your topology is actually able to run locally and I can see see the output from your bolt: “The new sum is xxx ”. Although (local)cluster.shutdown didn’t really work and keep printing out errors. I would like to take some time later to look into it. By the way, to upgrade from

Re: Apache Storm version 2.0 problem running local topology in maven

ral times and I never saw any output.. I will try it under Linux > and see if something changes.. > > Regards, > Damjan > > On Wed, 3 Jul 2019, 17:29 Ethan Li, <mailto:ethanopensou...@gmail.com>> wrote: > Hi Damjan, > > Your topology is actually able to ru

Re: Storm logger file

Hi Abhishek, You can use “storm set_log_level” command to set log level for your topology, for example, "storm set_log_level -l ROOT=DEBUG:30 topology-name”. You can run "storm help set_log_level” for the usage of it. Thanks. -Ethan > On Apr 5, 2019, at 7:08 AM, Abhishek Raj wrote: > > Hi,

Re: Storm Helm Chart example

s so > that we can take necessary guidance from it. Further, is there any version of > the docker images (https://hub.docker.com/_/storm/?tab=tags > <https://hub.docker.com/_/storm/?tab=tags>) that has support for SLES 15 ? > > On Fri, Aug 2, 2019 at 7:25 PM Ethan Li <m

Re: Storm Helm Chart example

Hi Indranil, I am aware of any of Helm Chart based installation reference. > On Aug 2, 2019, at 8:23 AM, Indranil Roy > wrote: > > Is there any official reference of Helm chart based installation of Storm ?

Re: Storm 2.0.0 build error

I just tried and it was able to compile using mvn clean install -DskipTests -P 'native,include-shaded-deps,!externals,!examples,!coverage’ [INFO] [INFO] Reactor Summary for Storm 2.0.0: [INFO] [INFO] Storm

Re: pacemaker question

Hi Ignor, If you are running 2.0.0 or higher version of topology on 2.x Storm cluster, thanks to the changes of https://issues.apache.org/jira/browse/STORM-2693 , https://github.com/apache/storm/pull/2433

Re: pacemaker question

Sorry Igor for not typing your name correctly. > On Aug 22, 2019, at 11:06 AM, Ethan Li wrote: > > Hi Ignor, > > If you are running 2.0.0 or higher version of topology on 2.x Storm cluster, > thanks to the changes of https://issues.apache.org/jira/browse/ST

Re: Decline in apache storm usage

Hi Gunjan, I can’t speak for general usages. But the company I work for is using Apache Storm very heavily. > On Aug 28, 2019, at 9:44 AM, Gunjan Dave wrote: > > > Hello devs, > Seeing that apache storms usage is declining. Do you attribute this to > anything? > Even though storm do

Re: Decline in apache storm usage

ow GCP has > done it for apache spark/gcp-proc and apache-beam/gcp-dataflow? > > > On Thu, Aug 29, 2019, 20:54 Ethan Li <mailto:ethanopensou...@gmail.com>> wrote: > Hi Gunjan, > > I can’t speak for general usages. But the company I work for is using Apache >

Re: pacemaker question

> > could you please confirm that Zookeeper is required in any Storm > configuration? > > On Thu, Aug 22, 2019, 19:06 Ethan Li <mailto:ethanopensou...@gmail.com>> wrote: > Hi Ignor, > > If you are running 2.0.0 or higher version of topology on 2.x

[ANNOUNCE] Apache Storm 2.1.0 Released

The Apache Storm community is pleased to announce the release of Apache Storm version 2.1.0. Apache Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more about Storm on the project

Re: Upgrading storm 1.1 to 2.0 Command Line Arguments Issue

Hi Ray, There is a known bug in 2.0.0 and it’s fixed in 2.1.0. Please try 2.1.0 and see if it works for you. Thanks, Ethan > On Oct 16, 2019, at 4:10 PM, Zac wrote: > > Hi all, > > While upgrading storm versions, I found that I could not submit command line > arguments to my topology

Re: Storm 2.1

Hi Simon, There are a few issues related to CLI that should be fixed shortly. We will then have a new RC for vote. > On Oct 23, 2019, at 4:29 AM, Simon Cooper > wrote: > > Hi, > > Do we have an ETA for Storm 2.1 yet? I know there were a couple of RCs a > while ago, but everything seems

Re: Storm Diagnostics Tool

Hi Sreeram, It’s great to see such tools built for helping to solve performance problem for topologies. Thanks for sharing! Best, Ethan > On Oct 4, 2019, at 7:34 AM, Sreeram wrote: > > Dear Storm Community, > > Greetings! > > We (GSTN India team) use Apache Storm for our workloads. To

Re: localorshuffle grouping doubt

They will not. Please refer to the code here https://github.com/apache/storm/blob/master/storm-client/src/jvm/org/apache/storm/daemon/GrouperFactory.java#L85-L88 The

Re: Storm 2.0 worker heartbeat

Hi Surajeet, In 2.0, with the change in STORM-2693 , supervisors will send worker heartbeat to nimbus directly. Pacemaker is not required but still can be used. Links: [1] https://issues.apache.org/jira/browse/STORM-2693

Re: Custom metrics not showing up in storm U - Storm version 1.1.0

com/apache/storm/blob/v1.1.0/docs/Metrics.md#metrics-consumer> Where you can get the metrics depending on how your metric consumer looks like. For example, with org.apache.storm.metric.LoggingMetricsConsumer, metrics will be written to the log file. Best, Ethan > On Mar 5, 2020, at 10:36 PM,

Re: Issues with logging in Storm workers/topology

Hi, I don’t claim to know what’s going on here. But just want to give you some input from my little experiment. By simply replacing https://github.com/apache/storm/blob/v2.1.0/log4j2/worker.xml#L21 with your pattern:

Re: Custom metrics not showing up in storm U - Storm version 1.1.0

Hi Bala, You need to register metric consumers. Please check https://github.com/apache/storm/blob/v1.1.0/docs/Metrics.md#metrics-consumer Best Ethan > On Feb 13, 2020, at 5:48 PM, Bala wrote: > > I am using the

Re: TTransportException during shutdown of topology

Hi It’s hard to know without more logs. But it could be other workers which connected to this worker shut down abruptly and the connection was closed unexpectedly, according to the code at

Re: Storm 1.2.1 - Excessive workerbeats causing long GC and thus disconneting zookeeper

en the assigned memory for that worker is 512MB. Hope that helps. > On Feb 17, 2020, at 8:55 AM, Narasimhan Chengalvarayan > wrote: > > Hi Ethan Li, > > > Sorry for the late reply. Please find the output where it is showing > -Xmx2048m for worker heap . But in storm

Re: Bolts Acking

Hi, If you don’t anchor at all, collector.ack will basically do nothing (because input.getMessageId().getAnchorsToIds() is empty). It will accumulate the metrics (acked count) though. If you don’t acking at all, you can set “topology.acker.executors” to zero. Code is here: Ack:

Re: Storm 1.2.1 - Excessive workerbeats causing long GC and thus disconneting zookeeper

ave >> set worker memory has 2GB. >> >> On Fri, 24 Jan 2020 at 01:25, Ethan Li wrote: >>> >>> >>> 1) What is stored in Workerbeats znode? >>> >>> >>> Worker periodically sends heartbeat to zookeeper under workerbeats node

Re: Storm 1.2.1 - Excessive workerbeats causing long GC and thus disconneting zookeeper

rg.mycompany.MyDecorator > # > ## Locations of the drpc servers > # drpc.servers: > # - "server1" > # - "server2" > > ## Metrics Consumers > # topology.metrics.consumer.register: > # - class: "org.apache.storm.metric.LoggingMetricsConsumer"

Re: Storm 1.2.1 - Excessive workerbeats causing long GC and thus disconneting zookeeper

> 1) What is stored in Workerbeats znode? Worker periodically sends heartbeat to zookeeper under workerbeats node. > 2) Which settings control the frequency of workerbeats update https://github.com/apache/storm/blob/1.x-branch/storm-core/src/jvm/org/apache/storm/Config.java#L1534-L1539

Re: Worker never starts

e to add some extra worker nodes. > > Thanks, > Srikanth > > On Fri, Jan 10, 2020 at 10:18 PM Ethan Li <mailto:ethanopensou...@gmail.com>> wrote: > Hi Srikanth, > > It’s hard to tell without enough logs/informations. Could you attach your > co

Re: Change Storm binding address in local mode

Hi Andrea, I don’t see a way to change binding address for zookeeper if you are using InProcessZookeeper or DevZookeeper. https://github.com/apache/storm/blob/master/storm-server/src/main/java/org/apache/storm/zookeeper/Zookeeper.java#L78

Re: Worker never starts

Hi Srikanth, It’s hard to tell without enough logs/informations. Could you attach your configs, supervisor logs to start with? Best, Ethan > On Jan 10, 2020, at 10:28 AM, Srikanth bathi wrote: > > > > -- Forwarded message - > From: Srikanth bathi

Re: Behavior of heartbeats in 2.1

Hi Andrew, I just replied to your question in another email thread. I hope it helps. In short, in 2.x, you don't need PaceMaker. On Thu, Mar 19, 2020 at 7:16 PM Andrew Neilson wrote: > We're working on moving from v0.9.5 to 2.1 right now and as you can > imagine there have been quite a few

Re: Storm Authentication using Kerberos

I have never done it before but I guess it should work, without security or using SASL with Digest-MD5. Storm components are merely Zookeeper Clients. On Sat, Mar 28, 2020 at 10:09 AM shrikant kalani wrote: > I have a storm cluster where I want to implement Kerberos Authentication > mechanism.

Re: Storm 2.0 worker heartbeat

+? It's not > totally clear to me from how that ticket is written but it looks like that > replaced all of the heartbeat logic that was managed by Pacemaker and ZK in > older versions. > > Thanks, > Andrew > > On Thu, Mar 5, 2020 at 7:08 PM Ethan Li wrote: > >> Hi

Re: Storm 2.1 doesn't work on multiple workers

Hi Simon, I just replied in the JIRA. I am interested and would like to look into it. We have been running storm 2.x without any serious problem. Can you provide a simple example to reproduce the issue? It will be very helpful for me to debug this. Thanks Best Ethan On Wed, Apr 1, 2020 at 5:22

Re: MTLS authentication for Apache Storm

Hi Shrikant, I am not aware of any plan on this currently. Proposal and code contributions are welcome. Thanks On Sat, Mar 28, 2020 at 10:25 AM shrikant kalani wrote: > Hi > > Are there any plans that storm nimbus and supervisor will support MTLS in > future releases ? > > Thanks > Srikant

Re: Storm 2.0 worker heartbeat

In summary, on 2.x, you shouldn't need to worry about PaceMaker. And I don't suggest to support running older version of topologies running on 2.x storm cluster. It is painful in practice. On Wed, Apr 8, 2020 at 10:44 PM Ethan Li wrote: > Hi Andrew, > > There are two places Pace

Re: Is anyone available for some in-depth storm 2.1 explaining?

Hi Peter, I am not very familiar with trident kafka spout. I found something https://github.com/apache/storm/blob/master/docs/storm-kafka-client.md#manual-partition-assignment-advanced related. See if it helps. Best Ethan On Tue, Apr 7, 2020 at 2:07 AM Peter Neubauer wrote: > Hi there, > we

Re: Storm 2.1 doesn't work on multiple workers

Hi Fan, To unsubscribe, please send an email to user-unsubscr...@storm.apache.org. Please refer to http://storm.apache.org/getting-help.html On Wed, Apr 8, 2020 at 11:22 PM Fan Jiang wrote: > unsubscribe > > On Thu, Apr 9, 2020, 12:21 AM Ethan Li wrote: > >> Hi Simon, &g

Re: Query: Storm SSL Support

Hi Harish, As far as I know, storm doesn’t have encryption between daemons (nimbus<—> supervisor, supervisor<—> supervisor) at this point. Yes we should be able to use SSL enabled thrift. But it is hard (at least for me) to say how much work is needed without looking into it. Contribution on

Re: [ANNOUNCE] Apache Storm 2.2.0 Released

Congratulations! Thanks Govind for the efforts on getting 2.2.0 released. And thanks to everyone for the contributions! - Ethan > On Jun 30, 2020, at 11:36 AM, Govind Menon wrote: > > The Apache Storm community is pleased to announce the release of Apache > Storm version 2.2.0. > > Apache

Re: [Storm 2.1.0] Adjusting topology assigned memory

Hi Pavithra, The assignedMemory in topology summary is the total memory assigned to your topologies. So it changes when you change your number of executors. You can visit the topology page and it will show you how much memory is assigned for each component. I suggest to not change Xmx in

Re: Kerberos in Storm

Hi Srikant, I am not able to tell without looking at more logs. Can you provide your storm.yaml and nimbus logs? This doc https://github.com/apache/storm/blob/master/SECURITY.md#authentication-kerberos might be

Re: Query: Storm SSL Support

truly hope we can contribute to the community as well > > Regards, > -Harish > > From: Ethan Li > Reply-To: "user@storm.apache.org" > Date: Tuesday, June 30, 2020 at 3:09 PM > To: "user@storm.apache.org" > Cc: "dev-h...@storm.apache.org"

Re: Backpressure causing deadlock with recursive topology structure

This is an interesting use case. But my understanding of storm is it only supports DAG. But to mitigate this issue, I think there are a few things that can be tried. 1. Enable acking, and anchor the tuple before emitting, then set the max.spout.pending

[ANNOUNCE] Apache Storm 2.2.1 Released

The Apache Storm community is pleased to announce the release of Apache Storm version 2.2.1. Apache Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more about Apache Storm on the

[ANNOUNCE] Apache Storm 1.2.4 Released

The Apache Storm community is pleased to announce the release of Apache Storm version 1.2.4. Apache Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more about Apache Storm on the

[ANNOUNCE] Apache Storm 2.1.1 Released

The Apache Storm community is pleased to announce the release of Apache Storm version 2.1.1. Apache Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more about Apache Storm on the

[ANNOUNCE] Apache Storm 2.3.0 Released

The Apache Storm community is pleased to announce the release of Apache Storm version 2.3.0. Apache Storm is a distributed, fault-tolerant, and high-performance realtime computation system that provides strong guarantees on the processing of data. You can read more about Apache Storm on the