Re: Storm Flux Viewer

[Jungtaek Lim]

Looks great! Thanks for sharing.

I think it's worth to include to the submodule of flux or one of page in
UI, only If you are open to contribute your work to Storm project.

- Jungtaek Lim (HeartSaVioR)

2017년 2월 3일 (금) 오후 12:03, Xin Wang 님이 작성:

> Hi Ambud,
>
> Thanks for your nice work. I tested it. Looks good. This can be a useful
> tool for flux users.
>
> - Xin
>
> 2017-02-03 5:08 GMT+08:00 Ambud Sharma :
>
> Put together a simple webpage to visualize Flux YAML files to help
> troubleshooting and development of Flux based topologies.
>
> https://github.com/ambud/flux-viewer
>
>
>

Re: Storm Flux Viewer

[Xin Wang]

Hi Ambud,

Thanks for your nice work. I tested it. Looks good. This can be a useful
tool for flux users.

- Xin

2017-02-03 5:08 GMT+08:00 Ambud Sharma :

> Put together a simple webpage to visualize Flux YAML files to help
> troubleshooting and development of Flux based topologies.
>
> https://github.com/ambud/flux-viewer
>
>

Re: Kafka monitor unable to get offset lag

[Priyank Shah]

Hi Igor,

By secured setup I meant how is the kafka cluster setup. What port is the 
broker listening on?

Looked at the code. Not sure why is it throwing that error in UI. The only 
difference in debug output and command you run on command line is that –t 
argument has a trailing comma. Code should handle it.

Can you try running the below two commands on command line and paste output for 
each ?

/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t gtp -g test -b 
dn-06.ru:6667,dn-10.ru:6667

/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t gtp, -g test -b 
dn-06.ru:6667,dn-10.ru:6667

From: Igor Kuzmenko 
Reply-To: "user@storm.apache.org" 
Date: Wednesday, February 1, 2017 at 11:53 AM
To: "user@storm.apache.org" 
Subject: Re: Kafka monitor unable to get offset lag

Yes running from command line gives me json with topic lag.
What is secured setup?

On Wed, Feb 1, 2017 at 10:28 PM, Priyank Shah 
mailto:ps...@hortonworks.com>> wrote:
Hi Igor,

Just to clarify, are you saying that running storm-kafka-monitor from command 
line with the arguments you pasted returns the spout lags correctly ? Also, can 
you confirm if you are running in a secured setup or not ?

From: Igor Kuzmenko mailto:f1she...@gmail.com>>
Reply-To: "user@storm.apache.org" 
mailto:user@storm.apache.org>>
Date: Wednesday, February 1, 2017 at 5:05 AM
To: "user@storm.apache.org" 
mailto:user@storm.apache.org>>
Subject: Re: Kafka monitor unable to get offset lag

Found in logs with debug level:
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] spout classname: 
org.apache.storm.kafka.spout.KafkaSpout
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] json configuration: 
{config.bootstrap.servers=dn-06.ru:6667,dn-10.ru:6667,
 config.topics=gtp,, config.groupid=test, topology.tasks=1}
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] Command to run: 
[/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor, -t, gtp,, -g, test, -b, 
dn-06.ru:6667,dn-10.ru:6667]
2017-02-01 14:07:36.338 o.a.s.u.TopologySpoutLag [DEBUG] JSON parsing failed, 
assuming message as error message: Unable to get offset lags for kafka. Reason: 
org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired while 
fetching topic metadata

Running command /usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t gtp -g 
test -b dn-06.ru:6667,dn-10.ru:6667 
 returns correct json with spout lags. There's no timeout command executing in 
a second.



On Wed, Feb 1, 2017 at 12:25 PM, Spico Florin 
mailto:spicoflo...@gmail.com>> wrote:
Hello!
  You can check how your topic is consumed  and its health via the scripts:
./kafka-consumer-groups.sh --new-consumer --bootstrap-server :6667 
--list
 ./kafka-consumer-groups.sh --new-consumer --bootstrap-server 
:6667 --describe --group 

I hope it helps.
 Florin

On Wed, Feb 1, 2017 at 11:01 AM, Igor Kuzmenko 
mailto:f1she...@gmail.com>> wrote:
Yes, topology process data and works fine.
I couldn't find any exceptions in storm logs.   access-web-ui.log contains only 
these lines

2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 
10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 
10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 
10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 
10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from: 
10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:

On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah 
mailto:ps...@hortonworks.com>> wrote:
Hi Igor,

When you say topology is working fine do you mean you see data flowing? Can you 
try to look up the logs for ui server and paste relevant lines here if any?

Priyank

Sent from my iPhone

On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko 
mailto:f1she...@gmail.com>> wrote:
I've launched topology with new kafka spout. Topology by it self working fine, 
but looking at storm UI I see kafka-monitor exception:
Unable to get offset lags for kafka. Reason: 
org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired while 
fetching topic metadata

Maybe I forgot to configure something, but then how topology reads messages?

Storm Flux Viewer

[Ambud Sharma]

Put together a simple webpage to visualize Flux YAML files to help
troubleshooting and development of Flux based topologies.

https://github.com/ambud/flux-viewer

Read of kafka topic using KafkaSpout

[Thomas Cristanis]

I have 10 partitions on a topic of Kafka, but when trying to read the
following documentation
 Storm
got the error below. I tried it with Storm core and with Trident, the error
is always the same.

int numPartition = 10;

Broker broker = new Broker(Configuration.BOOTSTRAP_SERVERS_CONFIG);

GlobalPartitionInformation info = new GlobalPartitionInformation(topicName);

for (int i = 0; i < numPartition; i++) {

info.addPartition(i, broker);

}

StaticHosts hosts = new StaticHosts(info);

SpoutConfig kafkaSpoutConfig = new SpoutConfig(hosts, topicName, "/brokers",
UUID.randomUUID().toString());

/* http://goo.gl/riljni | http://goo.gl/37ZUuV */

kafkaSpoutConfig.ignoreZkOffsets = true;

kafkaSpoutConfig.startOffsetTime = -1;

kafkaSpoutConfig.scheme = new org.apache.storm.spout.SchemeAsMultiScheme(new
StringScheme());


java.nio.channels.ClosedChannelException at
kafka.network.BlockingChannel.send(BlockingChannel.scala:110) at
kafka.consumer.SimpleConsumer.liftedTree1$1(SimpleConsumer.scala:98) at
kafka.consumer.SimpleConsumer.kafka$consumer$SimpleConsumer$$sendRequest(SimpleConsumer.scala:83)
at kafka.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:149)
at
kafka.javaapi.consumer.SimpleConsumer.getOffsetsBefore(SimpleConsumer.scala:79)
at org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:75) at
org.apache.storm.kafka.KafkaUtils.getOffset(KafkaUtils.java:65) at
org.apache.storm.kafka.PartitionManager.(PartitionManager.java:94) at
org.apache.storm.kafka.StaticCoordinator.(StaticCoordinator.java:35)
at org.apache.storm.kafka.KafkaSpout.open(KafkaSpout.java:83) at
org.apache.storm.daemon.executor$fn__7990$fn__8005.invoke(executor.clj:604)
at org.apache.storm.util$async_loop$fn__624.invoke(util.clj:482) at
clojure.lang.AFn.run(AFn.java:22) at java.lang.Thread.run(Thread.java:745)

--
Thomas Cristanis

Storm development : Using dependency injection

[Bastien DINE]

Hi everyone,

I'm trying to develop my new topologies using a proper design pattern,  to 
achieve :

-  Reusability of class

-  Unit testing / at least functional

o   Be able to mock database interaction through interfaces



I worked a lot with PHP & Symfony which is a great framework to achieve those 
goals using dependency injection pattern

I want to apply it to Storm topology development, but here is my problem :

How can I pass dependency in constructor (e.g Cassandra provider, or id 
resolver, or even object hydrator), the bolt are instantiated when calling 
"prepare" method
If I'm using a DI framework (like google Guice), how can I Mixed it with storm 
topology builder ?

One idea :
I think I can pass factories to my constructor and instantiate object in 
prepare method of my bolt
But I'm not sure if it a good way to do it..

Did  anybody ever experience it ?
Does anyone have some best practices to develop topologies ? (regarding code 
engineering and organization)

Thanks in advance,
Regards
Bastien

Re: Rebalance in apache Storm

[Jungtaek Lim]

Hi anshu,

It's up to the Spout implementation. In normal, if spout guarantees
at-least once via handling ack, messages will be replayed after rebalancing.

Regarding rebalance operation, I guess relevant issue (and PR) is open:
STORM-2286  which you
might want to take a look. I haven't reviewed the PR but would like to get
it later.

Thanks,
Jungtaek Lim (HeartSaVioR)

2017년 2월 2일 (목) 오후 2:27, anshu shukla 님이 작성:

> Hello all,
>
> I am research student and working on the dynamic scheduler for Storm.
>
> *I just want to know the impact of Re-balance operation on messages loss.
> i.e. What happens to the messages that are not processed within the wait
> time for rebalance ?*
>
> I found that on doing Re-balance operation only workers whose thread
> mapping is changed is being re-started. So I have 2 choices now either To
> reset mapping to all workers or design my logic in such way that minimal
> old mapping is impacted. I was trying to figure out that is there any
> benefit of doing one over the other.
>
> --
> Thanks & Regards,
> Anshu Shukla
>