Writing orc files with storm via java API

[Igor Kuzmenko]

Is there any implementation of storm bolt which can write files to HDFS in
ORC format, without using Hive Streaming API?
I've found java API for writing ORC files 
and I'm guessing is there any existing Hive bolts that uses it or any plans
to create such?

Old kafka spout perfomance tuning

[Igor Kuzmenko]

Storm topology with old kafka spout connected to local kafka show great
performance, which I'm satisfied with. But when I connect to external
kafka, which is located on separate cluster, spout performance
drops significantly. And the same topology works 10 times slower.
I've already tried to increase this spout setting:
- bufferSizeBytes up to 67108864
- fetchSizeBytes up to 67108864
- minFetchBytes up to 67108864
- maxSpoutPending was doubled of old value

All experiments gave me a slight increase in performance, but still far
from the values obtained when connecting to the local kafka. I thought my
network throughput is low, but when I've increased kafka spout parallelism,
storm throughput increased proportionally.

Re: New Kafka Spout doesn't move offset to the latest

[Igor Kuzmenko]

Sounds strange. Can you explain, how it will help me?

2 мар. 2017 г. 10:04 ПП пользователь "Sree V" <sree_at_ch...@yahoo.com>
написал:

> use different topology name and spout id and submit again.
>
>
> Thanking you.
> With Regards
> Sree
>
>
> On Thursday, March 2, 2017 10:54 AM, Igor Kuzmenko <f1she...@gmail.com>
> wrote:
>
>
> I'm using storm-kafka-client 1.1.1-SNAPSHOT build. After topology start
> kafka spout read all partitions from kafka exept one:
>
> IdTopicPartitionLatest OffsetSpout Committed OffsetLag
> Kafka Spout gtp 0 5726714188 5726700216 13972
> Kafka Spout gtp 1 5716936379 5716922137 14242
> Kafka Spout gtp 2 5725709217 5484094447 241614770
> Kafka Spout gtp 3 5713385013 5713370624 14389
> Kafka Spout gtp 4 5721077118 5721062942 14176
> Kafka Spout gtp 5 5717492246 5717478013 14233
> Kafka Spout gtp 6 5716438459 5716424263 14196
> Kafka Spout gtp 7 5719165064 5719150543 14521
>
> Reading partition #2 fails with:
>
> 2017-03-02 23:06:32.013 o.a.k.c.c.i.Fetcher [INFO] [51] [Thread-8-Kafka
> Spout-executor[9 9]] Fetch offset 5484094448 is out of range for partition
> gtp-2, resetting offset
>
> This is common problem for kafka consumer and can be solved by
> setting auto.offset.reset to 'earliest' or 'latest' value. So did I:
>
> props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
> getValueDeserializer().getCanonicalName());
>
>
> After redeploy topology nothing has changed. I still get this info message
> 'offset is out of range' and same commited offset for 2nd partition.
>
>
>

New Kafka Spout doesn't move offset to the latest

[Igor Kuzmenko]

I'm using storm-kafka-client 1.1.1-SNAPSHOT build. After topology start
kafka spout read all partitions from kafka exept one:

IdTopicPartitionLatest OffsetSpout Committed OffsetLag
Kafka Spout gtp 0 5726714188 5726700216 13972
Kafka Spout gtp 1 5716936379 5716922137 14242
Kafka Spout gtp 2 5725709217 5484094447 241614770
Kafka Spout gtp 3 5713385013 5713370624 14389
Kafka Spout gtp 4 5721077118 5721062942 14176
Kafka Spout gtp 5 5717492246 5717478013 14233
Kafka Spout gtp 6 5716438459 5716424263 14196
Kafka Spout gtp 7 5719165064 5719150543 14521

Reading partition #2 fails with:

2017-03-02 23:06:32.013 o.a.k.c.c.i.Fetcher [INFO] [51] [Thread-8-Kafka
Spout-executor[9 9]] Fetch offset 5484094448 is out of range for partition
gtp-2, resetting offset

This is common problem for kafka consumer and can be solved by
setting auto.offset.reset to 'earliest' or 'latest' value. So did I:

props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
getValueDeserializer().getCanonicalName());


After redeploy topology nothing has changed. I still get this info message
'offset is out of range' and same commited offset for 2nd partition.

Re: Kafka Spout enable.auto.commit=false

[Igor Kuzmenko]

Thanks, Hugo. That's all I want to know abot this.

On Tue, Feb 21, 2017 at 9:01 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com>
wrote:

> As per KafkaConsumer documentation
> <https://kafka.apache.org/0100/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html>
> :
>
> “ Setting enable.auto.commit=true means that offsets are committed
> automatically with a frequency controlled by the config
> auto.commit.interval.ms. “
>
> That means that the Spout disregards if the Tuple is acked or not;  every
> time interval (as controlled by auto.commit.interval.ms ) the offset will
> be committed. This can have have an impact on the delivery guarantees,
> because an offset may be committed, yet the tuple may fail.
>
> On Feb 20, 2017, at 8:15 AM, Igor Kuzmenko <f1she...@gmail.com> wrote:
>
> Hello, I'd like to understand difference between auto commit mode
> true/false in new KafkaSpout.
>
> With auto.commit.enabled = false KafkaSpout will move my offset relying on
> acked tuples, it seems easy.
>
> But what happens if I turn auto commit on?
> How Kafka make decision which offset to commit?
>
>
>

Kafka Spout enable.auto.commit=false

[Igor Kuzmenko]

Hello, I'd like to understand difference between auto commit mode
true/false in new KafkaSpout.

With auto.commit.enabled = false KafkaSpout will move my offset relying on
acked tuples, it seems easy.

But what happens if I turn auto commit on?
How Kafka make decision which offset to commit?

Re: Kafka spout stops commiting offsets on some partitions

[Igor Kuzmenko]

I'm using Hortonworks Data Platform 2.5.0.0 with included version of Storm
1.0.1.2.5.0.0-1245. I guess the closest branch in Apache Storm repo is
1.x-branch

On Fri, Feb 17, 2017 at 12:34 AM, Hugo Da Cruz Louro <hlo...@hortonworks.com
> wrote:

> Which version of Storm are you using ?
>
> On Feb 16, 2017, at 12:59 PM, Igor Kuzmenko <f1she...@gmail.com> wrote:
>
> Thanks for reply Hugo.
> I'll double check log tomorrow looking for KafkaSpoutRetryExponentialBackoff
> calls.
>
> I just noticed, that in log I have there's strange thing. First message is
> "*Unexpected offset found [2777849]*". It's strange because if you look
> on partition 10 commited offset, it is 2777978 which is a little bit
> higher then offset found. The next message in log was "*No offsets ready
> to commit.*"
>
> So, after checking *2777849 * offset it immediately stoped seeking new
> offset to commit.
>
> On Thu, Feb 16, 2017 at 8:23 PM, Hugo Da Cruz Louro <
> hlo...@hortonworks.com> wrote:
>
>> Hi,
>>
>> Most likely this is happening because some messages failed and/or got
>> acked out of order.
>>
>> For example, if you process messages with offsets 1,2,3,X,5,6,7,… where X
>> is message (with offset 4) that failed, the Spout will only commit offset
>> 3. Until the message with offset 4 is acked, or reaches max number of
>> retrials (which is configurable but by default is forever), the messages
>> with offsets 5,6,7,… will not get committed despite having been acked. That
>> is because you cannot do kafkaConsumer.commitSync(new
>> TopicPartion(test_topic,5)) if the message with offset 4 has not been acked
>> or discarded by reaching the max number of retrials. Until the spout moves
>> on from message with offset 4, the lag will increase when new messages come
>> in.
>>
>> You can try enabling the log level to ALL for
>> org.apache.storm.kafka.spout.KafkaSpoutRetryExponentialBackoff to see
>> which messages are getting retried. You can also set log level to DEBUG or
>> ALL org.apache.storm.kafka.spout.KafkaSpout to see exactly which
>> offsets/records are being processed. However, it will print a lot of
>> messages, and may slow down processing considerably.
>>
>> You can also set the maxNumberOfRetires to a small number (e.g. 3-5) to
>> see if that solves this situation.
>>
>> Hugo
>>
>> > On Feb 16, 2017, at 8:36 AM, Igor Kuzmenko <f1she...@gmail.com> wrote:
>> >
>> > Today in Storm UI I saw this Kafka Spouts Lag:
>> > IdTopic   Partition   Latest Offset   Spout
>> Committed Offset  Lag
>> > Kafka Spout   test_topic  0   5591087 5562814
>>28273
>> > Kafka Spout   test_topic  1   2803256 2789090
>>14166
>> > Kafka Spout   test_topic  2   2801927 2787767
>>14160
>> > Kafka Spout   test_topic  3   2800627 2800626
>>1
>> > Kafka Spout   test_topic  4   2799391 2785238
>>14153
>> > Kafka Spout   test_topic  5   2798126 2798125
>>1
>> > Kafka Spout   test_topic  6   2796874 2782726
>>14148
>> > Kafka Spout   test_topic  7   2795669 2781528
>>14141
>> > Kafka Spout   test_topic  8   2794419 2780280
>>14139
>> > Kafka Spout   test_topic  9   2793255 2793254
>>1
>> > Kafka Spout   test_topic  10  2792109 2777978
>>14131
>> > Kafka Spout   test_topic  11  2790939 2776817
>>14122
>> > Kafka Spout   test_topic  12  2789783 2775665
>>14118
>> > Kafka Spout   test_topic  13  2788651 2774539
>>14112
>> > Kafka Spout   test_topic  14  2787521 2773412
>>14109
>> >
>> >
>> > There was no new messages in that topic for a while, so I expected,
>> that my topology would process all messages. But lag shows me that there's
>> some uncommitted messages in most of topics. Topology stop working and
>> didn't process any messages for few ho

Re: Kafka spout stops commiting offsets on some partitions

[Igor Kuzmenko]

Thanks for reply Hugo.
I'll double check log tomorrow looking for KafkaSpoutRetryExponentialBackoff
calls.

I just noticed, that in log I have there's strange thing. First
message is "*Unexpected
offset found [2777849]*". It's strange because if you look on partition 10
commited offset, it is 2777978 which is a little bit higher then offset
found. The next message in log was "*No offsets ready to commit.*"

So, after checking *2777849 *offset it immediately stoped seeking new
offset to commit.

On Thu, Feb 16, 2017 at 8:23 PM, Hugo Da Cruz Louro <hlo...@hortonworks.com>
wrote:

> Hi,
>
> Most likely this is happening because some messages failed and/or got
> acked out of order.
>
> For example, if you process messages with offsets 1,2,3,X,5,6,7,… where X
> is message (with offset 4) that failed, the Spout will only commit offset
> 3. Until the message with offset 4 is acked, or reaches max number of
> retrials (which is configurable but by default is forever), the messages
> with offsets 5,6,7,… will not get committed despite having been acked. That
> is because you cannot do kafkaConsumer.commitSync(new
> TopicPartion(test_topic,5)) if the message with offset 4 has not been acked
> or discarded by reaching the max number of retrials. Until the spout moves
> on from message with offset 4, the lag will increase when new messages come
> in.
>
> You can try enabling the log level to ALL for org.apache.storm.kafka.spout.
> KafkaSpoutRetryExponentialBackoff to see which messages are getting
> retried. You can also set log level to DEBUG or ALL
> org.apache.storm.kafka.spout.KafkaSpout to see exactly which
> offsets/records are being processed. However, it will print a lot of
> messages, and may slow down processing considerably.
>
> You can also set the maxNumberOfRetires to a small number (e.g. 3-5) to
> see if that solves this situation.
>
> Hugo
>
> > On Feb 16, 2017, at 8:36 AM, Igor Kuzmenko <f1she...@gmail.com> wrote:
> >
> > Today in Storm UI I saw this Kafka Spouts Lag:
> > IdTopic   Partition   Latest Offset   Spout
> Committed Offset  Lag
> > Kafka Spout   test_topic  0   5591087 5562814
>  28273
> > Kafka Spout   test_topic  1   2803256 2789090
>  14166
> > Kafka Spout   test_topic  2   2801927 2787767
>  14160
> > Kafka Spout   test_topic  3   2800627 2800626
>  1
> > Kafka Spout   test_topic  4   2799391 2785238
>  14153
> > Kafka Spout   test_topic  5   2798126 2798125
>  1
> > Kafka Spout   test_topic  6   2796874 2782726
>  14148
> > Kafka Spout   test_topic  7   2795669 2781528
>  14141
> > Kafka Spout   test_topic  8   2794419 2780280
>  14139
> > Kafka Spout   test_topic  9   2793255 2793254
>  1
> > Kafka Spout   test_topic  10  2792109 2777978
>  14131
> > Kafka Spout   test_topic  11  2790939 2776817
>  14122
> > Kafka Spout   test_topic  12  2789783 2775665
>  14118
> > Kafka Spout   test_topic  13  2788651 2774539
>  14112
> > Kafka Spout   test_topic  14  2787521 2773412
>  14109
> >
> >
> > There was no new messages in that topic for a while, so I expected, that
> my topology would process all messages. But lag shows me that there's some
> uncommitted messages in most of topics. Topology stop working and didn't
> process any messages for few hours.
> >
> > In logs I found these messages:
> > 2017-02-16 14:50:20.187 o.a.s.k.s.KafkaSpout [DEBUG] Unexpected offset
> found [2777849]. OffsetEntry{topic-partition=test_topic-10,
> fetchOffset=2775755, committedOffset=2777978, 
> ackedMsgs=[{topic-partition=test_topic-10,
> offset=2777849, numFails=0}, {topic-partition=test_topic-10,
> offset=2777850, numFails=0},
> > 
> >  {topic-partition=test_topic-10, offset=2792107, numFails=0},
> {topic-partition=test_topic-10, offset=2792108, numFails=0}]}
> > 2017-02-16 14:50:20.201 o.a.s.k.s.KafkaSpout [DEBUG] No offsets ready to
> commit. OffsetEntry{topic-partition=test_topic-10, fetchOffset=2775755,
> committedOffset=277797

Kafka spout stops commiting offsets on some partitions

[Igor Kuzmenko]

Today in Storm UI I saw this Kafka Spouts Lag:
Id Topic Partition Latest Offset Spout Committed Offset Lag
Kafka Spout test_topic 0 5591087 5562814 28273
Kafka Spout test_topic 1 2803256 2789090 14166
Kafka Spout test_topic 2 2801927 2787767 14160
Kafka Spout test_topic 3 2800627 2800626 1
Kafka Spout test_topic 4 2799391 2785238 14153
Kafka Spout test_topic 5 2798126 2798125 1
Kafka Spout test_topic 6 2796874 2782726 14148
Kafka Spout test_topic 7 2795669 2781528 14141
Kafka Spout test_topic 8 2794419 2780280 14139
Kafka Spout test_topic 9 2793255 2793254 1
Kafka Spout test_topic 10 2792109 2777978 14131
Kafka Spout test_topic 11 2790939 2776817 14122
Kafka Spout test_topic 12 2789783 2775665 14118
Kafka Spout test_topic 13 2788651 2774539 14112
Kafka Spout test_topic 14 2787521 2773412 14109


There was no new messages in that topic for a while, so I expected, that my
topology would process all messages. But lag shows me that there's some
uncommitted messages in most of topics. Topology stop working and didn't
process any messages for few hours.

In logs I found these messages:
2017-02-16 14:50:20.187 o.a.s.k.s.KafkaSpout [DEBUG] Unexpected offset
found [2777849]. OffsetEntry{topic-partition=test_topic-10,
fetchOffset=2775755,
committedOffset=2777978, ackedMsgs=[{topic-partition=test_topic-10,
offset=2777849, numFails=0}, {topic-partition=test_topic-10,
offset=2777850, numFails=0},

 {topic-partition=test_topic-10, offset=2792107, numFails=0},
{topic-partition=test_topic-10, offset=2792108, numFails=0}]}
2017-02-16 14:50:20.201 o.a.s.k.s.KafkaSpout [DEBUG] No offsets ready to
commit. OffsetEntry{topic-partition=test_topic-10, fetchOffset=2775755,
committedOffset=2777978, ackedMsgs=[{topic-partition=test_topic-10,
offset=2777849, numFails=0},
...
{topic-partition=test_topic-10, offset=2792108, numFails=0}]}


So, I assume, messages, that showed as uncommitted, are actually processed
by topology and acked. After I start sending new messages to Kafka topic
topology start working, but spout lag increasing.
Why spout could stop committing to Kafka?

Re: Kafka monitor unable to get offset lag

[Igor Kuzmenko]

Yes running from command line gives me json with topic lag.
What is secured setup?

On Wed, Feb 1, 2017 at 10:28 PM, Priyank Shah <ps...@hortonworks.com> wrote:

> Hi Igor,
>
>
>
> Just to clarify, are you saying that running storm-kafka-monitor from
> command line with the arguments you pasted returns the spout lags correctly
> ? Also, can you confirm if you are running in a secured setup or not ?
>
>
>
> *From: *Igor Kuzmenko <f1she...@gmail.com>
> *Reply-To: *"user@storm.apache.org" <user@storm.apache.org>
> *Date: *Wednesday, February 1, 2017 at 5:05 AM
> *To: *"user@storm.apache.org" <user@storm.apache.org>
> *Subject: *Re: Kafka monitor unable to get offset lag
>
>
>
> Found in logs with debug level:
>
> 2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] spout classname:
> org.apache.storm.kafka.spout.KafkaSpout
>
> 2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] json
> configuration: {config.bootstrap.servers=dn-06.ru:6667,dn-10.ru:6667,
> config.topics=gtp,, config.groupid=test, topology.tasks=1}
>
> 2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] Command to run:
> [/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor, -t, gtp,, -g, test,
> -b, dn-06.ru:6667,dn-10.ru:6667]
>
> 2017-02-01 14:07:36.338 o.a.s.u.TopologySpoutLag [DEBUG] JSON parsing
> failed, assuming message as error message: Unable to get offset lags for
> kafka. Reason: org.apache.kafka.shaded.common.errors.TimeoutException:
> Timeout expired while fetching topic metadata
>
>
>
> Running command /usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t
> gtp -g test -b dn-06.ru:6667,dn-10.ru:6667  returns correct json with
> spout lags. There's no timeout command executing in a second.
>
>
>
>
>
>
>
> On Wed, Feb 1, 2017 at 12:25 PM, Spico Florin <spicoflo...@gmail.com>
> wrote:
>
> Hello!
>
>   You can check how your topic is consumed  and its health via the scripts:
>
> ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> :6667 --list
>
>  ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> :6667 --describe --group 
>
>
>
> I hope it helps.
>
>  Florin
>
>
>
> On Wed, Feb 1, 2017 at 11:01 AM, Igor Kuzmenko <f1she...@gmail.com> wrote:
>
> Yes, topology process data and works fine.
>
> I couldn't find any exceptions in storm logs.   access-web-ui.log contains
> only these lines
>
>
>
> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
>
> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary
> principal:
>
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
>
> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>
>
>
> On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <ps...@hortonworks.com>
> wrote:
>
> Hi Igor,
>
>
>
> When you say topology is working fine do you mean you see data flowing?
> Can you try to look up the logs for ui server and paste relevant lines here
> if any?
>
>
>
> Priyank
>
> Sent from my iPhone
>
>
> On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1she...@gmail.com> wrote:
>
> I've launched topology with new kafka spout. Topology by it self working
> fine, but looking at storm UI I see kafka-monitor exception:
>
> *Unable to get offset lags for kafka. Reason:
> org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired
> while fetching topic metadata*
>
>
>
> Maybe I forgot to configure something, but then how topology reads
> messages?
>
>
>
>
>
>
>

Re: Kafka monitor unable to get offset lag

[Igor Kuzmenko]

Found in logs with debug level:
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] spout classname:
org.apache.storm.kafka.spout.KafkaSpout
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] json
configuration: {config.bootstrap.servers=dn-06.ru:6667,dn-10.ru:6667,
config.topics=gtp,, config.groupid=test, topology.tasks=1}
2017-02-01 14:06:54.053 o.a.s.u.TopologySpoutLag [DEBUG] Command to run:
[/usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor, -t, gtp,, -g, test,
-b, dn-06.ru:6667,dn-10.ru:6667]
2017-02-01 14:07:36.338 o.a.s.u.TopologySpoutLag [DEBUG] JSON parsing
failed, assuming message as error message: Unable to get offset lags for
kafka. Reason: org.apache.kafka.shaded.common.errors.TimeoutException:
Timeout expired while fetching topic metadata

Running command /usr/hdp/2.5.0.0-1245/storm/bin/storm-kafka-monitor -t gtp
-g test -b dn-06.ru:6667,dn-10.ru:6667  returns correct json with spout
lags. There's no timeout command executing in a second.



On Wed, Feb 1, 2017 at 12:25 PM, Spico Florin <spicoflo...@gmail.com> wrote:

> Hello!
>   You can check how your topic is consumed  and its health via the scripts:
> ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> :6667 --list
>  ./kafka-consumer-groups.sh --new-consumer --bootstrap-server
> :6667 --describe --group 
>
> I hope it helps.
>  Florin
>
> On Wed, Feb 1, 2017 at 11:01 AM, Igor Kuzmenko <f1she...@gmail.com> wrote:
>
>> Yes, topology process data and works fine.
>> I couldn't find any exceptions in storm logs.   access-web-ui.log
>> contains only these lines
>>
>> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary
>> principal:
>> 2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary
>> principal:
>> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary
>> principal:
>> 2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
>> 10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
>>
>> On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <ps...@hortonworks.com>
>> wrote:
>>
>>> Hi Igor,
>>>
>>> When you say topology is working fine do you mean you see data flowing?
>>> Can you try to look up the logs for ui server and paste relevant lines here
>>> if any?
>>>
>>> Priyank
>>>
>>> Sent from my iPhone
>>>
>>> On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1she...@gmail.com> wrote:
>>>
>>> I've launched topology with new kafka spout. Topology by it self working
>>> fine, but looking at storm UI I see kafka-monitor exception:
>>> *Unable to get offset lags for kafka. Reason:
>>> org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired
>>> while fetching topic metadata*
>>>
>>> Maybe I forgot to configure something, but then how topology reads
>>> messages?
>>>
>>>
>>
>

Re: Kafka monitor unable to get offset lag

[Igor Kuzmenko]

Yes, topology process data and works fine.
I couldn't find any exceptions in storm logs.   access-web-ui.log contains
only these lines

2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:17.315 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/topology/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/cluster/summary principal:
2017-02-01 11:58:25.346 o.a.s.l.f.AccessLoggingFilter [INFO] Access from:
10.35.63.14 url: http://master001.s:8744/api/v1/nimbus/summary principal:

On Tue, Jan 31, 2017 at 5:19 PM, Priyank Shah <ps...@hortonworks.com> wrote:

> Hi Igor,
>
> When you say topology is working fine do you mean you see data flowing?
> Can you try to look up the logs for ui server and paste relevant lines here
> if any?
>
> Priyank
>
> Sent from my iPhone
>
> On Jan 31, 2017, at 4:34 AM, Igor Kuzmenko <f1she...@gmail.com> wrote:
>
> I've launched topology with new kafka spout. Topology by it self working
> fine, but looking at storm UI I see kafka-monitor exception:
> *Unable to get offset lags for kafka. Reason:
> org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired
> while fetching topic metadata*
>
> Maybe I forgot to configure something, but then how topology reads
> messages?
>
>

Kafka monitor unable to get offset lag

[Igor Kuzmenko]

I've launched topology with new kafka spout. Topology by it self working
fine, but looking at storm UI I see kafka-monitor exception:
*Unable to get offset lags for kafka. Reason:
org.apache.kafka.shaded.common.errors.TimeoutException: Timeout expired
while fetching topic metadata*

Maybe I forgot to configure something, but then how topology reads messages?

Re: Kafka spout stops emmiting messages

[Igor Kuzmenko]

Thanks for reply, Josh.
My maxUncommitedOffset was 250, increasing uncommited offset helped me, but
I still don't understand why spout completly stoped emiting tuples. You
said, that eventualy spout will produce new tuples, after old one will be
acked, but in my case didn't.




On Tue, Jan 24, 2017 at 4:24 AM, fanxi...@travelsky.com <
fanxi...@travelsky.com> wrote:

> Hi, Kuzmenko：
>
> please pay attention to the number about setMaxUncommittedOffsets, if this
> number is too small, the spout may stop  emitting until the pending tuple
> is acked by the down bolt. You can change the number to a large number.
>
> ------
> Josh
>
> *From:* Igor Kuzmenko <f1she...@gmail.com>
> *Date:* 2017-01-24 02:28
> *To:* user <user@storm.apache.org>
> *Subject:* Kafka spout stops emmiting messages
> Hello, I'm trying to upgrade my topology from old kafka spout
> <https://github.com/apache/storm/tree/v1.0.1/external/storm-kafka> (storm-
> kafka project) to new one
> <https://github.com/apache/storm/tree/v1.0.1/external/storm-kafka-client>
> (storm-kafka-client) version 1.0.1. I've configured new spout to work
> with my topology. After deploy it processes and acks few hundreds of
> tuples and then stops. Kafka topic definitely have new messages, and in
> storm UI I can see  kafka spout lag increasing. What could be the problem?
>
>

Kafka spout stops emmiting messages

[Igor Kuzmenko]

Hello, I'm trying to upgrade my topology from old kafka spout
 (storm-
kafka project) to new one

(storm-kafka-client) version 1.0.1. I've configured new spout to work with
my topology. After deploy it processes and acks few hundreds of tuples and
then stops. Kafka topic definitely have new messages, and in storm UI I can
see  kafka spout lag increasing. What could be the problem?

Re: Massive Number of Spout Failures

[Igor Kuzmenko]

We have such fails with two reasons:

1) Bolt doesn't ack tuple immidiatly, but collects a batch and at some
point ack's them all. In that case thes situation when batch bigger than
max_spout_pending and some tuples fails.

2) Bolt doesn't ack tuple at all. Make sure Bolt acks or fails tuples
without any exclusions.

On Wed, Jul 27, 2016 at 10:22 PM, Kevin Peek  wrote:

> We have a topology that is experiencing massive amounts of spout failures
> without corresponding bolt failures. We have been interpreting these as
> tuple timeouts, but we seem to be getting more of these failures than we
> understand to be possible with timeouts.
>
> Our topology uses a Kafka spout and the topology is configured with:
> topology.message.timeout.secs = 300
> topology.max.spout.pending = 2500
>
> Based on these settings, I would expect the topology to experience a
> maximum of 2500 tuple timeouts per 300 seconds. But from the Storm UI, we
> see that after running for about 10 minutes, the topology will show about
> 50K spout failures and zero bolt failures.
>
> Am I misunderstanding something that would allow more tuples to time out,
> or is there another source of spout failures?
>
> Thanks in advance,
> Kevin Peek
>

Max spout pending blocks tick tuple?

[Igor Kuzmenko]

Hello, I'd like to know does max_spout_pending setting affect emmiting tick
tuple?
The case is when I reached max_spout_pending value and spout stop emiting
new tuples, does toplogy stop emmiting tick tuples?

Where kafka spout stores offset

[Igor Kuzmenko]

I'm using Storm v 0.10.0. Offset obviously stored in zookeeper, but in wich
one?

First zookeeper I set in STORM settings: "storm.zookeeper.servers".
Second one is set in ZkHosts object, which is part of SpoutConfig: public
ZkHosts(String brokerZkStr)

Right now they are the same, in my case, but I'd like to know, what happens
if they will differ? Where offset will be stored?

Re: Topology code distribution takes too much time

[Igor Kuzmenko]

Thanks for replay, guys.
While deploying topology, CPU load is arround zero. When topology deployed
and started to work it raises up to 25%, so its not CPU overload or cgroup.
The network is able to load 150MB in less than a second, i checked it with
SCP. When deploying topology theres no major network activity, whatched it
with ambari.
JVMs settings are storm defaults except worker Xmx increased up to 2048m
(768 default). No OutOfMemory or other kind of exceptions was thrown in a
process.

I doubt that it's hardware issue, cause all worked fine with old version of
HDP.

On Sun, Mar 27, 2016 at 11:22 PM, Erik Weathers <eweath...@groupon.com>
wrote:

> Igor, *maybe* you have the supervisor in a cgroup that is limiting the CPU
> for the process?  (Don't know anything about the HDP packaged storm)
>
> - Erik
>
> On Sun, Mar 27, 2016 at 11:30 AM, Andrey Dudin <doodin...@gmail.com>
> wrote:
>
>> Hi Igor.
>> Try to dump threads and look on it. I think you can find problem in dump.
>>
>> Look at lan, cpu and ram load. Maybe you CPU overloaded or ram and lan.
>>
>> 2016-03-27 21:26 GMT+03:00 Igor Kuzmenko <f1she...@gmail.com>:
>>
>>> Hello, I'm using Hortonworks Data Platform v2.3.4 with included storm
>>> 0.10.
>>> After deploying my topology using "*storm jar*" command it takes about
>>> 3 min to distribute code. The network is 10Gb/s, topology jar is about
>>> 150MB, cluster has 2 nodes with supervisors on them, so I assume that this
>>> process shouldn't take much time. Actualy, it wasn't with Hortonworks
>>> platform 2.3.0, I can't understand what happend since than. Heres change
>>> log
>>> <http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.4/bk_HDP_RelNotes/content/patch_storm.html>
>>>  of
>>> HDP, hope you can help.
>>>
>>> Here's logs.
>>> Nimbus download jar pretty fast:
>>> *2016-02-25 17:10:13.040 b.s.d.nimbus [INFO] Uploading file from client
>>> to
>>> /opt/hadoop/storm/nimbus/inbox/stormjar-7352b097-8829-4268-a81f-29d820c0f311.jar*
>>> *2016-02-25 17:10:14.915 b.s.d.nimbus [INFO] Finished uploading file
>>> from client:
>>> /opt/hadoop/storm/nimbus/inbox/stormjar-7352b097-8829-4268-a81f-29d820c0f311.jar*
>>> 2016-02-25 17:10:14.928 b.s.d.nimbus [INFO] [req 22] Access from:
>>> principal: op:submitTopology
>>> 2016-02-25 17:10:14.933 b.s.d.nimbus [INFO] Received topology submission
>>> for CdrPerformanceTestTopology-phoenix-bolt-4 with conf
>>> {"topology.max.task.parallelism" nil, "hbase.conf" {"hbase.rootdir" "hdfs://
>>> sorm-master02.msk.mts.ru:8020/apps/hbase/data"},
>>> "topology.submitter.principal" "", "topology.acker.executors" nil,
>>> "topology.workers" 2, "topology.message.timeout.secs" 30, "topology.debug"
>>> true, "topology.max.spout.pending" 1, "storm.zookeeper.superACL" nil,
>>> "topology.users" (), "topology.submitter.user" "", "topology.kryo.register"
>>> {"org.apache.avro.util.Utf8" nil,
>>> "ru.mts.sorm.schemes.avro.cdr.CbossCdrRecord" nil},
>>> "topology.kryo.decorators" (), "storm.id"
>>> "CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414", "topology.name"
>>> "CdrPerformanceTestTopology-phoenix-bolt-4"}
>>> 2016-02-25 17:10:14.936 b.s.d.nimbus [INFO] nimbus file
>>> location:/opt/hadoop/storm/nimbus/stormdist/CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414
>>> 2016-02-25 17:10:15.160 b.s.c.LocalFileSystemCodeDistributor [INFO]
>>> Created meta file
>>> /opt/hadoop/storm/nimbus/stormdist/CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414/storm-code-distributor.meta
>>> upload successful.
>>> 2016-02-25 17:10:15.164 b.s.zookeeper [INFO] Node already in queue for
>>> leader lock.
>>> 2016-02-25 17:10:15.165 b.s.d.nimbus [INFO] desired replication count of
>>> 1 not achieved but we have hit the max wait time 60 so moving on with
>>> replication count = 1
>>> 2016-02-25 17:10:15.169 b.s.d.nimbus [INFO] Activating
>>> CdrPerformanceTestTopology-phoenix-bolt-4:
>>> CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414
>>> 2016-02-25 17:10:15.186 b.s.s.EvenScheduler [INFO] Available slots:
>>> (["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6700]
>>> ["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6701]
>>> ["0a6e4de5-dd83-4307-b7e6-1db60c4c0898&q

Topology code distribution takes too much time

[Igor Kuzmenko]

Hello, I'm using Hortonworks Data Platform v2.3.4 with included storm 0.10.
After deploying my topology using "*storm jar*" command it takes about 3
min to distribute code. The network is 10Gb/s, topology jar is about 150MB,
cluster has 2 nodes with supervisors on them, so I assume that this process
shouldn't take much time. Actualy, it wasn't with Hortonworks platform
2.3.0, I can't understand what happend since than. Heres change log

of
HDP, hope you can help.

Here's logs.
Nimbus download jar pretty fast:
*2016-02-25 17:10:13.040 b.s.d.nimbus [INFO] Uploading file from client to
/opt/hadoop/storm/nimbus/inbox/stormjar-7352b097-8829-4268-a81f-29d820c0f311.jar*
*2016-02-25 17:10:14.915 b.s.d.nimbus [INFO] Finished uploading file from
client:
/opt/hadoop/storm/nimbus/inbox/stormjar-7352b097-8829-4268-a81f-29d820c0f311.jar*
2016-02-25 17:10:14.928 b.s.d.nimbus [INFO] [req 22] Access from:
principal: op:submitTopology
2016-02-25 17:10:14.933 b.s.d.nimbus [INFO] Received topology submission
for CdrPerformanceTestTopology-phoenix-bolt-4 with conf
{"topology.max.task.parallelism" nil, "hbase.conf" {"hbase.rootdir" "hdfs://
sorm-master02.msk.mts.ru:8020/apps/hbase/data"},
"topology.submitter.principal" "", "topology.acker.executors" nil,
"topology.workers" 2, "topology.message.timeout.secs" 30, "topology.debug"
true, "topology.max.spout.pending" 1, "storm.zookeeper.superACL" nil,
"topology.users" (), "topology.submitter.user" "", "topology.kryo.register"
{"org.apache.avro.util.Utf8" nil,
"ru.mts.sorm.schemes.avro.cdr.CbossCdrRecord" nil},
"topology.kryo.decorators" (), "storm.id"
"CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414", "topology.name"
"CdrPerformanceTestTopology-phoenix-bolt-4"}
2016-02-25 17:10:14.936 b.s.d.nimbus [INFO] nimbus file
location:/opt/hadoop/storm/nimbus/stormdist/CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414
2016-02-25 17:10:15.160 b.s.c.LocalFileSystemCodeDistributor [INFO] Created
meta file
/opt/hadoop/storm/nimbus/stormdist/CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414/storm-code-distributor.meta
upload successful.
2016-02-25 17:10:15.164 b.s.zookeeper [INFO] Node already in queue for
leader lock.
2016-02-25 17:10:15.165 b.s.d.nimbus [INFO] desired replication count of 1
not achieved but we have hit the max wait time 60 so moving on with
replication count = 1
2016-02-25 17:10:15.169 b.s.d.nimbus [INFO] Activating
CdrPerformanceTestTopology-phoenix-bolt-4:
CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414
2016-02-25 17:10:15.186 b.s.s.EvenScheduler [INFO] Available slots:
(["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6700]
["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6701]
["0a6e4de5-dd83-4307-b7e6-1db60c4c0898" 6700]
["0a6e4de5-dd83-4307-b7e6-1db60c4c0898" 6701])
2016-02-25 17:10:15.189 b.s.d.nimbus [INFO] Setting new assignment for
topology id CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414:
#backtype.storm.daemon.common.Assignment{:master-code-dir
"/opt/hadoop/storm/nimbus/stormdist/CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414",
:node->host {"edef7b4d-2358-41c6-8a03-a7638d0f68c6" "sorm-data06.msk.mts.ru",
"0a6e4de5-dd83-4307-b7e6-1db60c4c0898" "sorm-data07.msk.mts.ru"},
:executor->node+port {[8 8] ["0a6e4de5-dd83-4307-b7e6-1db60c4c0898" 6700],
[2 2] ["0a6e4de5-dd83-4307-b7e6-1db60c4c0898" 6700], [7 7]
["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6700], [3 3]
["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6700], [1 1]
["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6700], [6 6]
["0a6e4de5-dd83-4307-b7e6-1db60c4c0898" 6700], [9 9]
["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6700], [11 11]
["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6700], [5 5]
["edef7b4d-2358-41c6-8a03-a7638d0f68c6" 6700], [10 10]
["0a6e4de5-dd83-4307-b7e6-1db60c4c0898" 6700], [4 4]
["0a6e4de5-dd83-4307-b7e6-1db60c4c0898" 6700]}, :executor->start-time-secs
{[8 8] 1456409415, [2 2] 1456409415, [7 7] 1456409415, [3 3] 1456409415, [1
1] 1456409415, [6 6] 1456409415, [9 9] 1456409415, [11 11] 1456409415, [5
5] 1456409415, [10 10] 1456409415, [4 4] 1456409415}}

But supervisor stuck somewhere:
*2016-02-25 17:10:15.198 b.s.d.supervisor [INFO] Downloading code for storm
id* CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414 from
/opt/hadoop/storm/nimbus/stormdist/CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414
2016-02-25 17:10:15.199 b.s.u.StormBoundedExponentialBackoffRetry [INFO]
The baseSleepTimeMs [2000] the maxSleepTimeMs [6] the maxRetries [5]
2016-02-25 17:10:15.205 b.s.u.StormBoundedExponentialBackoffRetry [INFO]
The baseSleepTimeMs [2000] the maxSleepTimeMs [6] the maxRetries [5]
2016-02-25 17:10:17.405 b.s.c.LocalFileSystemCodeDistributor [INFO]
Attempting to download meta file
/opt/hadoop/storm/nimbus/stormdist/CdrPerformanceTestTopology-phoenix-bolt-4-2-1456409414/stormjar.jar
from remote sorm-master02.msk.mts.ru:6627
2016-02-25 *17:10:17.406* 

Re: Storm creates lots of .tmp files

[Igor Kuzmenko]

Thats it, thanks.

On Thu, Mar 24, 2016 at 11:10 AM, Deepak Sharma <deepakmc...@gmail.com>
wrote:

> There was JIRA raised against this for Hive and this was already fixed as
> suggested in Erik's search result.
>
> --Deepak
>
> On Thu, Mar 24, 2016 at 1:30 PM, Erik Weathers <eweath...@groupon.com>
> wrote:
>
>> How about this question that I asked?
>>
>> > Do you have some storm configuration setting pointing at /tmp/storm?
>>
>> Those files don't seem specific to storm-core -- seems this is some cruft
>> from a topology.
>>
>> Searching google for "pipeout file" implies they might be coming from
>> Hive:
>>
>>-
>>
>> https://www.google.com/webhp?sourceid=chrome-instant=1=2=UTF-8#safe=off=pipeout%20file
>>
>> So do you have topologies using Hive?
>>
>> - Erik
>>
>> On Thu, Mar 24, 2016 at 12:45 AM, Igor Kuzmenko <f1she...@gmail.com>
>> wrote:
>>
>>> I'm using storm 0.10.0
>>> I recived few files from our system administrator, that's not actualy
>>> .tmp files. They are all zero sized and named like this:
>>>
>>> *000f9b90-1057-458a-a979-a52fe89042a44788427259950756302.pipeout*
>>>
>>> *00064dd7-d565-4211-99e8-c9dc78fac0ff7990926303561578787.pipeout*
>>> *00085e62-7012-4835-b646-55407e3330c68985771934554548937.pipeout*
>>>
>>> On Wed, Mar 23, 2016 at 10:21 PM, Erik Weathers <eweath...@groupon.com>
>>> wrote:
>>>
>>>> hi Igor,
>>>>
>>>> What is an example such file?  (Maybe they are ".so" files that the
>>>> java libraries you are using are stuffing there?  e.g., the
>>>> org.xerial.snappy:snappy-java library spews .so files all over /tmp for 
>>>> me.)
>>>>
>>>> What is the version of storm you are using?
>>>>
>>>> Do you have some storm configuration setting pointing at /tmp/storm?
>>>>
>>>> - Erik
>>>>
>>>> On Wed, Mar 23, 2016 at 9:29 AM, Igor Kuzmenko <f1she...@gmail.com>
>>>> wrote:
>>>>
>>>>> Today I get "*java.io.IOException: No space left on device*" because
>>>>> there was no more inodes left. The reason is that I've got a thousands of
>>>>> .tmp files under /tmp/storm created by storm. Why theres so much and what
>>>>> for?
>>>>>
>>>>>
>>>>
>>>
>>
>
>
> --
> Thanks
> Deepak
> www.bigdatabig.com
> www.keosha.net
>

Re: Storm creates lots of .tmp files

[Igor Kuzmenko]

I'm using storm 0.10.0
I recived few files from our system administrator, that's not actualy .tmp
files. They are all zero sized and named like this:

*000f9b90-1057-458a-a979-a52fe89042a44788427259950756302.pipeout*

*00064dd7-d565-4211-99e8-c9dc78fac0ff7990926303561578787.pipeout*
*00085e62-7012-4835-b646-55407e3330c68985771934554548937.pipeout*

On Wed, Mar 23, 2016 at 10:21 PM, Erik Weathers <eweath...@groupon.com>
wrote:

> hi Igor,
>
> What is an example such file?  (Maybe they are ".so" files that the java
> libraries you are using are stuffing there?  e.g., the
> org.xerial.snappy:snappy-java library spews .so files all over /tmp for me.)
>
> What is the version of storm you are using?
>
> Do you have some storm configuration setting pointing at /tmp/storm?
>
> - Erik
>
> On Wed, Mar 23, 2016 at 9:29 AM, Igor Kuzmenko <f1she...@gmail.com> wrote:
>
>> Today I get "*java.io.IOException: No space left on device*" because
>> there was no more inodes left. The reason is that I've got a thousands of
>> .tmp files under /tmp/storm created by storm. Why theres so much and what
>> for?
>>
>>
>

Storm creates lots of .tmp files

[Igor Kuzmenko]

Today I get "*java.io.IOException: No space left on device*" because there
was no more inodes left. The reason is that I've got a thousands of .tmp
files under /tmp/storm created by storm. Why theres so much and what for?