Hi nick,
Please find attachment for my profiler.properties file
*Start with 1 worker and increase the number of executors first*
What executor you mean here? in my profiler properties attached, i've
set profiler.worker = 1 & profiler.executors = 15
I've try to increase the executor component from rebalance command like
below :
*storm rebalance profiler -n 1 -e splitterBolt=3 -e hbaseBolt=3*
when i check it after rebalance process finish, splitterBolt and
hbaseBolt executor doesn't change. it still have 1 executor.
for your information. the data that proccessed is about 1000 rows in 5
seconds
any suggestion about this pls?
On 01/07/19 19:51, Nick Allen wrote:
I would assume that in those cases where you see the "rebalance... max
poll()" message that topology is unable to keep up with the input
throughput. The messages are not ack'd quickly enough, they fail, and
are retried. I would not focus on HBase because that is not likely
your bottleneck.
You need to tune the Profiler topology to keep up with your peak
incoming throughput. The same ideas for tuning any Storm topology
apply here. Start with 1 worker and increase the number of executors
first. You will also want to explore reducing your window lag and
tuning other parameters before increasing the number of workers. Try
to max out the performance of a single worker before adding more workers.
You will probably first start to see performance issues on the
Splitter bolt that has to consume every message and determine if that
message is needed by any of the profilers. Using the Storm UI watch
the metrics generated for that bolt first.
If asking for more in-depth help, these are the types of questions
that I would ask.
* What is your peak input throughput to the Profiler?
* What are your Profiler properties?
* How many profiles do you have and what are they doing? Provide the
profile definitions.
On Mon, Jul 1, 2019 at 3:04 AM tkg_cangkul <[email protected]
<mailto:[email protected]>> wrote:
Hi, i've a problem about hbase profiler on metron. i've found if
there
are some inconsistentcy data that insert to hbase. sometimes insert
normally but sometimes doesn't inserted. if not inserted, there
are an
error msg on storm (rebalance... Max poll()).
I've set 2 worker for profiler. profiler.hbase.batch 10.
profiler.hbase.flush.interval.seconds 30
any suggest about this? pls help
#
#
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
#
##### Storm #####
topology.worker.childopts=
topology.auto-credentials=[]
profiler.workers=1
profiler.executors=15
topology.message.timeout.secs=1500
topology.max.spout.pending=1000
topology.fall.back.on.java.serialization=true
topology.testing.always.try.serialize=false
topology.kryo.register=[ org.apache.metron.profiler.ProfileMeasurement, \
org.apache.metron.profiler.ProfilePeriod, \
org.apache.metroncommon.configuration.profiler.ProfileResult, \
org.apache.metroncommon.configuration.profiler.ProfileResultExpressions, \
org.apache.metroncommon.configuration.profiler.ProfileTriageExpressions, \
org.apache.metroncommon.configuration.profiler.ProfilerConfig, \
org.apache.metroncommon.configuration.profiler.ProfileConfig, \
org.json.simple.JSONObject, \
org.json.simple.JSONArray, \
java.util.LinkedHashMap, \
org.apache.metron.statistics.OnlineStatisticsProvider ]
##### Profiler #####
profiler.input.topic=indexing
profiler.output.topic=enrichments
profiler.period.duration=10
profiler.period.duration.units=MINUTES
profiler.window.duration=30
profiler.window.duration.units=SECONDS
profiler.ttl=30
profiler.ttl.units=MINUTES
profiler.window.lag=1
profiler.window.lag.units=MINUTES
profiler.max.routes.per.bolt=100000
##### HBase #####
profiler.hbase.salt.divisor=1000
profiler.hbase.table=profiler
profiler.hbase.column.family=P
profiler.hbase.batch=10
profiler.hbase.flush.interval.seconds=30
##### Kafka #####
kafka.zk=zookeeper3.metron.com:2181,zookeeper1.metron.com:2181,zookeeper2.metron.com:2181
kafka.broker=dn1.metron.com:6667,dn3.metron.com:6667,dn2.metron.com:6667
kafka.start=UNCOMMITTED_EARLIEST
