I have installed Metron on single-node HDP 2.5 using this article:
https://community.hortonworks.com/articles/60805/deploying-a-fresh-metron-cluster-using-ambari-serv.html
So far it works just fine except ElasticSearch, I am unable to make it
work properly on single-node, by default ElasticSearch configured for at
least 3 nodes. I have adjusted elasticsearch.yml to mimic single-node
vagrant installation, but still in my cluster health
(:9200/_cluster/health?pretty):
{
"cluster_name" : "metron",
"status" : "yellow",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 1,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 50.0
}
And here is the shards cat :9200/_cat/shards?v:
index shard prirep state docs store ip node
.kibana 0 p STARTED 1 3.1kb 10.10.110.65 Red Guardian
.kibana 0 r UNASSIGNED
Does anyone know why .kibana is trying to allocate two shards and how I
can make both of them allocated on single-node?
Please advise.
Thank you.
- Dima
On 10/09/2016 09:28 PM, Nick Allen wrote:
> Metron is built against Storm 0.10.x but you are trying to run it on Storm
> 1.0.x. Unfortunately that will not work. There is some development and
> testing needed to get Metron running on that version of Storm.
>
> On Oct 9, 2016 10:59 AM, "Dima Kovalyov" <[email protected]> wrote:
>
>> Thank you Nick,
>>
>> I've re-build topologies.
>>
>> Another thing I've noticed after upgrade, my Kafka Broker crashes with
>> started topologies after a while. The only relative to the crash message
>> I was able to find is this:
>>> [2016-10-09 11:30:35,013] ERROR Controller 1001 epoch 14 initiated
>>> state change for partition [parser_invalid,0] from OfflinePartition to
>>> OnlinePartition failed (state.change.logger)
>> I have also attached kafka.tar.gz archive with my kafka logs from the
>> moment of the Kafka Broker start till crash (15 minutes past).
>>
>> When I deploy topologies I receive following message, which I consider
>> is just a warning, not error:
>>> 20382 [main] WARN o.a.s.h.DefaultShader - Relocating
>>> backtype/storm/security/auth/ICredentialsRenewer to
>>> org/apache/storm/security/auth/ICredentialsRenewer in
>>> org/apache/storm/hdfs/common/security/AutoHDFS.class. please modify
>>> your code to use the new namespace
>> Because I have following set in storm.yaml:
>> client.jartransformer.class : 'org.apache.storm.hack.
>> StormShadeTransformer'
>> Which is exactly what you need to set for Storm 1.0+ to work with legacy
>> jars according to: https://storm.apache.org/releases/1.0.0/
>>
>> Last thing, when I start Kafka back again after crash and deploy
>> topologies to Storm I have following message displaying in Storm UI:
>>> Topology spouts lag error
>>> kafkaSpout KAFKA Offset lags for kafka not supported for older
>>> versions. Please update kafka spout to latest version.
>> Does anyone know how to fix topology offset lag or how to update kafka
>> spout to latest version?
>>
>> - Dima
>>
>>
>> On 09/28/2016 03:08 PM, Nick Allen wrote:
>>> You can re-build and deploy to your 'Quick Dev' environment by doing the
>>> following from the root of Metron's source code.
>>>
>>> mvn clean package -DskipTests
>>> cd metron-deployment/vagrant/quick-dev-platform
>>> ./run_enrichment_role.sh
>>>
>>>
>>> On Tue, Sep 27, 2016 at 11:57 PM, Dima Kovalyov <[email protected]
>>>
>>> wrote:
>>>
>>>> Thank you for reply James,
>>>>
>>>> I have performed Ambari 2.2 -> 2.4 and HDP 2.4 -> 2.5 upgrade on
>>>> quick-dev-platform in Vagrant Metron setup. Most of it went just fine,
>>>> except few intermittent errors.
>>>>
>>>> Can you please advise where do you want it to be documented and/or
>>>> discussed?
>>>>
>>>> Right now I am battling issue with Storm. After the upgrade I need to
>>>> re-build Storm Topology jars in order to propagate HDP 2.5 files in
>>>> them. Can you please advise how I can do that on already running setup?
>>>> (TLDR: how to build java jars manually?)
>>>>
>>>> Thank you!
>>>>
>>>> - Dima
>>>>
>>>> On 09/27/2016 06:59 PM, James Sirota wrote:
>>>>> We didn't test it enough to know how well it works. We don't expect it
>>>> to work at this time. If you can run it up for us and document all the
>>>> issues that go wrong that would be greatly appreciated.
>>>>> 26.09.2016, 16:54, "Satish Abburi" <[email protected]>:
>>>>>> All, any one using/tested this combination.
>>>>>>
>>>>>> Thanks,
>>>>>> Satish
>>>>> -------------------
>>>>> Thank you,
>>>>>
>>>>> James Sirota
>>>>> PPMC- Apache Metron (Incubating)
>>>>> jsirota AT apache DOT org
>>>>>
>>
