Hi,
After mooving NiFi on another server, Kafka correctly receive data contains
in my text file.
But i have a new error (Log from Storm UI):
"java.lang.IllegalStateException: Grok parser Error: Grok statement
produced a null message. Original message was: tata:titi and the parsed
message was: {} . Check the pattern at: /apps/metron/patterns/test_txt on
tata:titi"
It seem my parser is incorrect, but i had tested it on GrokConstructor and
he's very simple...Has anyone already had this error ?
*Metron node configuration :*
*Parser :*
/apps/metron/patterns/test_txt
TEST_TXT %{DATA:test1}\:%{DATA:test2}
*Parser configuration :*
/usr/metron/0.4.0/config/zookeeper/parsers/test_txt.json
{
"parserClassName": "org.apache.metron.parsers.GrokParser",
"sensorTopic": "test_txt",
"parserConfig": {
"grokPath": "/apps/metron/patterns/test_txt",
"patternLabel": "TEST_TXT"
}
}
2017-07-19 22:59 GMT+02:00 Lehuede Sebastien <[email protected]>:
> Nick,
>
> I will try tomorrow to increase memory and give you some news after.
>
> For the moment when i use this command :
> "bin/kafka-console-consumer.sh --zookeeper node1:2181 --topic test_topic
> --from-beginning" i'm able to see data contains in my text file passed
> though NiFi.
>
> Sebastien.
>
> Lehuédé Sebastien
>
> Le 19 juil. 2017 à 21:45, Nick Allen <[email protected]> a écrit :
>
> > I also had tried to increase Metron node memory, 8GB to 16GB directlty
> through VBox interface
>
> When changing the memory available, I would start over completely.
>
> 1. Run `vagrant destroy`.
> 2. Then edit the Vagrantfile here [1] and increase the memory from 8192 to
> your desired value.
> 3. Then try it again; `vagrant up`.
>
>
> > Otherwise, what logs i can check to see if Kafka correctly receive data
> from NiFi ?
>
> Use the `kafka-console-consumer` to check if any data is landing in Kafka
> from Nifi.
>
>
> [1] https://github.com/apache/metron/blob/master/metron-
> deployment/vagrant/full-dev-platform/Vagrantfile#L48
>
>
>
>
>
> On Wed, Jul 19, 2017 at 3:22 PM, Sebastien Lehuede <[email protected]>
> wrote:
>
>> Nick,
>>
>> I had tried to install NiFi on an other server, but result is the same. I
>> can see my data pass through NiFi, but Elasticsearch index is not created.
>> (My server successfully reach my Metron node on port 6667).
>>
>> I also had tried to increase Metron node memory, 8GB to 16GB directlty
>> through VBox interface. « top » command show 16GB memory on server but
>> Ambari show only 8GB, there is others actions to do to increase Metron node
>> memory ?
>>
>> Otherwise, what logs i can check to see if Kafka correctly receive data
>> from NiFi ?
>>
>> Sebastien.
>>
>> (Sorry for my English)
>>
>> Le 19 juil. 2017 à 16:56, Nick Allen <[email protected]> a écrit :
>>
>> That being said, I would highly suggest experimenting in a cluster; for
>> instance using our automated AWS deployment mechanism [1]. This will
>> spin-up a full cluster (a totally open, unsecured cluster that you need to
>> secure yourself) in AWS.
>>
>> Of course, this will cost you money, but should you have the funds, this
>> is the way to go.
>>
>> [1] https://github.com/apache/metron/tree/master/metron-depl
>> oyment/amazon-ec2
>>
>> On Wed, Jul 19, 2017 at 10:52 AM, Nick Allen <[email protected]> wrote:
>>
>>> Yes, you can try to increase the amount of memory, as long as you have
>>> enough physical memory on your machine. I don't know of any hard memory
>>> limits on that VM.
>>>
>>> On Wed, Jul 19, 2017 at 9:44 AM, Lehuede sebastien <[email protected]>
>>> wrote:
>>>
>>>> Hi Nick,
>>>>
>>>> Yes, i run Nifi directly in "Full Dev VM". I will try to reinstall and
>>>> reconfigure Nifi in an other VM or is it possible to simply increase amount
>>>> of Memory for "Full Dev VM ?
>>>>
>>>> Thanks,
>>>>
>>>> Sebastien.
>>>>
>>>> 2017-07-19 15:13 GMT+02:00 Nick Allen <[email protected]>:
>>>>
>>>>> Are you also running Nifi in your "Full Dev" VM? If so, I would
>>>>> assume that you just don't have enough resources to run all that on one
>>>>> VM. Just running Metron's components alone in "Full Dev" has taken
>>>>> considerable effort to tune. Adding Nifi would probably cause some weird
>>>>> errors due to resource constraints.
>>>>>
>>>>> On Wed, Jul 19, 2017 at 8:50 AM, Lehuede sebastien <
>>>>> [email protected]> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I have successfully deployed Metron 0.4.0 full dev platform. After
>>>>>> that, i have tried to add a new telemetry data source with this tutorial
>>>>>> :
>>>>>> https://cwiki.apache.org/confluence/display/METRON
>>>>>> /Adding+a+New+Telemetry+Data+Source. But when i start NiFi
>>>>>> processor, i don't see any logs in Storm UI and no Elasticsearch index
>>>>>> are
>>>>>> created.
>>>>>>
>>>>>> I had no error, in NiFi interface when i start processor everything
>>>>>> seem to be OK and in my Ambari interface all services are started, except
>>>>>> "Metron REST".
>>>>>>
>>>>>> I have setting up a very simple grok parser
>>>>>> ("%{DATA:toto}\:%{DATA:tata}") for testing purpose. My goal is to parse a
>>>>>> simple text file who contains things like "bob:alice".
>>>>>>
>>>>>> How can i troubleshoot this problem ? With which logs ?
>>>>>> Has anyone ever had this problem ?
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Sebastien.
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>
