I think you might want this instead:
TEST_TXT %{DATA:test1}\:%{GREEDYDATA:test2}
You can actually use the Management UI to create your Grok parser and test
your pattern against data to see what it does.
On Thu, Jul 20, 2017 at 4:20 AM, Lehuede sebastien <[email protected]>
wrote:
> Hi,
>
> After mooving NiFi on another server, Kafka correctly receive data
> contains in my text file.
>
> But i have a new error (Log from Storm UI):
>
> "java.lang.IllegalStateException: Grok parser Error: Grok statement
> produced a null message. Original message was: tata:titi and the parsed
> message was: {} . Check the pattern at: /apps/metron/patterns/test_txt on
> tata:titi"
>
> It seem my parser is incorrect, but i had tested it on GrokConstructor and
> he's very simple...Has anyone already had this error ?
>
>
>
> *Metron node configuration :*
>
>
> *Parser :*
> /apps/metron/patterns/test_txt
>
> TEST_TXT %{DATA:test1}\:%{DATA:test2}
>
> *Parser configuration :*
> /usr/metron/0.4.0/config/zookeeper/parsers/test_txt.json
>
> {
> "parserClassName": "org.apache.metron.parsers.GrokParser",
> "sensorTopic": "test_txt",
> "parserConfig": {
> "grokPath": "/apps/metron/patterns/test_txt",
> "patternLabel": "TEST_TXT"
> }
> }
>
>
> 2017-07-19 22:59 GMT+02:00 Lehuede Sebastien <[email protected]>:
>
>> Nick,
>>
>> I will try tomorrow to increase memory and give you some news after.
>>
>> For the moment when i use this command :
>> "bin/kafka-console-consumer.sh --zookeeper node1:2181 --topic test_topic
>> --from-beginning" i'm able to see data contains in my text file passed
>> though NiFi.
>>
>> Sebastien.
>>
>> Lehuédé Sebastien
>>
>> Le 19 juil. 2017 à 21:45, Nick Allen <[email protected]> a écrit :
>>
>> > I also had tried to increase Metron node memory, 8GB to 16GB directlty
>> through VBox interface
>>
>> When changing the memory available, I would start over completely.
>>
>> 1. Run `vagrant destroy`.
>> 2. Then edit the Vagrantfile here [1] and increase the memory from 8192
>> to your desired value.
>> 3. Then try it again; `vagrant up`.
>>
>>
>> > Otherwise, what logs i can check to see if Kafka correctly receive
>> data from NiFi ?
>>
>> Use the `kafka-console-consumer` to check if any data is landing in Kafka
>> from Nifi.
>>
>>
>> [1] https://github.com/apache/metron/blob/master/metron-depl
>> oyment/vagrant/full-dev-platform/Vagrantfile#L48
>>
>>
>>
>>
>>
>> On Wed, Jul 19, 2017 at 3:22 PM, Sebastien Lehuede <[email protected]>
>> wrote:
>>
>>> Nick,
>>>
>>> I had tried to install NiFi on an other server, but result is the same.
>>> I can see my data pass through NiFi, but Elasticsearch index is not
>>> created. (My server successfully reach my Metron node on port 6667).
>>>
>>> I also had tried to increase Metron node memory, 8GB to 16GB directlty
>>> through VBox interface. « top » command show 16GB memory on server but
>>> Ambari show only 8GB, there is others actions to do to increase Metron node
>>> memory ?
>>>
>>> Otherwise, what logs i can check to see if Kafka correctly receive data
>>> from NiFi ?
>>>
>>> Sebastien.
>>>
>>> (Sorry for my English)
>>>
>>> Le 19 juil. 2017 à 16:56, Nick Allen <[email protected]> a écrit :
>>>
>>> That being said, I would highly suggest experimenting in a cluster; for
>>> instance using our automated AWS deployment mechanism [1]. This will
>>> spin-up a full cluster (a totally open, unsecured cluster that you need to
>>> secure yourself) in AWS.
>>>
>>> Of course, this will cost you money, but should you have the funds, this
>>> is the way to go.
>>>
>>> [1] https://github.com/apache/metron/tree/master/metron-depl
>>> oyment/amazon-ec2
>>>
>>> On Wed, Jul 19, 2017 at 10:52 AM, Nick Allen <[email protected]> wrote:
>>>
>>>> Yes, you can try to increase the amount of memory, as long as you have
>>>> enough physical memory on your machine. I don't know of any hard memory
>>>> limits on that VM.
>>>>
>>>> On Wed, Jul 19, 2017 at 9:44 AM, Lehuede sebastien <[email protected]
>>>> > wrote:
>>>>
>>>>> Hi Nick,
>>>>>
>>>>> Yes, i run Nifi directly in "Full Dev VM". I will try to reinstall and
>>>>> reconfigure Nifi in an other VM or is it possible to simply increase
>>>>> amount
>>>>> of Memory for "Full Dev VM ?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Sebastien.
>>>>>
>>>>> 2017-07-19 15:13 GMT+02:00 Nick Allen <[email protected]>:
>>>>>
>>>>>> Are you also running Nifi in your "Full Dev" VM? If so, I would
>>>>>> assume that you just don't have enough resources to run all that on one
>>>>>> VM. Just running Metron's components alone in "Full Dev" has taken
>>>>>> considerable effort to tune. Adding Nifi would probably cause some weird
>>>>>> errors due to resource constraints.
>>>>>>
>>>>>> On Wed, Jul 19, 2017 at 8:50 AM, Lehuede sebastien <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I have successfully deployed Metron 0.4.0 full dev platform. After
>>>>>>> that, i have tried to add a new telemetry data source with this
>>>>>>> tutorial :
>>>>>>> https://cwiki.apache.org/confluence/display/METRON
>>>>>>> /Adding+a+New+Telemetry+Data+Source. But when i start NiFi
>>>>>>> processor, i don't see any logs in Storm UI and no Elasticsearch index
>>>>>>> are
>>>>>>> created.
>>>>>>>
>>>>>>> I had no error, in NiFi interface when i start processor everything
>>>>>>> seem to be OK and in my Ambari interface all services are started,
>>>>>>> except
>>>>>>> "Metron REST".
>>>>>>>
>>>>>>> I have setting up a very simple grok parser
>>>>>>> ("%{DATA:toto}\:%{DATA:tata}") for testing purpose. My goal is to parse
>>>>>>> a
>>>>>>> simple text file who contains things like "bob:alice".
>>>>>>>
>>>>>>> How can i troubleshoot this problem ? With which logs ?
>>>>>>> Has anyone ever had this problem ?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Sebastien.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
