Nick, I will try tomorrow to increase memory and give you some news after.
For the moment when i use this command : "bin/kafka-console-consumer.sh --zookeeper node1:2181 --topic test_topic --from-beginning" i'm able to see data contains in my text file passed though NiFi. Sebastien. Lehuédé Sebastien > Le 19 juil. 2017 à 21:45, Nick Allen <[email protected]> a écrit : > > > I also had tried to increase Metron node memory, 8GB to 16GB directlty > > through VBox interface > > When changing the memory available, I would start over completely. > 1. Run `vagrant destroy`. > 2. Then edit the Vagrantfile here [1] and increase the memory from 8192 to > your desired value. > 3. Then try it again; `vagrant up`. > > > Otherwise, what logs i can check to see if Kafka correctly receive data > > from NiFi ? > > Use the `kafka-console-consumer` to check if any data is landing in Kafka > from Nifi. > > > [1] > https://github.com/apache/metron/blob/master/metron-deployment/vagrant/full-dev-platform/Vagrantfile#L48 > > > > > >> On Wed, Jul 19, 2017 at 3:22 PM, Sebastien Lehuede <[email protected]> >> wrote: >> Nick, >> >> I had tried to install NiFi on an other server, but result is the same. I >> can see my data pass through NiFi, but Elasticsearch index is not created. >> (My server successfully reach my Metron node on port 6667). >> >> I also had tried to increase Metron node memory, 8GB to 16GB directlty >> through VBox interface. « top » command show 16GB memory on server but >> Ambari show only 8GB, there is others actions to do to increase Metron node >> memory ? >> >> Otherwise, what logs i can check to see if Kafka correctly receive data from >> NiFi ? >> >> Sebastien. >> >> (Sorry for my English) >> >>> Le 19 juil. 2017 à 16:56, Nick Allen <[email protected]> a écrit : >>> >>> That being said, I would highly suggest experimenting in a cluster; for >>> instance using our automated AWS deployment mechanism [1]. This will >>> spin-up a full cluster (a totally open, unsecured cluster that you need to >>> secure yourself) in AWS. >>> >>> Of course, this will cost you money, but should you have the funds, this is >>> the way to go. >>> >>> [1] >>> https://github.com/apache/metron/tree/master/metron-deployment/amazon-ec2 >>> >>>> On Wed, Jul 19, 2017 at 10:52 AM, Nick Allen <[email protected]> wrote: >>>> Yes, you can try to increase the amount of memory, as long as you have >>>> enough physical memory on your machine. I don't know of any hard memory >>>> limits on that VM. >>>> >>>>> On Wed, Jul 19, 2017 at 9:44 AM, Lehuede sebastien <[email protected]> >>>>> wrote: >>>>> Hi Nick, >>>>> >>>>> Yes, i run Nifi directly in "Full Dev VM". I will try to reinstall and >>>>> reconfigure Nifi in an other VM or is it possible to simply increase >>>>> amount of Memory for "Full Dev VM ? >>>>> >>>>> Thanks, >>>>> >>>>> Sebastien. >>>>> >>>>> 2017-07-19 15:13 GMT+02:00 Nick Allen <[email protected]>: >>>>>> Are you also running Nifi in your "Full Dev" VM? If so, I would assume >>>>>> that you just don't have enough resources to run all that on one VM. >>>>>> Just running Metron's components alone in "Full Dev" has taken >>>>>> considerable effort to tune. Adding Nifi would probably cause some >>>>>> weird errors due to resource constraints. >>>>>> >>>>>>> On Wed, Jul 19, 2017 at 8:50 AM, Lehuede sebastien >>>>>>> <[email protected]> wrote: >>>>>>> Hello, >>>>>>> >>>>>>> I have successfully deployed Metron 0.4.0 full dev platform. After >>>>>>> that, i have tried to add a new telemetry data source with this >>>>>>> tutorial : https://cwiki.apache.org/confluence/display/METRON >>>>>>> /Adding+a+New+Telemetry+Data+Source. But when i start NiFi processor, i >>>>>>> don't see any logs in Storm UI and no Elasticsearch index are created. >>>>>>> >>>>>>> I had no error, in NiFi interface when i start processor everything >>>>>>> seem to be OK and in my Ambari interface all services are started, >>>>>>> except "Metron REST". >>>>>>> >>>>>>> I have setting up a very simple grok parser >>>>>>> ("%{DATA:toto}\:%{DATA:tata}") for testing purpose. My goal is to parse >>>>>>> a simple text file who contains things like "bob:alice". >>>>>>> >>>>>>> How can i troubleshoot this problem ? With which logs ? >>>>>>> Has anyone ever had this problem ? >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Sebastien. >>>>>>> >>>>>> >>>>> >>>> >>> >> >
