Hi Moshe, I have seen that the offset lag error goes away once you push a few events into your topic. See if that works for you.
For enabling debug logs on the topologies, check out this article: https://community.hortonworks.com/articles/36151/debugging-an-apache-storm-topology.html Regards, Anand From: Matt Foley <mfo...@hortonworks.com<mailto:mfo...@hortonworks.com>> Reply-To: "user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>" <user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>> Date: Monday, March 13, 2017 at 11:12 AM To: "user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>" <user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>> Subject: Re: a proper entry level documentation Hi Moshe, Re the Elastic Search port question, ES uses both 9200 and 9300. This is consistent with the answer to the stackoverflow question you cited. Port 9200 is used for RESTful API and Browser interactions. Port 9300 is used for Java Client APIs and intra-cluster communications. The RESTful port configuration is stated to be 9200-9300 (meaning, I think, 9200-9299, which should have been the answer to that stackoverflow question), I’m not sure how anything but port 9200 would actually be used given the behavior of REST, but perhaps it is. However, the Java/intra-cluster port usage is stated to be 9300-9400 (probably also meaning 9300-9399), and these clients, which are provided by the ES application code, apparently do know how to switch ports within that range, in cooperation with the server, to support multiple simultaneous persistent connections. Both port ranges must be configured for ES to work correctly, but of course you can change the actual port range values if needful in your environment. From: moshe jarusalem <tuu...@gmail.com<mailto:tuu...@gmail.com>> Reply-To: "user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>" <user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>> Date: Sunday, March 12, 2017 at 12:46 AM To: "user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>" <user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>> Subject: Re: a proper entry level documentation Thanks for all your efforts. I have followed https://cwiki.apache.org/confluence/display/METRON/2016/04/28/Metron+Tutorial+-+Fundamentals+Part+2%3A+Creating+a+New+Enrichment Unfortuanatly it is not working as expected. I get one error from kafka "Offset lags for kafka not supported for older versions. Please update kafka spout to latest version." I checked storm kafka version but it seems our storm version already contains the fix. Yet another question is elastic search port is defined to be 9300 in configurations but actually it runs only on 9200. http://stackoverflow.com/questions/24624980/port-9300-on-elasticsearch Would you please also tell how to enable debug logs for topologies so that we can flow message flow and troubleshoot ? Cheers, On Fri, Mar 10, 2017 at 9:22 PM, Umesh Kaushik <umesh.kaus...@bhujang.net<mailto:umesh.kaus...@bhujang.net>> wrote: Thanks Matt! Kudos! Cheers, Umesh Kaushik (Full stack developer- cyber security analyst: Bhujang innovations) (9620023458) Sent from mobile device, kindly excuse any typographic error. On Mar 10, 2017 11:49 PM, "Matt Foley" <ma...@apache.org<mailto:ma...@apache.org>> wrote: Thanks for this contribution, Umesh. I’ve edited it lightly, and posted it to the wiki at https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=68718548 along with the other installation-related help files. Regards, --Matt From: Umesh Kaushik <umesh.kaus...@bhujang.net<mailto:umesh.kaus...@bhujang.net>> Date: Wednesday, March 8, 2017 at 5:18 PM To: Matt Foley <ma...@apache.org<mailto:ma...@apache.org>> Cc: "user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>" <user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>> Subject: Re: a proper entry level documentation Hi Matt, Yes I have no issues in that, in fact I would be happy if this document could be even a little bit of useful. Yes, it was default master version of metron at the time I installed. It was in the beginning of January for poc and yes ambari was also default. Please let me know if I could assist you any further. Cheers, Umesh Kaushik (9620023458) Sent from mobile device, kindly excuse any typographic error. On Mar 9, 2017 12:57 AM, "Matt Foley" <ma...@apache.org<mailto:ma...@apache.org>> wrote: Umesh, thanks very much for writing this up. I’d like to add it to the Metron wiki. Is that okay with you? Also, what version of Metron was used? Your instructions imply you built from “master” branch of github site, which means you would be on the newest post-0.3.1 code (call it 0.3.2-SNAPSHOT). Is that correct? And what version of Ambari did you use? I think the packaging defaults to version 2.4.0.0, is that what was used? Thanks, --Matt From: Umesh Kaushik <umesh.kaus...@bhujang.net<mailto:umesh.kaus...@bhujang.net>> Reply-To: "user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>" <user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>> Date: Wednesday, March 8, 2017 at 2:04 AM To: "user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>" <user@metron.incubator.apache.org<mailto:user@metron.incubator.apache.org>> Subject: Re: a proper entry level documentation and allotted 2 cores of processor. On 8 March 2017 at 15:32, Umesh Kaushik <umesh.kaus...@bhujang.net<mailto:umesh.kaus...@bhujang.net>> wrote: Hi Khurram Ahmed, Pardon me for the vague answer, however I just created a random Vm, with 100 gb hdd and 20 gb ram, I have a server with 128 gb of ram and 1tb hdd. So, I just chose a random config. On 8 March 2017 at 15:13, Khurram Ahmed <khurramah...@gmail.com<mailto:khurramah...@gmail.com>> wrote: Excellent could you mention the specs that allowed you to run the platform without a hitch? Mention RAM, CPU and hard drive space plz. On Mar 8, 2017 2:38 PM, "Umesh Kaushik" <umesh.kaus...@bhujang.net<mailto:umesh.kaus...@bhujang.net>> wrote: Hi Guys, This is the bare minimum installation document I prepared while installing metron on ubuntu. Please pardon if there are typos or unprofessionalism in the document. Hope this helps! On 8 March 2017 at 14:19, Umesh Kaushik <umesh.kaus...@bhujang.net<mailto:umesh.kaus...@bhujang.net>> wrote: I tried installing on ubuntu, and I documented all the steps and issues I faced, I would love to attach it in the email thread, please let me know if that suits you all, but that is basic installation. I did not document configuring sqid and other addons. Please let me know if it is needed. On 6 March 2017 at 21:56, Earl Hinkle <e...@thehinkles.org<mailto:e...@thehinkles.org>> wrote: I tried on centos 6 and had problems getting python 2.7 installed and some of the other prerequisite. So i triend centos 7, got all the packages installed (python, ansible,etc) and got further into the build but it still failed. I just saw the previous email about a centos 6 script so will try that this week and see how it goes. Sent with AquaMail for Android http://www.aqua-mail.com On March 5, 2017 3:10:08 PM "zeo...@gmail.com<mailto:zeo...@gmail.com>" <zeo...@gmail.com<mailto:zeo...@gmail.com>> wrote: What OS are you trying with now? Like I mentioned, the scripts I put together before will do Mac and centos 6 but it should be possible to get working on more than that. Your choice, but regardless I plan to work further on documentation. If there's anything specific you'd like addressed or shown please let me know. Jon On Sun, Mar 5, 2017, 4:31 PM Earl Hinkle <e...@thehinkles.org<mailto:e...@thehinkles.org>> wrote: I would be willing to help using centos if i could get it up and working. Sent with AquaMail for Android http://www.aqua-mail.com On March 5, 2017 2:23:32 AM moshe jarusalem <tuu...@gmail.com<mailto:tuu...@gmail.com>> wrote: Hi All, I have been trying to get involved with the metron project for a while. I have tried release 0.3.0 and 0.3.1. Unfortunately, it is very difficult to set up a running environment and understand the picture how it works. There are many technologies prerequisite such as Hadoop, nifi, storm, Kafka etc. I assume most of you guys are expert at these technologies so you solve your problems. I have also browsed the user's list and there are many people struggling to set up a proper environment. Would you guys make a document describing step by step setting up a development environment with a specific release (0.3.1 as its the latest) ? Can you describe each technology shortly and give references and most importantly metron's usage of them? Indeed a video recording or a webinar would be much more useful so that there can be some discussions. Regards, Moshe -- Jon Sent from my mobile device -- Cheers, Umesh Kaushik (9620023458) -- Cheers, Umesh Kaushik (9620023458) -- Cheers, Umesh Kaushik (9620023458) -- Cheers, Umesh Kaushik (9620023458)
