Hi Rainer, Just one quick comment: when I said "manually delete logs" I meant via a curl command :) Manual was opposed to the "keep logs for X days" which removes them without any action on your behalf. Anyway, local install is easy, as you've just seen :D
Looking forward to your other questions! Best regards, Radu -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ On Mon, Sep 1, 2014 at 4:22 PM, Rainer Gerhards <[email protected]> wrote: > On Mon, Sep 1, 2014 at 2:26 PM, Radu Gheorghe <[email protected]> > wrote: > > > Hello Rainer, I've been waiting for this :) > > > > I'll reply inline. > > > > On Mon, Sep 1, 2014 at 1:44 PM, Rainer Gerhards < > [email protected]> > > wrote: > > > > > > Questions now: > > > > > > - what should I install to get an ES testenv (possibly something that > > > doesn't need babysitting or manual updates in the next year or two). > > > > > > > Two options: > > 1. Install ES locally. It's mostly about getting the DEB > > <http://www.elasticsearch.org/download/> and doing `dpkg -i > > elasticsearch*.deb`. Besides that, you need Java. The not-so-open Oracle > > version > > < > > > http://ubuntuhandbook.org/index.php/2014/02/install-oracle-java-6-7-or-8-ubuntu-14-04/ > > > > > is typically recommended. This requires pulling another DEB in the next > > year or two if major changes occur in the API (not super-likely, but who > > knows) > > 2. Use Logsene <http://sematext.com/logsene/index.html>, which exposes > the > > ES API. The free plan will do, as it's good for 1M documents. You can > > remove > > logs manually > > < > > > https://sematext.atlassian.net/wiki/display/PUBLOGSENE/Logsene+FAQ#LogseneFAQ-Deletinglogs > > >, > > and you can adjust the number of days they're kept through the UI. We'll > > take care of ES upgrades for you, as we typically release ES often and > > typically upgrade ES on the way. If you have any questions about Logsene, > > you know where to find me :D > > > > > I'll give #1 a try. For the testbench, manual steps do not make any sense > ;) > > After I started, I also found this: > http://blog.bekijkhet.com/2014/06/install-elasticsearch-in-ubuntu-1404.html > Which seems to do the trick (but I used the Oracle Java as you suggested). > > This seemed to have done the trick :-) > > > > > > > > To run a test I need to do via a script > > > > > > - clean up any previous work done on an ES index > > > > > > > curl -XDELETE localhost:9200/_all # this wipes everything > > > > ah, that's great for a reset! > > > > curl -XDELETE localhost:9200/my_index # this removes an index > > > > > > > - be able to "export" a whole ES index into a text file > > > > > > > This depends on how big your index is and how you want the text file to > be > > formatted. Let's assume you're good with a big pretty-formatted JSON for > > now. > > > > If the index is small (say 100 or 1000 docs), this will do: > > > > > I can go with that. Usually, the tests inject between a few thousand and > some hunderedthousand messages. But for starters 1000 is fine. > > > > curl localhost:9200/my_index/_search?pretty > /tmp/destination_file > > > > If the index is big, you need to scan and scroll > > < > > > http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/scan-scroll.html > > >. > > For example, this scroll ID expires in 1m. The timer is reset each time > you > > get the scroll > > curl 'localhost:9200/my_index/_search?search_type=scan&scroll=1m' > > > > You'll get back a scroll ID. You should put it in a variable like: > > SCROLL_ID=curl -s > > 'localhost:9200/_search?search_type=scan&scroll=1m&pretty' | grep > scroll_id > > | cut -d '"' -f 4 > > > > # then, to get a batch of results, you scroll: > > curl 'localhost:9200/_search/scroll?scroll=1m&pretty' -d $SCROLL_ID > > > > But each time you do that, you need a new scroll ID. That new scroll ID > > needs to be used for the next fetch and so on, until you have no more > hits > > in the hits array. A bit complicated in shell, may have to use Python or > > something like that. There are ready-made scripts that may do that for > you, > > like this one: > > https://github.com/miku/estab > > > > > > > > > > How is this done automatically via shell script? > > > > > > > I hope I've answered this question above. > > > > > thanks, I think this should get me going. I'll be posting more questions as > they come up (I guess there will be some ;)). > > > > > > > > > If I got those pieces together, I think I can add a basic test. Once > this > > > is done, we may be able to do more (especially checking for error > cases), > > > but let's first get the basics going. > > > > > > Any help is appreciated. > > > > > > > I did some omelasticsearch tests a while ago. Though I can't find them > > right now. I think they provide a good base. Should I dig deeper into > > Emails and githubs or do you already know where they are? > > > > > Seen your other post, and will definitely look at them (and I remember we > talked about this quite a while ago ... :-( ). > > Rainer > > > > Best regards, > > Radu > > _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
