Thanks Jorg! I'll take it up to the hosting company... (Azure!)
On Sunday, August 24, 2014 3:52:54 PM UTC+3, Jörg Prante wrote: > > The company which is providing the hosting service had shut down the > virtual machine, "hv_utils" is a message from the hypervisor. > > This is not related to Elasticsearch at all. > > Jörg > > > On Sun, Aug 24, 2014 at 11:19 AM, Mark Walkom <[email protected] > <javascript:>> wrote: > >> What version of ES are you running? >> Are you running on a hosting service and if so do you have a firewall >> protecting the host - ie it's not open to the entire internet? >> >> Regards, >> Mark Walkom >> >> Infrastructure Engineer >> Campaign Monitor >> email: [email protected] <javascript:> >> web: www.campaignmonitor.com >> >> >> On 24 August 2014 19:05, Eitan Vesely <[email protected] <javascript:>> >> wrote: >> >>> >>> I did find the shutdown request in the syslog: >>> >>> Aug 23 16:49:01 medisafelog2 kernel: [3361057.489168] hv_utils: Shutdown >>> request received - graceful shutdown initiated >>> >>> yet i have no idea who or what initiated it... how can i dig in? >>> >>> >>> On Sunday, August 24, 2014 12:01:50 PM UTC+3, Eitan Vesely wrote: >>>> >>>> >>>> Thanks Mark, >>>> >>>> auth.log doesnt show any login or sudo at the time of the elastic >>>> stopping... >>>> nothing else is running on that machine - it is a dedicated ES server. >>>> >>>> what i did find in the auth log is that someone is trying to hack into >>>> the system, yet i dont see how it got to do with elastic stopping? >>>> >>>> On Sunday, August 24, 2014 4:35:41 AM UTC+3, Mark Walkom wrote: >>>>> >>>>> Something is stopping the service. >>>>> >>>>> If you are on linux check the auth log, if anyone is using sudo to >>>>> stop it then you will see that logged. Otherwise, what else runs on the >>>>> machine? >>>>> >>>>> Regards, >>>>> Mark Walkom >>>>> >>>>> Infrastructure Engineer >>>>> Campaign Monitor >>>>> email: [email protected] >>>>> web: www.campaignmonitor.com >>>>> >>>>> >>>>> On 24 August 2014 06:15, Eitan Vesely <[email protected]> wrote: >>>>> >>>>>> Hi Guys, >>>>>> i've installed ES a month ago and its working just fine. >>>>>> >>>>>> today, for some reason, ES just went down for no visible reason: >>>>>> >>>>>> here is what i see in the log file : >>>>>> >>>>>> [2014-08-23 16:47:11,272][DEBUG][action.search.type ] >>>>>> [Plunderer] [g30nm0bi2j663tgu6ud][1], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], >>>>>> s[STARTED]: Failed to execute >>>>>> [org.elasticsearch.action.search.SearchRequest@5531dfad] >>>>>> lastShard [true] >>>>>> org.elasticsearch.search.SearchParseException: >>>>>> [g30nm0bi2j663tgu6ud][1]: from[-1],size[-1]: Parse Failure [Failed to >>>>>> parse >>>>>> source [{"facets":{"0":{"date_histogram":{"key_field":"@ >>>>>> timestamp","value_field":"user_count","interval":"1h"}," >>>>>> global":true,"facet_filter":{"fquery":{"query":{"filtered":{ >>>>>> "query":{"query_string":{"query":"*"}},"filter":{"bool": >>>>>> {"must":[{"range":{"@timestamp":{"from":1407602785182,"to": >>>>>> 1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to": >>>>>> 1408811520255}}}]}}}}}}}},"size":0}]] >>>>>> at org.elasticsearch.search.SearchService.parseSource( >>>>>> SearchService.java:649) >>>>>> at org.elasticsearch.search.SearchService.createContext( >>>>>> SearchService.java:511) >>>>>> at org.elasticsearch.search.SearchService.createAndPutContext( >>>>>> SearchService.java:483) >>>>>> at org.elasticsearch.search.SearchService.executeQueryPhase( >>>>>> SearchService.java:252) >>>>>> at org.elasticsearch.search.action.SearchServiceTransportAction$ >>>>>> 5.call(SearchServiceTransportAction.java:206) >>>>>> at org.elasticsearch.search.action.SearchServiceTransportAction$ >>>>>> 5.call(SearchServiceTransportAction.java:203) >>>>>> at org.elasticsearch.search.action.SearchServiceTransportAction$ >>>>>> 23.run(SearchServiceTransportAction.java:517) >>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>> ThreadPoolExecutor.java:1145) >>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>> ThreadPoolExecutor.java:615) >>>>>> at java.lang.Thread.run(Thread.java:744) >>>>>> Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException: >>>>>> Facet [0]: (key) field [@timestamp] not found >>>>>> at org.elasticsearch.search.facet.datehistogram. >>>>>> DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160) >>>>>> at org.elasticsearch.search.facet.FacetParseElement.parse( >>>>>> FacetParseElement.java:93) >>>>>> at org.elasticsearch.search.SearchService.parseSource( >>>>>> SearchService.java:633) >>>>>> ... 9 more >>>>>> [2014-08-23 16:47:11,273][DEBUG][action.search.type ] >>>>>> [Plunderer] [g30nm0bi2j663tgu6ud][0], node[Vc4xSuh1S1qQOvQdv-wD_A], [P], >>>>>> s[STARTED]: Failed to execute [org.elasticsearch.action. >>>>>> search.SearchRequest@5531dfad] >>>>>> org.elasticsearch.search.SearchParseException: >>>>>> [g30nm0bi2j663tgu6ud][0]: from[-1],size[-1]: Parse Failure [Failed to >>>>>> parse >>>>>> source [{"facets":{"0":{"date_histogram":{"key_field":"@ >>>>>> timestamp","value_field":"user_count","interval":"1h"}," >>>>>> global":true,"facet_filter":{"fquery":{"query":{"filtered":{ >>>>>> "query":{"query_string":{"query":"*"}},"filter":{"bool": >>>>>> {"must":[{"range":{"@timestamp":{"from":1407602785182,"to": >>>>>> 1408812385182}}},{"range":{"@timestamp":{"from":1408516424602,"to": >>>>>> 1408811520255}}}]}}}}}}}},"size":0}]] >>>>>> at org.elasticsearch.search.SearchService.parseSource( >>>>>> SearchService.java:649) >>>>>> at org.elasticsearch.search.SearchService.createContext( >>>>>> SearchService.java:511) >>>>>> at org.elasticsearch.search.SearchService.createAndPutContext( >>>>>> SearchService.java:483) >>>>>> at org.elasticsearch.search.SearchService.executeQueryPhase( >>>>>> SearchService.java:252) >>>>>> at org.elasticsearch.search.action.SearchServiceTransportAction$ >>>>>> 5.call(SearchServiceTransportAction.java:206) >>>>>> at org.elasticsearch.search.action.SearchServiceTransportAction$ >>>>>> 5.call(SearchServiceTransportAction.java:203) >>>>>> at org.elasticsearch.search.action.SearchServiceTransportAction$ >>>>>> 23.run(SearchServiceTransportAction.java:517) >>>>>> at java.util.concurrent.ThreadPoolExecutor.runWorker( >>>>>> ThreadPoolExecutor.java:1145) >>>>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run( >>>>>> ThreadPoolExecutor.java:615) >>>>>> at java.lang.Thread.run(Thread.java:744) >>>>>> Caused by: org.elasticsearch.search.facet.FacetPhaseExecutionException: >>>>>> Facet [0]: (key) field [@timestamp] not found >>>>>> at org.elasticsearch.search.facet.datehistogram. >>>>>> DateHistogramFacetParser.parse(DateHistogramFacetParser.java:160) >>>>>> at org.elasticsearch.search.facet.FacetParseElement.parse( >>>>>> FacetParseElement.java:93) >>>>>> at org.elasticsearch.search.SearchService.parseSource( >>>>>> SearchService.java:633) >>>>>> ... 9 more >>>>>> [2014-08-23 16:49:05,401][INFO ][node ] >>>>>> [Plunderer] stopping ... >>>>>> [2014-08-23 16:49:06,737][INFO ][node ] >>>>>> [Plunderer] stopped >>>>>> [2014-08-23 16:49:06,737][INFO ][node ] >>>>>> [Plunderer] closing ... >>>>>> [2014-08-23 16:49:06,772][INFO ][node ] >>>>>> [Plunderer] closed >>>>>> [2014-08-23 20:02:09,693][WARN ][common.jna ] Unable to >>>>>> lock JVM memory (ENOMEM). This can result in part of the JVM being >>>>>> swapped >>>>>> out. Increase RLIMIT_MEMLOCK or run elasticsearch as root. >>>>>> [2014-08-23 20:02:09,921][INFO ][node ] [Ahmet >>>>>> Abdol] version[1.2.2], pid[2715], build[9902f08/2014-07-09T12:02:32Z] >>>>>> [2014-08-23 20:02:09,921][INFO ][node ] [Ahmet >>>>>> Abdol] initializing ... >>>>>> [2014-08-23 20:02:09,940][INFO ][plugins ] [Ahmet >>>>>> Abdol] loaded [], sites [kopf] >>>>>> [2014-08-23 20:02:14,692][INFO ][node ] [Ahmet >>>>>> Abdol] initialized >>>>>> >>>>>> any ideas?? >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit https://groups.google.com/d/ >>>>>> msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f% >>>>>> 40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/55571116-a9ac-4517-b530-fc2e0f13501f%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected] <javascript:>. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/bfc73608-e6fe-489a-8f38-33f033b61b66%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqUgi4uJV6wzjvfM9YWxvDY9xkfgCHJFSRm_6D5vDUQ%40mail.gmail.com >> >> <https://groups.google.com/d/msgid/elasticsearch/CAEM624ZmqUgi4uJV6wzjvfM9YWxvDY9xkfgCHJFSRm_6D5vDUQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/265cb108-5644-4798-a70c-db067e6530d2%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
