http://serverfault.com/questions/412114/cannot-switch-ssh-to-specific-user-su-cannot-set-user-id-resource-temporaril
Looks like I have the same issue, is it normal that ES spawns that much process, over 1000 ? On Wednesday, July 16, 2014 9:23:45 AM UTC-4, Bastien Chong wrote: > > I'm not sure how to find answer that, I use the default settings in ES. > The cluster is composed of 2 read/write node, and a read-only node. > There is 1 Logstash instance that simply output 2 type of data to ES. > Nothing fancy. > > I need to delete documents older than a day, for this particular thing, I > can't create a daily index. Is there a better way ? > > I'm using an EC2 m3.large instance, ES has 1.5GB of heap. > > It seems like I'm hitting an OS limit, I can't "su - elasticsearch" : > > su: /bin/bash: Resource temporarily unavailable > > Stopping elasticsearch fix this issue, so this is directly linked. > >> -bash-4.1$ ulimit -a >> core file size (blocks, -c) 0 >> data seg size (kbytes, -d) unlimited >> scheduling priority (-e) 0 >> file size (blocks, -f) unlimited >> pending signals (-i) 29841 >> max locked memory (kbytes, -l) unlimited >> max memory size (kbytes, -m) unlimited >> open files (-n) 65536 >> pipe size (512 bytes, -p) 8 >> POSIX message queues (bytes, -q) 819200 >> real-time priority (-r) 0 >> stack size (kbytes, -s) 8192 >> cpu time (seconds, -t) unlimited >> max user processes (-u) 1024 >> virtual memory (kbytes, -v) unlimited >> file locks (-x) unlimited >> > > > > > On Tuesday, July 15, 2014 6:35:22 PM UTC-4, Mark Walkom wrote: >> >> It'd depend on your config I'd guess, in particular how many >> workers/threads you have and what ES output you are using in LS. >> >> Why are you cleaning an index like this anyway? It seems horribly >> inefficient. >> Basically the error is "OutOfMemoryError", which means you've run out of >> heap for the operation to complete. What are the specs for your node, how >> much heap does ES have? >> >> Regards, >> Mark Walkom >> >> Infrastructure Engineer >> Campaign Monitor >> email: [email protected] >> web: www.campaignmonitor.com >> >> >> On 16 July 2014 00:43, Bastien Chong <[email protected]> wrote: >> >>> I have a basic setup with a logstash shipper, an indexer and an >>> elasticsearch cluster. >>> Elasticsearch listen on the standart 9200/9300 and logstash indexer >>> 9301/9302. >>> >>> When I do a netstat | wc -l for the ES process: 184 found >>> (sample) >>> >>>> tcp 0 0 ::ffff:172.17.7.87:9300 ::ffff: >>>> 172.17.8.39:59573 ESTABLISHED 23224/java >>>> tcp 0 0 ::ffff:172.17.7.87:9300 ::ffff: >>>> 172.17.7.87:47609 ESTABLISHED 23224/java >>>> tcp 0 0 ::ffff:172.17.7.87:53493 ::ffff:172.17.7.87:9302 >>>> >>>> ESTABLISHED 23224/java >>>> tcp 0 0 ::ffff:172.17.7.87:9300 ::ffff: >>>> 172.17.8.39:59564 ESTABLISHED 23224/java >>>> tcp 0 0 ::ffff:172.17.7.87:9300 ::ffff: >>>> 172.17.7.87:47657 ESTABLISHED 23224/java >>>> >>> >>> Same thing for the logstash indexer : 160 found >>> (sample) >>> >>>> tcp 0 0 ::ffff:172.17.7.87:50132 ::ffff:172.17.8.39:9300 >>>> >>>> ESTABLISHED 1516/java >>>> tcp 0 0 ::ffff:172.17.7.87:9301 ::ffff: >>>> 172.17.7.87:60153 ESTABLISHED 1516/java >>>> tcp 0 0 ::ffff:172.17.7.87:9301 ::ffff: >>>> 172.17.7.87:60145 ESTABLISHED 1516/java >>>> tcp 0 0 ::ffff:172.17.7.87:50129 ::ffff:172.17.8.39:9300 >>>> >>>> ESTABLISHED 1516/java >>>> tcp 0 0 ::ffff:172.17.7.87:9302 ::ffff: >>>> 172.17.7.87:53501 ESTABLISHED 1516/java >>>> >>> >>> Also, not sure if related, when I try to delete some documents by query >>> ( curl -XDELETE 'http://localhost:9200/check/_query?pretty=1' -d >>> '{"query":{"range":{"@timestamp":{"from":"2014-07-10T00:00:00","to":"2014-07-14T05:00:00"}}}}' >>> >>> ) >>> >>> "RemoteTransportException[[Stonewall][inet[/172.17.8.39:9300]][deleteByQuery/shard]]; >>> >>> nested: OutOfMemoryError[unable to create new native thread]; " >>> >>> I have a script that run this kind of query every 30 seconds to clean >>> up this particular index. >>> >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "elasticsearch" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/1167d4a6-6b87-45a9-835a-eba0ba696825%40googlegroups.com >>> >>> <https://groups.google.com/d/msgid/elasticsearch/1167d4a6-6b87-45a9-835a-eba0ba696825%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c1950ee8-3401-4cf8-929c-181c2a6fb870%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
