Hi Chamila, Does the output of `cat /proc/<pid>/limits` show the ulimit properly set for BAM as well? If it was set properly for BAM, it is strange that it doesn't set properly for CEP...?
You might have stumbled upon across something like the article in [1]. If not, can you go through and see if the user-level and system-wide maximum file descriptor limits are set properly. Also are you running both BAM and CEP as the same user? [1] http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ Thanks, Lasantha On 9 February 2014 17:46, Chamila Wijayarathna <[email protected]> wrote: > When I opened CEP wso2server.sh file it has "ulimit -n 100000 " line on > it which I uncommented. > > > On Sun, Feb 9, 2014 at 5:41 PM, Nirmal Fernando <[email protected]> wrote: > >> It's not set it seems. You should see something like: >> >> Max open files 100000 100000 files >> >> >> On Sun, Feb 9, 2014 at 5:37 PM, Chamila Wijayarathna <[email protected]>wrote: >> >>> It's about the same time as the previous occasion. >>> >>> limit file had the following content. >>> >>> Limit Soft Limit Hard Limit Units >>> Max cpu time unlimited unlimited >>> seconds >>> Max file size unlimited unlimited bytes >>> Max data size unlimited unlimited bytes >>> Max stack size 8388608 unlimited bytes >>> Max core file size 0 unlimited bytes >>> Max resident set unlimited unlimited bytes >>> Max processes unlimited unlimited >>> processes >>> Max open files 32768 32768 files >>> Max locked memory 65536 65536 bytes >>> Max address space unlimited unlimited bytes >>> Max file locks unlimited unlimited locks >>> Max pending signals 16382 16382 >>> signals >>> Max msgqueue size 819200 819200 bytes >>> Max nice priority 0 0 >>> Max realtime priority 0 0 >>> Max realtime timeout unlimited unlimited us >>> >>> Thank You. >>> >>> >>> On Sun, Feb 9, 2014 at 5:16 PM, Nirmal Fernando <[email protected]> wrote: >>> >>>> hmm.. is it at least after sometime than the previous occasion? Also, >>>> did you verify that the limit has been set correctly (using the limits file >>>> of process)? >>>> >>>> >>>> On Sun, Feb 9, 2014 at 5:04 PM, Chamila Wijayarathna >>>> <[email protected]>wrote: >>>> >>>>> Hi, I did edit the wso2server.sh files of both CEP and BAM. Now BAM is >>>>> not giving the exception. But still CEP gives the same exception. >>>>> >>>>> Thank You. >>>>> >>>>> >>>>> On Sun, Feb 9, 2014 at 1:04 PM, Chamila Wijayarathna <[email protected] >>>>> > wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> Thank You for information. >>>>>> >>>>>> >>>>>> On Sun, Feb 9, 2014 at 12:54 PM, Nirmal Fernando <[email protected]>wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Sun, Feb 9, 2014 at 12:51 PM, Chamila Wijayarathna < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi , >>>>>>>> Thank you for the information. >>>>>>>> We are planning to run this for without stopping for a long time. >>>>>>>> So will the 100000 limit exceed some day and give this error again? >>>>>>>> If so is there any way of avoiding that? >>>>>>>> >>>>>>> >>>>>>> Which is highly unlikely! So, don't worry! :) >>>>>>> >>>>>>>> >>>>>>>> Thank You. >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Feb 9, 2014 at 12:41 PM, Bhathiya Jayasekara < >>>>>>>> [email protected]> wrote: >>>>>>>> >>>>>>>>> FYI: You can do the same by editing /etc/security/limits.conf file >>>>>>>>> too. >>>>>>>>> >>>>>>>>> http://www.datastax.com/docs/1.0/troubleshooting/index#toomany >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Feb 9, 2014 at 12:18 PM, Nirmal Fernando >>>>>>>>> <[email protected]>wrote: >>>>>>>>> >>>>>>>>>> To avoid this please increase the open files limit of your OS. >>>>>>>>>> 'ulimit -n' would give you the current number (by default it is 1024 >>>>>>>>>> - most >>>>>>>>>> of the cases). >>>>>>>>>> >>>>>>>>>> You can uncomment >>>>>>>>>> >>>>>>>>>> #ulimit -n 100000 (after uncommenting it would be : ulimit -n >>>>>>>>>> 100000 ) >>>>>>>>>> >>>>>>>>>> in the wso2server.sh file of wso2 products in order to increase >>>>>>>>>> the value. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sun, Feb 9, 2014 at 12:02 PM, Chamila Wijayarathna < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hello all, >>>>>>>>>>> I am working on "Infra Log Analyzer" intern project which uses >>>>>>>>>>> BAM and CEP. Here we uses 2 instances and BAM and CEP communicating >>>>>>>>>>> with >>>>>>>>>>> each other. >>>>>>>>>>> When I add these two into a remote server and ran it I am >>>>>>>>>>> getting following warning and error. >>>>>>>>>>> >>>>>>>>>>> [*2014-02-09 05:05:42,669] WARN >>>>>>>>>>> {org.apache.cassandra.thrift.CustomTThreadPoolServer} - Transport >>>>>>>>>>> error >>>>>>>>>>> occurred during acceptance of message.* >>>>>>>>>>> *org.apache.thrift.transport.TTransportException: >>>>>>>>>>> java.net.SocketException: Too many open files* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:103)* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:39)* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:100)* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:213)* >>>>>>>>>>> *Caused by: java.net.SocketException: Too many open files* >>>>>>>>>>> * at java.net.PlainSocketImpl.socketAccept(Native Method)* >>>>>>>>>>> * at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)* >>>>>>>>>>> * at java.net.ServerSocket.implAccept(ServerSocket.java:462)* >>>>>>>>>>> * at java.net.ServerSocket.accept(ServerSocket.java:430)* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:97)* >>>>>>>>>>> * ... 4 more* >>>>>>>>>>> *[2014-02-09 05:05:42,666] ERROR >>>>>>>>>>> {org.apache.thrift.transport.TSocket} - Could not configure >>>>>>>>>>> socket.* >>>>>>>>>>> *java.net.SocketException: Too many open files* >>>>>>>>>>> * at java.net.Socket.createImpl(Socket.java:397)* >>>>>>>>>>> * at java.net.Socket.getImpl(Socket.java:460)* >>>>>>>>>>> * at java.net.Socket.setSoLinger(Socket.java:900)* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.thrift.transport.TSocket.initSocket(TSocket.java:116)* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.thrift.transport.TSocket.<init>(TSocket.java:107)* >>>>>>>>>>> * at >>>>>>>>>>> me.prettyprint.cassandra.connection.client.HThriftClient.open(HThriftClient.java:123)* >>>>>>>>>>> * at >>>>>>>>>>> me.prettyprint.cassandra.connection.client.HThriftClient.open(HThriftClient.java:26)* >>>>>>>>>>> * at >>>>>>>>>>> me.prettyprint.cassandra.connection.ConcurrentHClientPool.createClient(ConcurrentHClientPool.java:147)* >>>>>>>>>>> * at >>>>>>>>>>> me.prettyprint.cassandra.connection.ConcurrentHClientPool.<init>(ConcurrentHClientPool.java:53)* >>>>>>>>>>> * at >>>>>>>>>>> me.prettyprint.cassandra.connection.RoundRobinBalancingPolicy.createConnection(RoundRobinBalancingPolicy.java:67)* >>>>>>>>>>> * at >>>>>>>>>>> me.prettyprint.cassandra.connection.HConnectionManager.<init>(HConnectionManager.java:67)* >>>>>>>>>>> * at >>>>>>>>>>> me.prettyprint.cassandra.service.AbstractCluster.<init>(AbstractCluster.java:67)* >>>>>>>>>>> * at >>>>>>>>>>> me.prettyprint.cassandra.service.ThriftCluster.<init>(ThriftCluster.java:21)* >>>>>>>>>>> * at >>>>>>>>>>> org.wso2.carbon.bam.cassandra.data.archive.mapred.CassandraMapReduceRowDeletion$RowKeyMapper.setup(CassandraMapReduceRowDeletion.java:119)* >>>>>>>>>>> * at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)* >>>>>>>>>>> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)* >>>>>>>>>>> * at >>>>>>>>>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:211)* >>>>>>>>>>> >>>>>>>>>>> I get this when server runs for a while about 1.5 - 2 hours. I >>>>>>>>>>> get the same error in both BAM and CEP. >>>>>>>>>>> I didn't get this when I ran them on my local machine. >>>>>>>>>>> What is the reason for this? How can I avoid this? >>>>>>>>>>> >>>>>>>>>>> Thank You! >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Chamila Wijayarathna* >>>>>>>>>>> Engineering Intern, >>>>>>>>>>> WSO2 Inc. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> _______________________________________________ >>>>>>>>>>> Dev mailing list >>>>>>>>>>> [email protected] >>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> >>>>>>>>>> Thanks & regards, >>>>>>>>>> Nirmal >>>>>>>>>> >>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>>>> Mobile: +94715779733 >>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> _______________________________________________ >>>>>>>>>> Dev mailing list >>>>>>>>>> [email protected] >>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> *Bhathiya Jayasekara* >>>>>>>>> *Software Engineer,* >>>>>>>>> *WSO2 inc., http://wso2.com <http://wso2.com>* >>>>>>>>> >>>>>>>>> *Phone: +94715478185 <%2B94715478185>* >>>>>>>>> *LinkedIn: http://www.linkedin.com/in/bhathiyaj >>>>>>>>> <http://www.linkedin.com/in/bhathiyaj>* >>>>>>>>> *Twitter: https://twitter.com/bhathiyax >>>>>>>>> <https://twitter.com/bhathiyax>* >>>>>>>>> *Blog: http://movingaheadblog.blogspot.com >>>>>>>>> <http://movingaheadblog.blogspot.com/>* >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Chamila Wijayarathna* >>>>>>>> Engineering Intern, >>>>>>>> WSO2 Inc. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Thanks & regards, >>>>>>> Nirmal >>>>>>> >>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>> Mobile: +94715779733 >>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> *Chamila Wijayarathna* >>>>>> Engineering Intern, >>>>>> WSO2 Inc. >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Chamila Wijayarathna* >>>>> Engineering Intern, >>>>> WSO2 Inc. >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> *Chamila Wijayarathna* >>> Engineering Intern, >>> WSO2 Inc. >>> >>> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > *Chamila Wijayarathna* > Engineering Intern, > WSO2 Inc. > > > _______________________________________________ > Dev mailing list > [email protected] > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- *Lasantha Fernando* Software Engineer - Data Technologies Team WSO2 Inc. http://wso2.com email: [email protected] mobile: (+94) 71 5247551
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
