I debugged with Chamila, and seems like in his remote server, normal users are not allowed to have a ulimit >32768. That's why even we set to 100000, it showed as 32768 for both BAM and CEP. Then in order to validate whether my above argument is correct, I've asked him to login as root user and set ulimit manually in a terminal. And then we could see that the ulimit got set correctly.
Then we've started servers as root user and got the correct limit values. There might be a setting to change the limit put upon normal user, which needs to be Googled :) On Sun, Feb 9, 2014 at 6:23 PM, Chamila Wijayarathna <[email protected]>wrote: > Hi, > I ran "ulimit -n 100000" as root user and then ran BAM as root user. > Now it shows as "Max open files 100000 100000 > files" in /proc/<pid>/limits file. > > Thank You. > > > On Sun, Feb 9, 2014 at 6:15 PM, Lasantha Fernando <[email protected]>wrote: > >> Hi Chamila, >> >> Does the output of `cat /proc/<pid>/limits` show the ulimit properly set >> for BAM as well? If it was set properly for BAM, it is strange that it >> doesn't set properly for CEP...? >> >> You might have stumbled upon across something like the article in [1]. If >> not, can you go through and see if the user-level and system-wide maximum >> file descriptor limits are set properly. Also are you running both BAM and >> CEP as the same user? >> >> [1] >> http://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/ >> >> Thanks, >> Lasantha >> >> >> On 9 February 2014 17:46, Chamila Wijayarathna <[email protected]> wrote: >> >>> When I opened CEP wso2server.sh file it has "ulimit -n 100000 " line on >>> it which I uncommented. >>> >>> >>> On Sun, Feb 9, 2014 at 5:41 PM, Nirmal Fernando <[email protected]> wrote: >>> >>>> It's not set it seems. You should see something like: >>>> >>>> Max open files 100000 100000 >>>> files >>>> >>>> >>>> On Sun, Feb 9, 2014 at 5:37 PM, Chamila Wijayarathna >>>> <[email protected]>wrote: >>>> >>>>> It's about the same time as the previous occasion. >>>>> >>>>> limit file had the following content. >>>>> >>>>> Limit Soft Limit Hard Limit >>>>> Units >>>>> Max cpu time unlimited unlimited >>>>> seconds >>>>> Max file size unlimited unlimited >>>>> bytes >>>>> Max data size unlimited unlimited >>>>> bytes >>>>> Max stack size 8388608 unlimited >>>>> bytes >>>>> Max core file size 0 unlimited >>>>> bytes >>>>> Max resident set unlimited unlimited >>>>> bytes >>>>> Max processes unlimited unlimited >>>>> processes >>>>> Max open files 32768 32768 >>>>> files >>>>> Max locked memory 65536 65536 >>>>> bytes >>>>> Max address space unlimited unlimited >>>>> bytes >>>>> Max file locks unlimited unlimited >>>>> locks >>>>> Max pending signals 16382 16382 >>>>> signals >>>>> Max msgqueue size 819200 819200 >>>>> bytes >>>>> Max nice priority 0 0 >>>>> Max realtime priority 0 0 >>>>> Max realtime timeout unlimited unlimited us >>>>> >>>>> Thank You. >>>>> >>>>> >>>>> On Sun, Feb 9, 2014 at 5:16 PM, Nirmal Fernando <[email protected]>wrote: >>>>> >>>>>> hmm.. is it at least after sometime than the previous occasion? Also, >>>>>> did you verify that the limit has been set correctly (using the limits >>>>>> file >>>>>> of process)? >>>>>> >>>>>> >>>>>> On Sun, Feb 9, 2014 at 5:04 PM, Chamila Wijayarathna < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Hi, I did edit the wso2server.sh files of both CEP and BAM. Now BAM >>>>>>> is not giving the exception. But still CEP gives the same exception. >>>>>>> >>>>>>> Thank You. >>>>>>> >>>>>>> >>>>>>> On Sun, Feb 9, 2014 at 1:04 PM, Chamila Wijayarathna < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> Thank You for information. >>>>>>>> >>>>>>>> >>>>>>>> On Sun, Feb 9, 2014 at 12:54 PM, Nirmal Fernando >>>>>>>> <[email protected]>wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> On Sun, Feb 9, 2014 at 12:51 PM, Chamila Wijayarathna < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi , >>>>>>>>>> Thank you for the information. >>>>>>>>>> We are planning to run this for without stopping for a long time. >>>>>>>>>> So will the 100000 limit exceed some day and give this error again? >>>>>>>>>> If so is there any way of avoiding that? >>>>>>>>>> >>>>>>>>> >>>>>>>>> Which is highly unlikely! So, don't worry! :) >>>>>>>>> >>>>>>>>>> >>>>>>>>>> Thank You. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Sun, Feb 9, 2014 at 12:41 PM, Bhathiya Jayasekara < >>>>>>>>>> [email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> FYI: You can do the same by editing /etc/security/limits.conf file >>>>>>>>>>> too. >>>>>>>>>>> >>>>>>>>>>> http://www.datastax.com/docs/1.0/troubleshooting/index#toomany >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Sun, Feb 9, 2014 at 12:18 PM, Nirmal Fernando < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> To avoid this please increase the open files limit of your OS. >>>>>>>>>>>> 'ulimit -n' would give you the current number (by default it is >>>>>>>>>>>> 1024 - most >>>>>>>>>>>> of the cases). >>>>>>>>>>>> >>>>>>>>>>>> You can uncomment >>>>>>>>>>>> >>>>>>>>>>>> #ulimit -n 100000 (after uncommenting it would be : ulimit -n >>>>>>>>>>>> 100000 ) >>>>>>>>>>>> >>>>>>>>>>>> in the wso2server.sh file of wso2 products in order to increase >>>>>>>>>>>> the value. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Sun, Feb 9, 2014 at 12:02 PM, Chamila Wijayarathna < >>>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hello all, >>>>>>>>>>>>> I am working on "Infra Log Analyzer" intern project which uses >>>>>>>>>>>>> BAM and CEP. Here we uses 2 instances and BAM and CEP >>>>>>>>>>>>> communicating with >>>>>>>>>>>>> each other. >>>>>>>>>>>>> When I add these two into a remote server and ran it I am >>>>>>>>>>>>> getting following warning and error. >>>>>>>>>>>>> >>>>>>>>>>>>> [*2014-02-09 05:05:42,669] WARN >>>>>>>>>>>>> {org.apache.cassandra.thrift.CustomTThreadPoolServer} - >>>>>>>>>>>>> Transport error >>>>>>>>>>>>> occurred during acceptance of message.* >>>>>>>>>>>>> *org.apache.thrift.transport.TTransportException: >>>>>>>>>>>>> java.net.SocketException: Too many open files* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:103)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:39)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.cassandra.thrift.CustomTThreadPoolServer.serve(CustomTThreadPoolServer.java:100)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.cassandra.thrift.CassandraDaemon$ThriftServer.run(CassandraDaemon.java:213)* >>>>>>>>>>>>> *Caused by: java.net.SocketException: Too many open files* >>>>>>>>>>>>> * at java.net.PlainSocketImpl.socketAccept(Native Method)* >>>>>>>>>>>>> * at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)* >>>>>>>>>>>>> * at java.net.ServerSocket.implAccept(ServerSocket.java:462)* >>>>>>>>>>>>> * at java.net.ServerSocket.accept(ServerSocket.java:430)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.cassandra.thrift.TCustomServerSocket.acceptImpl(TCustomServerSocket.java:97)* >>>>>>>>>>>>> * ... 4 more* >>>>>>>>>>>>> *[2014-02-09 05:05:42,666] ERROR >>>>>>>>>>>>> {org.apache.thrift.transport.TSocket} - Could not configure >>>>>>>>>>>>> socket.* >>>>>>>>>>>>> *java.net.SocketException: Too many open files* >>>>>>>>>>>>> * at java.net.Socket.createImpl(Socket.java:397)* >>>>>>>>>>>>> * at java.net.Socket.getImpl(Socket.java:460)* >>>>>>>>>>>>> * at java.net.Socket.setSoLinger(Socket.java:900)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.thrift.transport.TSocket.initSocket(TSocket.java:116)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.thrift.transport.TSocket.<init>(TSocket.java:107)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> me.prettyprint.cassandra.connection.client.HThriftClient.open(HThriftClient.java:123)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> me.prettyprint.cassandra.connection.client.HThriftClient.open(HThriftClient.java:26)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> me.prettyprint.cassandra.connection.ConcurrentHClientPool.createClient(ConcurrentHClientPool.java:147)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> me.prettyprint.cassandra.connection.ConcurrentHClientPool.<init>(ConcurrentHClientPool.java:53)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> me.prettyprint.cassandra.connection.RoundRobinBalancingPolicy.createConnection(RoundRobinBalancingPolicy.java:67)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> me.prettyprint.cassandra.connection.HConnectionManager.<init>(HConnectionManager.java:67)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> me.prettyprint.cassandra.service.AbstractCluster.<init>(AbstractCluster.java:67)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> me.prettyprint.cassandra.service.ThriftCluster.<init>(ThriftCluster.java:21)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.wso2.carbon.bam.cassandra.data.archive.mapred.CassandraMapReduceRowDeletion$RowKeyMapper.setup(CassandraMapReduceRowDeletion.java:119)* >>>>>>>>>>>>> * at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)* >>>>>>>>>>>>> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)* >>>>>>>>>>>>> * at >>>>>>>>>>>>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:211)* >>>>>>>>>>>>> >>>>>>>>>>>>> I get this when server runs for a while about 1.5 - 2 hours. I >>>>>>>>>>>>> get the same error in both BAM and CEP. >>>>>>>>>>>>> I didn't get this when I ran them on my local machine. >>>>>>>>>>>>> What is the reason for this? How can I avoid this? >>>>>>>>>>>>> >>>>>>>>>>>>> Thank You! >>>>>>>>>>>>> >>>>>>>>>>>>> -- >>>>>>>>>>>>> *Chamila Wijayarathna* >>>>>>>>>>>>> Engineering Intern, >>>>>>>>>>>>> WSO2 Inc. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>>> Dev mailing list >>>>>>>>>>>>> [email protected] >>>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> -- >>>>>>>>>>>> >>>>>>>>>>>> Thanks & regards, >>>>>>>>>>>> Nirmal >>>>>>>>>>>> >>>>>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>>>>>> Mobile: +94715779733 >>>>>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> _______________________________________________ >>>>>>>>>>>> Dev mailing list >>>>>>>>>>>> [email protected] >>>>>>>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> *Bhathiya Jayasekara* >>>>>>>>>>> *Software Engineer,* >>>>>>>>>>> *WSO2 inc., http://wso2.com <http://wso2.com>* >>>>>>>>>>> >>>>>>>>>>> *Phone: +94715478185 <%2B94715478185>* >>>>>>>>>>> *LinkedIn: http://www.linkedin.com/in/bhathiyaj >>>>>>>>>>> <http://www.linkedin.com/in/bhathiyaj>* >>>>>>>>>>> *Twitter: https://twitter.com/bhathiyax >>>>>>>>>>> <https://twitter.com/bhathiyax>* >>>>>>>>>>> *Blog: http://movingaheadblog.blogspot.com >>>>>>>>>>> <http://movingaheadblog.blogspot.com/>* >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> *Chamila Wijayarathna* >>>>>>>>>> Engineering Intern, >>>>>>>>>> WSO2 Inc. >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> >>>>>>>>> Thanks & regards, >>>>>>>>> Nirmal >>>>>>>>> >>>>>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>>>>> Mobile: +94715779733 >>>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> *Chamila Wijayarathna* >>>>>>>> Engineering Intern, >>>>>>>> WSO2 Inc. >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Chamila Wijayarathna* >>>>>>> Engineering Intern, >>>>>>> WSO2 Inc. >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Chamila Wijayarathna* >>>>> Engineering Intern, >>>>> WSO2 Inc. >>>>> >>>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> *Chamila Wijayarathna* >>> Engineering Intern, >>> WSO2 Inc. >>> >>> >>> _______________________________________________ >>> Dev mailing list >>> [email protected] >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> *Lasantha Fernando* >> Software Engineer - Data Technologies Team >> WSO2 Inc. http://wso2.com >> >> email: [email protected] >> mobile: (+94) 71 5247551 >> > > > > -- > *Chamila Wijayarathna* > Engineering Intern, > WSO2 Inc. > > -- Thanks & regards, Nirmal Senior Software Engineer- Platform Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
