Hi Imesh, I had a chat with Reka and found out due to high loadAverage, more containers are spawned by AS. This causes the "No space left on device". Since I am working with 1 minion, at first there is 3 applications for dev,test and prod and when they started to scale the VM left with no space.
On Wed, Jul 15, 2015 at 6:33 AM, Imesh Gunaratne <[email protected]> wrote: > May be you need more disk space on each minion host. > > On Tue, Jul 14, 2015 at 12:00 PM, Punnadi Gunarathna <[email protected]> > wrote: > >> Hi Manisha, >> >> No it can't be done with puppet scripts at the moment. I was referring to >> kubernets minion which had space issue. But sure I will have to figure out >> a way to solve this. >> >> On Tue, Jul 14, 2015 at 7:26 AM, Manisha Gayathri <[email protected]> >> wrote: >> >>> If we are doing a new setup, we need to clean the old existing >>> instances, don't we? >>> >>> Hope we can automate this step via puppet scripts. >>> >>> Thanks >>> Manisha >>> >>> On Mon, Jul 13, 2015 at 7:22 PM, Punnadi Gunarathna <[email protected]> >>> wrote: >>> >>>> Hi All, >>>> >>>> Despite cleaning pods via kubernetes client, I noticed there are number >>>> of old docker containers running on node-01 with below command: >>>> >>>> docker ps -a >>>> >>>> >>>> So I deleted them with below command: >>>> >>>> docker ps -a | grep '<CREATED>' | awk '{print $1}' | xargs >>>> --no-run-if-empty docker rm -f >>>> >>>> e.g: >>>> >>>> docker ps -a | grep '3 hours ago' | awk '{print $1}' | xargs >>>> --no-run-if-empty docker rm -f >>>> >>>> Hope this will solve the problem. >>>> >>>> On Mon, Jul 13, 2015 at 6:59 PM, Punnadi Gunarathna <[email protected]> >>>> wrote: >>>> >>>>> hi All, >>>>> >>>>> I got below error in all the docker containers for the setup i just >>>>> did and haven't seen it before. >>>>> >>>>> In WSO2AS: >>>>> >>>>> TID: [2015-07-13 13:09:38,746] INFO >>>>> {org.wso2.carbon.server.extensions.PatchInstaller} - Patch changes >>>>> detected {org.wso2.carbon.server.extensions.PatchInstaller} >>>>> TID: [2015-07-13 13:09:57,267] INFO >>>>> {org.wso2.carbon.server.util.PatchUtils.console} - Backed up plugins to >>>>> patch0000 {org.wso2.carbon.server.util.PatchUtils.console} >>>>> TID: [2015-07-13 13:10:05,378] ERROR >>>>> {org.wso2.carbon.server.extensions.PatchInstaller} - Error occurred while >>>>> applying patches {org.wso2.carbon.server.extensions.PatchInstaller} >>>>> java.io.IOException: No space left on device >>>>> at java.io.FileOutputStream.writeBytes(Native Method) >>>>> at java.io.FileOutputStream.write(FileOutputStream.java:345) >>>>> at org.wso2.carbon.server.util.FileUtils.copy(FileUtils.java:230) >>>>> at >>>>> org.wso2.carbon.server.util.FileUtils.copyFile(FileUtils.java:152) >>>>> at >>>>> org.wso2.carbon.server.util.PatchUtils.copyServicepacksAndPatches(PatchUtils.java:90) >>>>> at >>>>> org.wso2.carbon.server.util.PatchUtils.applyServicepacksAndPatches(PatchUtils.java:45) >>>>> at >>>>> org.wso2.carbon.server.extensions.PatchInstaller.perform(PatchInstaller.java:58) >>>>> at org.wso2.carbon.server.Main.invokeExtensions(Main.java:152) >>>>> at org.wso2.carbon.server.Main.main(Main.java:94) >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) >>>>> at >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> at java.lang.reflect.Method.invoke(Method.java:606) >>>>> at org.wso2.carbon.bootstrap.Bootstrap.loadClass(Bootstrap.java:63) >>>>> at org.wso2.carbon.bootstrap.Bootstrap.main(Bootstrap.java:45) >>>>> >>>>> CA: >>>>> > [2015-07-13 13:10:29,565] DEBUG >>>>> {eventhandler.py:on_complete_tenant_event} - Processing Complete tenant >>>>> event... >>>>> > Traceback (most recent call last): >>>>> > File "/usr/lib/python2.7/logging/__init__.py", line 874, in emit >>>>> > self.flush() >>>>> > File "/usr/lib/python2.7/logging/__init__.py", line 834, in flush >>>>> > self.stream.flush() >>>>> > IOError: [Errno 28] No space left on device >>>>> > Logged from file eventhandler.py, line 221 >>>>> > [2015-07-13 13:10:29,565] DEBUG >>>>> {eventhandler.py:on_complete_tenant_event} - Complete tenants:[] >>>>> > Traceback (most recent call last): >>>>> > File "/usr/lib/python2.7/logging/__init__.py", line 874, in emit >>>>> > self.flush() >>>>> > File "/usr/lib/python2.7/logging/__init__.py", line 834, in flush >>>>> > self.stream.flush() >>>>> > IOError: [Errno 28] No space left on device >>>>> >>>>> >>>>> Whenever I do a new setup, I always clean all the pods with kubernetes >>>>> client. Any thoughts on this? >>>>> -- >>>>> Thanks and Regards, >>>>> >>>>> Punnadi Gunarathna >>>>> Senior Software Engineer, >>>>> WSO2, Inc.; http://wso2.com <http://wso2> >>>>> Blog: http://hi-my-world.blogspot.com/ >>>>> Tel : 94 11 214 5345 >>>>> Fax :94 11 2145300 >>>>> >>>>> >>>>> >>>>> <http://lalajisureshika.blogspot.com/> >>>>> >>>> >>>> >>>> >>>> -- >>>> Thanks and Regards, >>>> >>>> Punnadi Gunarathna >>>> Senior Software Engineer, >>>> WSO2, Inc.; http://wso2.com <http://wso2> >>>> Blog: http://hi-my-world.blogspot.com/ >>>> Tel : 94 11 214 5345 >>>> Fax :94 11 2145300 >>>> >>>> >>>> >>>> <http://lalajisureshika.blogspot.com/> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> [email protected] >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> ~Regards >>> *Manisha Eleperuma* >>> Senior Software Engineer >>> WSO2, Inc.: http://wso2.com >>> lean.enterprise.middleware >>> >>> *blog: http://manisha-eleperuma.blogspot.com/ >>> <http://manisha-eleperuma.blogspot.com/>* >>> *mobile: +94 71 8279777 <%2B94%2071%208279777>* >>> >>> >> >> >> -- >> Thanks and Regards, >> >> Punnadi Gunarathna >> Senior Software Engineer, >> WSO2, Inc.; http://wso2.com <http://wso2> >> Blog: http://hi-my-world.blogspot.com/ >> Tel : 94 11 214 5345 >> Fax :94 11 2145300 >> >> >> >> <http://lalajisureshika.blogspot.com/> >> >> _______________________________________________ >> Dev mailing list >> [email protected] >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *Imesh Gunaratne* > Senior Technical Lead > WSO2 Inc: http://wso2.com > T: +94 11 214 5345 M: +94 77 374 2057 > W: http://imesh.gunaratne.org > Lean . Enterprise . Middleware > > -- Thanks and Regards, Punnadi Gunarathna Senior Software Engineer, WSO2, Inc.; http://wso2.com <http://wso2> Blog: http://hi-my-world.blogspot.com/ Tel : 94 11 214 5345 Fax :94 11 2145300 <http://lalajisureshika.blogspot.com/>
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
