Re: Hard Disk is full because of OpenShift Origin

2017-10-27 Thread Graham Dumpleton 
A question for OP. Are you using options to oc cluster up to persist data when 
shutting it down?

> On 27 Oct 2017, at 10:58 pm, Mauricio Améndola  
> wrote:
> 
> Hello,
> The correct way to remove old images is using “oadm prune….”[1]  command. I 
> remember that there are two folders that increase a lot due to tmp files. 
> 
> - /var/lib/origin
> - /var/lib/docker
> 
> Try oadm prune and give some feedback
> Regards,
> 
> [1] 
> https://docs.openshift.com/container-platform/3.6/admin_guide/pruning_resources.html
>  
> 
> 
> 
>> On Oct 26, 2017, at 6:37 PM, Tien Hung Nguyen > > wrote:
>> 
>> Hi everybody,
>> 
>> I have a problem with my hard drive space. Since I'm using OpenShift locally 
>> with Docker, I have the problem that my hard drive space gets full very fast 
>> and I can't remove it. I have already run the commands 'oc cluster down' and 
>> the 'docker rmi [imageip] commands to deleted unused images but it has no 
>> effect.
>> 
>> Please, could you tell me how to free up my disk space properly?
>> 
>> Thank you!
>> 
>> ___
>> users mailing list
>> users@lists.openshift.redhat.com 
>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
> 
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Hard Disk is full because of OpenShift Origin

2017-10-27 Thread Mauricio Améndola 
Hello,
The correct way to remove old images is using “oadm prune….”[1]  command. I 
remember that there are two folders that increase a lot due to tmp files. 

- /var/lib/origin
- /var/lib/docker

Try oadm prune and give some feedback
Regards,

[1] 
https://docs.openshift.com/container-platform/3.6/admin_guide/pruning_resources.html
 



> On Oct 26, 2017, at 6:37 PM, Tien Hung Nguyen  
> wrote:
> 
> Hi everybody,
> 
> I have a problem with my hard drive space. Since I'm using OpenShift locally 
> with Docker, I have the problem that my hard drive space gets full very fast 
> and I can't remove it. I have already run the commands 'oc cluster down' and 
> the 'docker rmi [imageip] commands to deleted unused images but it has no 
> effect.
> 
> Please, could you tell me how to free up my disk space properly?
> 
> Thank you!
> 
> ___
> users mailing list
> users@lists.openshift.redhat.com
> http://lists.openshift.redhat.com/openshiftmm/listinfo/users

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

faulty diagnostics?

2017-10-27 Thread Tim Dudgeon 
I've been looking at using the diagnostics (oc adm diagnostics) to test 
the status of a cluster installed with the ansible installer and 
consistently see things that seem to be false alarms. The cluster 
appears to be functions (builds run, can push to registry and routes are 
working etc.). This is with origin 3.6.0.


1. This is consistently seen, and a restart of the master  does not fix 
it. The name docker-registry.default.svc resolve tot he ip address 
172.30.200.62


ERROR: [DClu1019 from diagnostic 
ClusterRegistry@openshift/origin/pkg/diagnostics/cluster/registry.go:343]

   Diagnostics created a test ImageStream and compared the registry IP
   it received to the registry IP available via the 
docker-registry service.


   docker-registry  : 172.30.200.62:5000
   ImageStream registry : docker-registry.default.svc:5000

   They do not match, which probably means that an administrator 
re-created
   the docker-registry service but the master has cached the old 
service
   IP address. Builds or deployments that use ImageStreams with 
the wrong

   docker-registry IP will fail under this condition.

   To resolve this issue, restarting the master (to clear the 
cache) should

   be sufficient. Existing ImageStreams may need to be re-created.

2. This warning is seen

WARN:  [DClu0003 from diagnostic 
NodeDefinition@openshift/origin/pkg/diagnostics/cluster/node_definitions.go:113]
   Node ip-10-0-247-194.eu-west-1.compute.internal is ready but is 
marked Unschedulable.

   This is usually set manually for administrative reasons.
   An administrator can mark the node schedulable with:
   oadm manage-node ip-10-0-247-194.eu-west-1.compute.internal 
--schedulable=true


   While in this state, pods should not be scheduled to deploy on 
the node.
   Existing pods will continue to run until completed or evacuated 
(see

   other options for 'oadm manage-node').

This is for the master node which by default is non-schedulable.

3. If metrics and logging are not deployed you see this warning:

WARN:  [DH0005 from diagnostic 
MasterConfigCheck@openshift/origin/pkg/diagnostics/host/check_master_config.go:52]
   Validation of master config file 
'/etc/origin/master/master-config.yaml' warned:
   assetConfig.loggingPublicURL: Invalid value: "": required to 
view aggregated container logs in the console
   assetConfig.metricsPublicURL: Invalid value: "": required to 
view cluster metrics in the console
   auditConfig.auditFilePath: Required value: audit can not be 
logged to a separate file


Whilst 2 and 3 could be considered minor irritations, 1 might scare 
people that something is actually wrong.


Also, the 'oc adm diagnostics' command need to be run as root or with 
sudo otherwise you get some file permissions related errors. I don't 
think this is mentioned in the docs.


Tim

___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users

Re: Hard Disk is full because of OpenShift Origin

2017-10-27 Thread Graham Dumpleton 
What platform are you on?

If you are on MacOS X or Windows try the following.

Run:

docker run --rm -it -v /:/mnt busybox /bin/sh

This will give you an interactive shell in busybox, but where /mnt has the 
Docker VM file system mounted.

Then go to:

   /mnt/var/lib/origin

and see how much space is being taken up by running 'du -ks .'.

There is a chance that if you were using persistent volumes in OpenShift, that 
any data is still taking up space if the directories weren't cleaned up.

This though is data that the Docker VM would be taking up. If the VM disk file 
has expanded, then removing anything in it may not reclaim it. Not sure how VM 
disk space works and whether can reclaim or whether just expands.

I am not sure where the comparable directory is when using Docker service on 
Linux.

For that you could perhaps run:

docker exec -it origin bash

when oc cluster up is running and then go to:

/var/lib/origin/

in the origin container and see space consumed there. Will be same directory as 
get to above, and mounted from external place Docker service has it.

Graham

> On 27 Oct 2017, at 4:50 pm, Tien Hung Nguyen  wrote:
> 
> I have checked the volumes already using the docker command docker volume ls. 
> However, everything seems pretty normal there.
> 
> Furthermore, I have checked my images with docker images ls. I have found a 
> lot of old images created by OpenShift, which used a lot of GBs on my Hard 
> Disk. That‘s why I started to manually delete those images with the docker 
> command: docker rmi [imageid], but that didn‘t change my hard disk space at 
> all. 
> 
> Do you have any other ideas?
> 
>> Am 27.10.2017 um 01:24 schrieb Graham Dumpleton :
>> 
>> See if you have a lot of docker volumes that haven't been cleaned up 
>> properly.
>> 
>>   docker volume ls
>> 
>> Maybe that is hanging onto space.
>> 
>> Graham
>> 
>>> On 27 Oct 2017, at 8:37 am, Tien Hung Nguyen  
>>> wrote:
>>> 
>>> Hi everybody,
>>> 
>>> I have a problem with my hard drive space. Since I'm using OpenShift 
>>> locally with Docker, I have the problem that my hard drive space gets full 
>>> very fast and I can't remove it. I have already run the commands 'oc 
>>> cluster down' and the 'docker rmi [imageip] commands to deleted unused 
>>> images but it has no effect.
>>> 
>>> Please, could you tell me how to free up my disk space properly?
>>> 
>>> Thank you!
>>> 
>>> ___
>>> users mailing list
>>> users@lists.openshift.redhat.com
>>> http://lists.openshift.redhat.com/openshiftmm/listinfo/users
>> 


___
users mailing list
users@lists.openshift.redhat.com
http://lists.openshift.redhat.com/openshiftmm/listinfo/users