Yes. We're moving the filesystem from ext4 to xfs. ext4 doesn't allow for dynamically increasing the inode allocation without also increasing the storage space. xfs does allow for this retuning of the inode allocation on the fly allowing us to more sanely shift the space between inodes and actual storage.
As for the purging of really old logs. We're trying to avoid doing that if at all possible but we will if it ends up being the best way forward to getting this issued cleared. -Andy- On 02/05/2018 07:13 AM, Sam Hague wrote: > Will the new fs allow keeping logs for 6 months again? > > On Mon, Feb 5, 2018 at 9:49 AM, Thanh Ha <[email protected] > <mailto:[email protected]>> wrote: > > On Mon, Feb 5, 2018 at 9:27 AM, Thanh Ha > <[email protected] <mailto:[email protected]>> > wrote: > > Hi Everyone, > > We're temporarily taking Nexus down. This is unscheduled > maintenance as we have a serious issue with Nexus not being able > to create new files at the moment. On Saturday Nexus ran out of > inodes on the file system and we temporarily mitigated it while > we are waiting for data to be migrated to a new file system that > an address more inodes. Unfortunately we hit this issue again > this morning (which is why many jobs failed with 500 - Internal > Server Error) when they try to upload artifacts to Nexus. > > We're going to be removing some older logs on the server to > mitigate this issue while our data migration is still > continuing. Nexus is being shutdown temporarily so we can > restart it after we free up some inodes. > > Regards, > Thanh > > > We've cleared up about 3 million inodes and restarted Nexus. Jenkins > is now back online and should be starting jobs again shortly. We're > going to look into removing older logs from the system as logs > create many small files which means they are easy targets to get > back some inode space... Normally we keep logs for 6 months but we > will look into removing 4 month old logs to get back some more > inodes while our server continues to migrate data to the new file > system. > > Apologies for the inconvenience. > > Regards, > Thanh > > > _______________________________________________ > infrastructure mailing list > [email protected] > <mailto:[email protected]> > https://lists.opendaylight.org/mailman/listinfo/infrastructure > <https://lists.opendaylight.org/mailman/listinfo/infrastructure> > > > > > _______________________________________________ > infrastructure mailing list > [email protected] > https://lists.opendaylight.org/mailman/listinfo/infrastructure > -- Andrew J Grimberg Lead, IT Release Engineering The Linux Foundation
signature.asc
Description: OpenPGP digital signature
_______________________________________________ infrastructure mailing list [email protected] https://lists.opendaylight.org/mailman/listinfo/infrastructure
