sodonnel commented on PR #4857: URL: https://github.com/apache/ozone/pull/4857#issuecomment-1584332318
I only quickly looked over the code, but I was wondering how this is intended to be used? There doesn't seem to be any way to direct certain containers to SSD / RAM / HDD? It appears you set the placement policy on all or some DNs to be SSD / HDD and then only on replication the containers will goto that type of disk on the target DN. Replication only happens if something goes wrong on the cluster (eg node goes down, unhealthy container etc) or the balancer is run. If open containers are created on HDD and then the replication policy says SSD, containers are just going to be on a mixture of disk types with no real control over where they are going or when they might move some replicas to a different storage type. I wonder if we need a wider design around storage policies, as it is a more difficult problem in Ozone than in HDFS. For example in HDFS a block is a standalone entity associated with a file. That file can have a storage policy set against it. In Ozone, a block is in a container which is shared by many different keys of the same replication type. It is not so simple to say a given key should be marked HOT or Archive it affects the entire contents of the container. I also think there is some parallel work going on with the data temperature heat map, and using that to move containers around, but I don't think I have seen anything about how that heat map is going to deal with containers holding a mix of hot / cold / archive data. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
