SSTATE_DIR and DL_DIR are writable. Sharing them among different builds
owned/controlled by the same user is OK, but sharing these writable
directories among different users does not seem to be a good idea.
You could consider using the PREMIRRORS and SSTATE_MIRRORS.
Regards,
Qi
On 9/11/23 09:49, Li Chen via lists.yoctoproject.org wrote:
Hi Yocto experts,
I'm a Yocto newbiee and I have a concern regarding the slowness of Yocto for
our team(around 100 people). There are two main reasons for this:
1. Yocto itself is slower compared to other build systems.
2. Each user in our team has their own private SSTATE_DIR and DL_DIR, which
means the artifacts cannot be shared. This leads to unnecessary disk and CPU
usage.
3. Every user works on SATA devices, so the IOPS and throughput are both low.
Since it's hard for us to invest in expensive enterprise NVMe for all build
servers, I have an idea to address this issue. My suggestion is to use
Lustre/NFS/Glustrefs along with small NVMe to share everyone's SSTATE/DL_DIR.
I prefer Lustre over NFS because Lustre offers superior IO performance
scalability, making it suitable for data-intensive tasks. It splits the data,
allowing it to be requested from one server but sent from one or more other
servers.
For the storage of SSTATE/DL_DIR only, the size requirement would not be high,
so a small NVMe should be sufficient. NVMe can provide much higher throughput
and IOPS compared to our current slow SATA, so I guess it can meet our needs.
If necessary, we can easily add another NVMe to another server since Lustre is
a distributed file system. The overall cost would not be significant.
My questions are:
1. Are there any errors in my understanding?
2. Is this solution feasible? If not, why? If yes, can it be further improved?
I look forward to hearing from you!
Regards,
Li
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#60964): https://lists.yoctoproject.org/g/yocto/message/60964
Mute This Topic: https://lists.yoctoproject.org/mt/101284965/21656
Group Owner: [email protected]
Unsubscribe: https://lists.yoctoproject.org/g/yocto/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
