My 2 cents... >>> I have 141 projects, >>> 30,373 builds, and 1,073,591 junit XML test results. It took me over 20 >>> minutes of CPU time to get these values by querying the file system. >> >> Yes, build storage is a well-known architectural problem. PR 1641 will >> not help you here.
Just to add another data point that might be interesting. I am running a large Jenkins instance (version 1.565.1) with 123 different plugins installed. It contains, 1,021 projects (711 are matrix jobs) with 26,040 builds (454,575 matrix sub-job builds) and 14,988,971 test results. The build results take up about 1.8TB disk space. The Jenkins master runs on a 24 core machine, with 64GB RAM, 2.5TB RAID 10 storage (~ 200 MB/sec peak I/O). The build slaves are run on secondary machines. The startup time for this Jenkins master is about 40-60min. Taking the raw I/O performance on this system into account, the theoretical absolute lower limit should be just under 6min. Reading all my test XML reports directly from the filesystem took about 31min. For comparison, accessing these same results through the REST API takes about 45min. >>> Will Jenkins 2.0 have database backend? >> >> No, the changes required are too big. There is no way to do it that >> quickly. I think we *should* do these changes sooner or later. For me >> this seems both more important and more disruptive than the code >> changes currently being discussed for 2.0. > > Yeah, I tend to agree. It is an important issue, but as Jesse says, > I'm afraid it's too disruptive than all the boundaries that we operate in. > > We should still take it on, but it shouldn't be tied to Jenkins 2.0. Not knowing much about the internals of Jenkins, my suspicion is that most plugins deal with I/O by themselves. If this is indeed the case, then in my experience it is usually bad for performance to let everyone do their own thing for I/O. I would then suggest as a first step to enforce plugins to use a unified I/O interface. Later it would be much easier to redirect that to a DB backend. Perhaps this idea has already been raised by the core developers. :-) Kind regards. Artur -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/3C350C9D1C9DEE47BAEA43102B433C9EBE5DD5BE%40HQMBX2.ads.eso.org. For more options, visit https://groups.google.com/d/optout.
