I wonder the Java compiler take advantage for multiple cores/threads? Gary
On Sun, May 1, 2022, 13:11 Matt Sicker <[email protected]> wrote: > So I think we’re back at the build optimization problem. If we could > enable more parallelism there, that could help a lot as most of the modules > compile and test within a few seconds each. > > — > Matt Sicker > > > On May 1, 2022, at 06:56, Gary Gregory <[email protected]> wrote: > > > > And yes, one repo per module is another option, a la Maven as you point > > out, but what is being proposed is some halfway solution which to me > feels > > confusing and arbitrary just because today this and that module is in > favor > > but that some others. > > > > Gaty > > > >> On Sun, May 1, 2022, 07:54 Gary Gregory <[email protected]> wrote: > >> > >> Hi Volkan, > >> > >> I see how you build and how you do version control as orthogonal > concerns. > >> Currently we build and deliver everything from the root of the git > checkout > >> directory. It does not have to be that way is all I'm saying. One could > >> just cd to a sub directory and work there. You could still have the > option > >> to build the world from the root as a convenient way to make sure > >> everything works together and there is no breakage. > >> > >> https://en.m.wikipedia.org/wiki/Monorepo > >> > >> Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9] > all > >> employ very large monorepos with varying strategies to scale build > systems > >> and version control software with a large volume of code and daily > changes. > >> > >> Log4j is tiny in comparison. > >> > >> Gary > >> > >>> On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <[email protected]> wrote: > >>> > >>> I disagree that multi-repo setup is a recipe for disaster. It works > >>> perfectly well for Maven. Actually their setup is way more complicated: > >>> dedicated repo for each module! But let's assume you're right and > >>> multi-repos decrease the shelf time. Isn't that something good? I like > >>> repos that rot, it gives a clear message about that project: it simply > is > >>> dead. If we migrate to multiple repos and some never get to make a > >>> release, > >>> what does this mean? Apparently nothing interesting is happening here. > If > >>> there is a demand from the community to warrant a release, I am sure > that > >>> release will come. After all, the very same community attempted to > >>> resurrect Log4j 1. > >>> > >>> I also like single repo setups where build and test times are within > >>> bearable limits, though Log4j is not one of them. Note that multi-repo > >>> discussion started, because we don't know any other way to solve the > >>> current set of problems. If you can think of how to address these > concerns > >>> in some other way (e.g., "move folders around"), I am all ears. > >>> > >>> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <[email protected]> > >>> wrote: > >>> > >>>> Breaking our mono repo into a bunch of smaller repo pieces is a recipe > >>> for > >>>> software rot and orphanage IMO. > >>>> > >>>> For example, we have never released out of the tools repo, and I have > >>> folks > >>>> at work at use code from that repo for some internal facing work. > Now, I > >>>> could take the time and do a release of course but... priorities. > OTOH, > >>> I > >>>> could have gotten a release for "free" if that code was in the mono > >>> repo (I > >>>> am not advocating to put it back). > >>>> > >>>> This email might be better articulated in a call but I am trying to be > >>>> constructive: I think we can keep our mono repo and develop faster by > >>>> reorganizing what we have now. I like having everything in one place > >>>> myself. I am not sure how to actually move folders around to achieve > >>>> whatever new release cycle/split up we want, but I am sure it is do > >>> able. > >>>> > >>>> Gary > >>>> > >>>> On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <[email protected]> wrote: > >>>> > >>>>> I have raised this issue in the past. I even came up with a proposal > >>> on > >>>> the > >>>>> repository structure > >>>>> <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3> > >>> to be > >>>>> followed. I am a big proponent of breaking up the logging-log4j2 > >>>> repository > >>>>> into multiple ones. The current size of the code base is simply > >>>>> incomprehensible and slows down the development cycle tremendously > >>> due to > >>>>> excessive compilation and test times. > >>>>> > >>>>> *The proposal* > >>>>> > >>>>> Let me share the break down I have in mind: > >>>>> > >>>>> *logging-log4j2-api* > >>>>> log4j-1.2-api > >>>>> log4j-api > >>>>> > >>>>> *logging-log4j2* > >>>>> log4j-bom > >>>>> log4j-core > >>>>> log4j-core-its > >>>>> log4j-gctests > >>>>> log4j-iostreams > >>>>> log4j-perf > >>>>> log4j-plugins > >>>>> > >>>>> *logging-log4j2-appender* > >>>>> log4j-cassandra > >>>>> log4j-couchdb > >>>>> log4j-csv > >>>>> log4j-flume-ng > >>>>> log4j-mongodb3 > >>>>> log4j-mongodb4 > >>>>> log4j-smtp > >>>>> > >>>>> *logging-log4j2-appender-jdbc* > >>>>> log4j-jdbc (?) > >>>>> log4j-jdbc-dbcp2 (?) > >>>>> log4j-jpa > >>>>> > >>>>> *logging-log4j2-appender-queue* > >>>>> log4j-jeromq > >>>>> log4j-jms > >>>>> log4j-kafka > >>>>> log4j-redis > >>>>> > >>>>> *logging-log4j2-binding* > >>>>> log4j-jcl > >>>>> log4j-jpl > >>>>> log4j-jul > >>>>> log4j-liquibase > >>>>> log4j-slf4j18-impl > >>>>> log4j-slf4j-impl > >>>>> log4j-to-slf4j > >>>>> > >>>>> *logging-log4j2-container* > >>>>> log4j-docker > >>>>> log4j-kubernetes > >>>>> > >>>>> *logging-log4j2-gui* > >>>>> log4j-jmx-gui > >>>>> > >>>>> *logging-log4j2-jee* > >>>>> log4j-appserver > >>>>> log4j-taglib > >>>>> log4j-web > >>>>> > >>>>> *logging-log4j2-layout-json* > >>>>> log4j-layout-gelf > >>>>> log4j-layout-jackson > >>>>> log4j-layout-jackson-json > >>>>> log4j-layout-jackson-xml > >>>>> log4j-layout-jackson-yaml > >>>>> log4j-layout-template-json > >>>>> > >>>>> *logging-log4j2-osgi* > >>>>> log4j-osgi > >>>>> > >>>>> *logging-log4j2-spring* > >>>>> log4j-spring-boot > >>>>> log4j-spring-cloud-config > >>>>> > >>>>> Note that above breakdown contains every available module in master, > >>>> except > >>>>> log4j-samples, which has the following sub-modules: > >>>>> > >>>>> log4j-samples-loggerProperties (contains 2 very trivial example > >>> classes, > >>>>> log4j-core [tests] contains more comprehensive alternatives) > >>>>> log4j-samples-flume-remote (flume-specific) > >>>>> log4j-samples-flume-embedded (flume-specific) > >>>>> log4j-samples-flume-common (flume-specific) > >>>>> log4j-samples-configuration (contains 3 very trivial example classes, > >>>>> log4j-core [tests] contains more comprehensive alternatives) > >>>>> > >>>>> I propose removing the log4j-samples module and, if necessary, moving > >>>>> Flume-related parts to the log4j-flume-ng. > >>>>> > >>>>> *The problem with multiple repositories* > >>>>> > >>>>> I made this proposal in June 2021. Since then the more my familiarity > >>>> with > >>>>> the website and manual generation increased, the less I became > >>> inclined > >>>> to > >>>>> move forward. I am not much concerned about the increased release > >>> labor > >>>> the > >>>>> Ralph has mentioned – a majority of these repos will hardly get an > >>> update > >>>>> to warrant a release. Though the fact that current website generation > >>>>> depends on the presence of these modules to be in the same repository > >>> and > >>>>> with multiple repositories, the manual and source codes will be in > >>>>> different repositories... These concern me a lot. I think, the steps > >>> to > >>>>> overhaul the project structure should better be in the following > >>> order: > >>>>> > >>>>> 1. Replace `site` phase (`site` phase won't be needed anymore. > >>> Website > >>>>> will independently be generated in a single step via JBake.) > >>>>> 2. Rewrite the manual > >>>>> 3. Migrate to a multi-repo structure (Each repo will have its own > >>>>> website and manual, ala Maven. Here the code in steps 1 and 2 will > >>> be > >>>>> reused.) > >>>>> > >>>>> > >>>>> > >>>>> On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers < > >>> [email protected]> > >>>>> wrote: > >>>>> > >>>>>> I wouldn’t necessarily want to penalize them. At the same time, I > >>> don’t > >>>>>> think we want a ton of extra repos each of which needs its own > >>> release. > >>>>>> > >>>>>> Ralph > >>>>>> > >>>>>>> On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz < > >>>>> [email protected]> > >>>>>> wrote: > >>>>>>> > >>>>>>> Hello, > >>>>>>> > >>>>>>> On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <[email protected]> > >>> wrote: > >>>>>>> > >>>>>>>> +1 with moving Cassandra et al. to their own repo > >>>>>>>> > >>>>>>> > >>>>>>> I agree: Cassandra's tests were one of the most common reasons for > >>>>>> failing > >>>>>>> CI. > >>>>>>> Maybe it could share a repository with other database appenders. > >>>>>>> > >>>>>>> Piotr > >>>>>> > >>>>>> > >>>>> > >>>> > >>> > >> >
