I tried to enable tests to parallelize themselves. While that’s a far more challenging task, I don’t think it should be that hard to figure out how to make the -T option in Maven work properly. Each module could be compiled in parallel while running tests sequentially within each module; that should work without code changes in theory. While that would still leave Core as a long running module, that should also reduce a bit as we start splitting that up further.
— Matt Sicker > On May 1, 2022, at 13:48, Ralph Goers <[email protected]> wrote: > > I believe there is a set of repos that does make sense. Volkan was on the > right > track but got a few things incorrect, which I previously called out. > > Ralph > >> On May 1, 2022, at 1:56 PM, Gary Gregory <[email protected]> wrote: >> >> And yes, one repo per module is another option, a la Maven as you point >> out, but what is being proposed is some halfway solution which to me feels >> confusing and arbitrary just because today this and that module is in favor >> but that some others. >> >> Gaty >> >>> On Sun, May 1, 2022, 07:54 Gary Gregory <[email protected]> wrote: >>> >>> Hi Volkan, >>> >>> I see how you build and how you do version control as orthogonal concerns. >>> Currently we build and deliver everything from the root of the git checkout >>> directory. It does not have to be that way is all I'm saying. One could >>> just cd to a sub directory and work there. You could still have the option >>> to build the world from the root as a convenient way to make sure >>> everything works together and there is no breakage. >>> >>> https://en.m.wikipedia.org/wiki/Monorepo >>> >>> Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9] all >>> employ very large monorepos with varying strategies to scale build systems >>> and version control software with a large volume of code and daily changes. >>> >>> Log4j is tiny in comparison. >>> >>> Gary >>> >>>> On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <[email protected]> wrote: >>> >>>> I disagree that multi-repo setup is a recipe for disaster. It works >>>> perfectly well for Maven. Actually their setup is way more complicated: >>>> dedicated repo for each module! But let's assume you're right and >>>> multi-repos decrease the shelf time. Isn't that something good? I like >>>> repos that rot, it gives a clear message about that project: it simply is >>>> dead. If we migrate to multiple repos and some never get to make a >>>> release, >>>> what does this mean? Apparently nothing interesting is happening here. If >>>> there is a demand from the community to warrant a release, I am sure that >>>> release will come. After all, the very same community attempted to >>>> resurrect Log4j 1. >>>> >>>> I also like single repo setups where build and test times are within >>>> bearable limits, though Log4j is not one of them. Note that multi-repo >>>> discussion started, because we don't know any other way to solve the >>>> current set of problems. If you can think of how to address these concerns >>>> in some other way (e.g., "move folders around"), I am all ears. >>>> >>>> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <[email protected]> >>>> wrote: >>>> >>>>> Breaking our mono repo into a bunch of smaller repo pieces is a recipe >>>> for >>>>> software rot and orphanage IMO. >>>>> >>>>> For example, we have never released out of the tools repo, and I have >>>> folks >>>>> at work at use code from that repo for some internal facing work. Now, I >>>>> could take the time and do a release of course but... priorities. OTOH, >>>> I >>>>> could have gotten a release for "free" if that code was in the mono >>>> repo (I >>>>> am not advocating to put it back). >>>>> >>>>> This email might be better articulated in a call but I am trying to be >>>>> constructive: I think we can keep our mono repo and develop faster by >>>>> reorganizing what we have now. I like having everything in one place >>>>> myself. I am not sure how to actually move folders around to achieve >>>>> whatever new release cycle/split up we want, but I am sure it is do >>>> able. >>>>> >>>>> Gary >>>>> >>>>> On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <[email protected]> wrote: >>>>> >>>>>> I have raised this issue in the past. I even came up with a proposal >>>> on >>>>> the >>>>>> repository structure >>>>>> <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3> >>>> to be >>>>>> followed. I am a big proponent of breaking up the logging-log4j2 >>>>> repository >>>>>> into multiple ones. The current size of the code base is simply >>>>>> incomprehensible and slows down the development cycle tremendously >>>> due to >>>>>> excessive compilation and test times. >>>>>> >>>>>> *The proposal* >>>>>> >>>>>> Let me share the break down I have in mind: >>>>>> >>>>>> *logging-log4j2-api* >>>>>> log4j-1.2-api >>>>>> log4j-api >>>>>> >>>>>> *logging-log4j2* >>>>>> log4j-bom >>>>>> log4j-core >>>>>> log4j-core-its >>>>>> log4j-gctests >>>>>> log4j-iostreams >>>>>> log4j-perf >>>>>> log4j-plugins >>>>>> >>>>>> *logging-log4j2-appender* >>>>>> log4j-cassandra >>>>>> log4j-couchdb >>>>>> log4j-csv >>>>>> log4j-flume-ng >>>>>> log4j-mongodb3 >>>>>> log4j-mongodb4 >>>>>> log4j-smtp >>>>>> >>>>>> *logging-log4j2-appender-jdbc* >>>>>> log4j-jdbc (?) >>>>>> log4j-jdbc-dbcp2 (?) >>>>>> log4j-jpa >>>>>> >>>>>> *logging-log4j2-appender-queue* >>>>>> log4j-jeromq >>>>>> log4j-jms >>>>>> log4j-kafka >>>>>> log4j-redis >>>>>> >>>>>> *logging-log4j2-binding* >>>>>> log4j-jcl >>>>>> log4j-jpl >>>>>> log4j-jul >>>>>> log4j-liquibase >>>>>> log4j-slf4j18-impl >>>>>> log4j-slf4j-impl >>>>>> log4j-to-slf4j >>>>>> >>>>>> *logging-log4j2-container* >>>>>> log4j-docker >>>>>> log4j-kubernetes >>>>>> >>>>>> *logging-log4j2-gui* >>>>>> log4j-jmx-gui >>>>>> >>>>>> *logging-log4j2-jee* >>>>>> log4j-appserver >>>>>> log4j-taglib >>>>>> log4j-web >>>>>> >>>>>> *logging-log4j2-layout-json* >>>>>> log4j-layout-gelf >>>>>> log4j-layout-jackson >>>>>> log4j-layout-jackson-json >>>>>> log4j-layout-jackson-xml >>>>>> log4j-layout-jackson-yaml >>>>>> log4j-layout-template-json >>>>>> >>>>>> *logging-log4j2-osgi* >>>>>> log4j-osgi >>>>>> >>>>>> *logging-log4j2-spring* >>>>>> log4j-spring-boot >>>>>> log4j-spring-cloud-config >>>>>> >>>>>> Note that above breakdown contains every available module in master, >>>>> except >>>>>> log4j-samples, which has the following sub-modules: >>>>>> >>>>>> log4j-samples-loggerProperties (contains 2 very trivial example >>>> classes, >>>>>> log4j-core [tests] contains more comprehensive alternatives) >>>>>> log4j-samples-flume-remote (flume-specific) >>>>>> log4j-samples-flume-embedded (flume-specific) >>>>>> log4j-samples-flume-common (flume-specific) >>>>>> log4j-samples-configuration (contains 3 very trivial example classes, >>>>>> log4j-core [tests] contains more comprehensive alternatives) >>>>>> >>>>>> I propose removing the log4j-samples module and, if necessary, moving >>>>>> Flume-related parts to the log4j-flume-ng. >>>>>> >>>>>> *The problem with multiple repositories* >>>>>> >>>>>> I made this proposal in June 2021. Since then the more my familiarity >>>>> with >>>>>> the website and manual generation increased, the less I became >>>> inclined >>>>> to >>>>>> move forward. I am not much concerned about the increased release >>>> labor >>>>> the >>>>>> Ralph has mentioned – a majority of these repos will hardly get an >>>> update >>>>>> to warrant a release. Though the fact that current website generation >>>>>> depends on the presence of these modules to be in the same repository >>>> and >>>>>> with multiple repositories, the manual and source codes will be in >>>>>> different repositories... These concern me a lot. I think, the steps >>>> to >>>>>> overhaul the project structure should better be in the following >>>> order: >>>>>> >>>>>> 1. Replace `site` phase (`site` phase won't be needed anymore. >>>> Website >>>>>> will independently be generated in a single step via JBake.) >>>>>> 2. Rewrite the manual >>>>>> 3. Migrate to a multi-repo structure (Each repo will have its own >>>>>> website and manual, ala Maven. Here the code in steps 1 and 2 will >>>> be >>>>>> reused.) >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers < >>>> [email protected]> >>>>>> wrote: >>>>>> >>>>>>> I wouldn’t necessarily want to penalize them. At the same time, I >>>> don’t >>>>>>> think we want a ton of extra repos each of which needs its own >>>> release. >>>>>>> >>>>>>> Ralph >>>>>>> >>>>>>>> On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz < >>>>>> [email protected]> >>>>>>> wrote: >>>>>>>> >>>>>>>> Hello, >>>>>>>> >>>>>>>> On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <[email protected]> >>>> wrote: >>>>>>>> >>>>>>>>> +1 with moving Cassandra et al. to their own repo >>>>>>>>> >>>>>>>> >>>>>>>> I agree: Cassandra's tests were one of the most common reasons for >>>>>>> failing >>>>>>>> CI. >>>>>>>> Maybe it could share a repository with other database appenders. >>>>>>>> >>>>>>>> Piotr >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >
