I really don’t see how this helps. We aren’t going to release subcomponents of a Git repo. I wouldn’t even begin to know how to do that at the ASF.
The problems I have are all related to how long it takes to prepare and perform a release. I can definitely say that I have encountered many times where builds haven’t worked because people are only building the things they changed without realizing they were impacting other components, and they do this because it is simply too time consuming to run a full build. While separating the components won’t fix this, it will make it so that what is being released has been tested. So, for example, if a change in core breaks something in an Appender in another module that will be fixed in that component before its next release. The main reason we don’t do frequent releases is because I need to find 2 or 3 free days to be able to do it. This is largely because I go through the issues and look for stuff we broke in previous releases - we have 3 or 4 of those right now, and I build the web site usually 4 or 5 times because the site build always has problems and no one ever runs it except when I do it during a release. So for me, the two key items are: 1. Stop using Maven site and build the web site in logging-log4j2-site using a much simplified process. 2. Break up the build into smaller pieces so that full development builds can be done on committers laptops in no more than 5 minutes and that the release build doesn’t take much longer than that. Although we discuss parallel builds, to date every attempt to do that has failed. The effort to go through all the tests and verify that they have no side effects that could impact other tests seems to be more than any of us can do. If we can’t even get log4j-api to build without problems I don’t see how there is any hope for log4j-core to do that. Ralph > On May 1, 2022, at 1:54 PM, Gary Gregory <[email protected]> wrote: > > Hi Volkan, > > I see how you build and how you do version control as orthogonal concerns. > Currently we build and deliver everything from the root of the git checkout > directory. It does not have to be that way is all I'm saying. One could > just cd to a sub directory and work there. You could still have the option > to build the world from the root as a convenient way to make sure > everything works together and there is no breakage. > > https://en.m.wikipedia.org/wiki/Monorepo > > Google,[5] Facebook,[6] Microsoft,[7] Uber,[8] Airbnb, and Twitter[9] all > employ very large monorepos with varying strategies to scale build systems > and version control software with a large volume of code and daily changes. > > Log4j is tiny in comparison. > > Gary > > On Sat, Apr 30, 2022, 16:20 Volkan Yazıcı <[email protected]> wrote: > >> I disagree that multi-repo setup is a recipe for disaster. It works >> perfectly well for Maven. Actually their setup is way more complicated: >> dedicated repo for each module! But let's assume you're right and >> multi-repos decrease the shelf time. Isn't that something good? I like >> repos that rot, it gives a clear message about that project: it simply is >> dead. If we migrate to multiple repos and some never get to make a release, >> what does this mean? Apparently nothing interesting is happening here. If >> there is a demand from the community to warrant a release, I am sure that >> release will come. After all, the very same community attempted to >> resurrect Log4j 1. >> >> I also like single repo setups where build and test times are within >> bearable limits, though Log4j is not one of them. Note that multi-repo >> discussion started, because we don't know any other way to solve the >> current set of problems. If you can think of how to address these concerns >> in some other way (e.g., "move folders around"), I am all ears. >> >> On Sat, Apr 30, 2022 at 2:54 PM Gary Gregory <[email protected]> >> wrote: >> >>> Breaking our mono repo into a bunch of smaller repo pieces is a recipe >> for >>> software rot and orphanage IMO. >>> >>> For example, we have never released out of the tools repo, and I have >> folks >>> at work at use code from that repo for some internal facing work. Now, I >>> could take the time and do a release of course but... priorities. OTOH, I >>> could have gotten a release for "free" if that code was in the mono repo >> (I >>> am not advocating to put it back). >>> >>> This email might be better articulated in a call but I am trying to be >>> constructive: I think we can keep our mono repo and develop faster by >>> reorganizing what we have now. I like having everything in one place >>> myself. I am not sure how to actually move folders around to achieve >>> whatever new release cycle/split up we want, but I am sure it is do able. >>> >>> Gary >>> >>> On Fri, Apr 29, 2022, 16:03 Volkan Yazıcı <[email protected]> wrote: >>> >>>> I have raised this issue in the past. I even came up with a proposal on >>> the >>>> repository structure >>>> <https://lists.apache.org/thread/sdv92pw6m9bst8zzn88jyln8z8ozshw3> to >> be >>>> followed. I am a big proponent of breaking up the logging-log4j2 >>> repository >>>> into multiple ones. The current size of the code base is simply >>>> incomprehensible and slows down the development cycle tremendously due >> to >>>> excessive compilation and test times. >>>> >>>> *The proposal* >>>> >>>> Let me share the break down I have in mind: >>>> >>>> *logging-log4j2-api* >>>> log4j-1.2-api >>>> log4j-api >>>> >>>> *logging-log4j2* >>>> log4j-bom >>>> log4j-core >>>> log4j-core-its >>>> log4j-gctests >>>> log4j-iostreams >>>> log4j-perf >>>> log4j-plugins >>>> >>>> *logging-log4j2-appender* >>>> log4j-cassandra >>>> log4j-couchdb >>>> log4j-csv >>>> log4j-flume-ng >>>> log4j-mongodb3 >>>> log4j-mongodb4 >>>> log4j-smtp >>>> >>>> *logging-log4j2-appender-jdbc* >>>> log4j-jdbc (?) >>>> log4j-jdbc-dbcp2 (?) >>>> log4j-jpa >>>> >>>> *logging-log4j2-appender-queue* >>>> log4j-jeromq >>>> log4j-jms >>>> log4j-kafka >>>> log4j-redis >>>> >>>> *logging-log4j2-binding* >>>> log4j-jcl >>>> log4j-jpl >>>> log4j-jul >>>> log4j-liquibase >>>> log4j-slf4j18-impl >>>> log4j-slf4j-impl >>>> log4j-to-slf4j >>>> >>>> *logging-log4j2-container* >>>> log4j-docker >>>> log4j-kubernetes >>>> >>>> *logging-log4j2-gui* >>>> log4j-jmx-gui >>>> >>>> *logging-log4j2-jee* >>>> log4j-appserver >>>> log4j-taglib >>>> log4j-web >>>> >>>> *logging-log4j2-layout-json* >>>> log4j-layout-gelf >>>> log4j-layout-jackson >>>> log4j-layout-jackson-json >>>> log4j-layout-jackson-xml >>>> log4j-layout-jackson-yaml >>>> log4j-layout-template-json >>>> >>>> *logging-log4j2-osgi* >>>> log4j-osgi >>>> >>>> *logging-log4j2-spring* >>>> log4j-spring-boot >>>> log4j-spring-cloud-config >>>> >>>> Note that above breakdown contains every available module in master, >>> except >>>> log4j-samples, which has the following sub-modules: >>>> >>>> log4j-samples-loggerProperties (contains 2 very trivial example >> classes, >>>> log4j-core [tests] contains more comprehensive alternatives) >>>> log4j-samples-flume-remote (flume-specific) >>>> log4j-samples-flume-embedded (flume-specific) >>>> log4j-samples-flume-common (flume-specific) >>>> log4j-samples-configuration (contains 3 very trivial example classes, >>>> log4j-core [tests] contains more comprehensive alternatives) >>>> >>>> I propose removing the log4j-samples module and, if necessary, moving >>>> Flume-related parts to the log4j-flume-ng. >>>> >>>> *The problem with multiple repositories* >>>> >>>> I made this proposal in June 2021. Since then the more my familiarity >>> with >>>> the website and manual generation increased, the less I became inclined >>> to >>>> move forward. I am not much concerned about the increased release labor >>> the >>>> Ralph has mentioned – a majority of these repos will hardly get an >> update >>>> to warrant a release. Though the fact that current website generation >>>> depends on the presence of these modules to be in the same repository >> and >>>> with multiple repositories, the manual and source codes will be in >>>> different repositories... These concern me a lot. I think, the steps to >>>> overhaul the project structure should better be in the following order: >>>> >>>> 1. Replace `site` phase (`site` phase won't be needed anymore. >> Website >>>> will independently be generated in a single step via JBake.) >>>> 2. Rewrite the manual >>>> 3. Migrate to a multi-repo structure (Each repo will have its own >>>> website and manual, ala Maven. Here the code in steps 1 and 2 will >> be >>>> reused.) >>>> >>>> >>>> >>>> On Wed, Apr 27, 2022 at 1:35 PM Ralph Goers < >> [email protected]> >>>> wrote: >>>> >>>>> I wouldn’t necessarily want to penalize them. At the same time, I >> don’t >>>>> think we want a ton of extra repos each of which needs its own >> release. >>>>> >>>>> Ralph >>>>> >>>>>> On Apr 27, 2022, at 12:26 PM, Piotr P. Karwasz < >>>> [email protected]> >>>>> wrote: >>>>>> >>>>>> Hello, >>>>>> >>>>>> On Wed, 27 Apr 2022 at 11:33, Volkan Yazıcı <[email protected]> >> wrote: >>>>>> >>>>>>> +1 with moving Cassandra et al. to their own repo >>>>>>> >>>>>> >>>>>> I agree: Cassandra's tests were one of the most common reasons for >>>>> failing >>>>>> CI. >>>>>> Maybe it could share a repository with other database appenders. >>>>>> >>>>>> Piotr >>>>> >>>>> >>>> >>> >>
