Re: ASF Source Assembly now contains .gitignore and .gitattributes
>“.gitignore” and “.gitattributes” files .gitignore and .gitattributes can easily be helpful during the build, so it does sound strange to exclude them. For instance, Apache Calcite uses .gitattribute to configure that specific files should have an "executable" bit in the resulting tar.gz archive: https://github.com/apache/calcite/blob/e41555f568510df04d3883516b2195393a93426d/.gitattributes#L20-L21 If one deletes the .gitattributes file, the resulting tar.gz will be broken. It does sound strange to me to exclude .gitattributes and .gitignore on the basis of "SCM specific metadata". Vladimir
Re: [DISCUSS] Modello release
>correctly even if the encoding changes, because that will lead to the file >being overwritten. Once again: decoding does **not** guarantee you get **invalid** string when decoding fails. The replacement string might look like a regular string, and it might even collide with the input string. Here's an example: 1. Suppose someone writes out.txt file a Cyrillic letter ф in UTF-8 encoding. jshell> "ф".getBytes("UTF-8") $1 ==> byte[2] { -47, -124 } 2. Suppose there's an encoding UTF-Z that replaces unmappable bytes with ? (ASCII "?") Suppose UTF-Z treats all "negative bytes" as undecodable. 3. Suppose someone attempts to write "??" ( two ASCII question marks) using UTF-Z encoding to the file out.txt The expected file contents should be like 63, 63. "CachingWriter" would try to decode file contents using UTF-Z, and it would end up with "??" (two ASCII question marks because the file contains two negative bytes). 4. The result of decoding would artificially match to the desired string contents (remember, that at step 3 we wanted to store "??"), so the caching writer would **skip** file overwrite even though the file must be updated. Note: the example does not work for UTF-8 because UTF-8 uses a special "illegal" character for unmappable chars. However, there's no guarantee every encoding would use illegal chars for undecodable cases. Vladimir
Re: [DISCUSS] Modello release
>the result string will be corrupted when read The javadoc does not guarantee the string will be "corrupted" :-/ The "default replacement string" is quite a vague definition, so there might be an encoding where "default replacement string" means regular ASCII question mark "?". All in all, String->byte is way easier to reason about. WDYT on having something like that in Maven core? Vladimir
Re: [DISCUSS] Modello release
I believe the added CachingWriter is might become a cause of silent failures. What CachingWriter does is an attempt to read the file and decode it with the provided encoding. Apparently, the decoding might fail since the file might be written in another encoding or it might be corrupted. A better approach would be to convert the created String into `byte[]`, and then compare the bytes with file contents. Then you never really need to decode the file Could someone with commit privileges just fix it? The fix is just to remove readString and writeString methods, and use Arrays.equals to compare the contents. Then, you can compare file contents and avoid reading the file if you know the length differs (it optimizes the "file differs" case). I remember I faced "Maven recompiling everything" issue multiple times, so I wonder if Maven itself has a solution for that. I understand tracking inputs and outputs might take a while to implement, so what if Maven had a solid API for generating files that skips overwriting if the contents are the same? Sample issues on the top of my head: https://issues.apache.org/jira/browse/MRRESOURCES-91 https://github.com/freemarker/fmpp/issues/11 https://github.com/julianhyde/hydromatic-resource/pull/4 Why does every plugin have to reinvent a half-broken caching wheel? Vladimir
Re: Release maven anton plugin...sorry
Enrico>signed by another Apache committer (not strictly from Maven project) AFAIK that is not required: https://infra.apache.org/release-distribution.html#sigs-and-sums The key must be in Maven's KEYS file though. Vladimir
This project has been banned from the build due to previous failures
Hi, Do you think $subj message can be more user-friendly? Can it include the reasons for the failure and steps to resolve the issue rather than merely "This project has been banned"? The message is extremely confusing :(( In case you wonder, here's a sample build log: https://download.copr.fedorainfracloud.org/results/@pgjdbc/pgjdbc-travis/fedora-rawhide-x86_64/01282986-postgresql-jdbc/builder-live.log.gz The build is a Fedora Copr =that calls=> Python =that calls=> Maven, and the wording of "This project has been banned" sounds to me more like "Fedora banned the project" rather than "oh, you are just trying to execute a non-existing goal". PS. Frankly speaking, I have no idea which version of Maven is running, however, as far as I see, master branch contains the same message: https://github.com/apache/maven/blob/f2e9afd788de919646717532d26eca38826e9924/maven-embedder/src/main/java/org/apache/maven/cli/event/ExecutionEventLogger.java#L300 Vladimir
Re: is it legal to shade "gson" packages in Maven?
> Copyright problem in Google sources. Are you going to copy gson sources to maven source code? If you do that, you need to keep the copyright. Are you going to take the jar and optionally shade it? In that case you need to check if gson.jar is accompanied with NOTICE file. If notice is there, you need append it to maven's notice. If notice is not there, updating maven's notice is not needed. The above assumes gson license is Apache-2.0 Vladimir
Re: is it legal to shade "gson" packages in Maven?
>Why not using XML and built in java support? It is no longer built-in: https://www.oracle.com/technetwork/java/javase/11-relnote-issues-5012449.html#JDK-8190378 Vladimir
Re: is it legal to shade "gson" packages in Maven?
> Here the GSON library is only one https://github.com/FasterXML/jackson-jr is 100KiB or so. Have you checked it? Vladimir
Re: Maven Wrapper
Robert>Once the Maven Wrapper has it's final location, one can provide new PRs on our codebase Do you think Wrapper script should verify the integrity of the downloaded file? AFAIK the current implementation just downloads a file and executes it. Executing random files from the internet does not sound like a nice idea. Vladimir
Re: Maven Ant Tasks - cannot connect to Maven Central anymore
Cristi>Is there any place to contribute the change? Here: https://github.com/apache/maven-ant-tasks/blob/ed4d9a3bd1a41613791f2466f19c225898d45519/src/main/java/org/apache/maven/artifact/ant/AbstractArtifactWithRepositoryTask.java#L55 PS. https://issues.apache.org/jira/browse/MANTTASKS-206 seems to specify the way to override the URL. Vladimir
Re: More checkstyle API changes
Robert>last time their argument was that they didn't have enough committers to also maintain a maven plugin TL;DR: I'm neither Maven nor Checkstyle committer, and I'm inclined that the PR should be declined. The case is as follows: there's a method in Checkstyle which is a no-op for 3 years or so. For some reasons (aesthetics?), Checkstyle developers decided to drop the method. They know the method is used by third parties. ^^^ What is the reason to drop a method you know is used by third parties? The consequences would be: 1) Maven committers and PMC would have to release a new plugin version (stage, verify, vote, publish dist) 2) Lots of users will face an error while trying to use never Checkstyle version. For example, In 99% of the cases, I just upgrade Checkstyle version, and I have no clue what is my version of maven-checkstyle-plugin. 3) Of course, their builds will fail. Of course, they will be able to Google the error and they will learn they need to bump maven-checkstyle-plugin. I'm not sure Checkstyle developers understand ASF release process, and I'm not sure they fully understand that "making a release" is not like a "pushing Git tag" :( I'm not sure Checkstyle developers understand they are creating a painful experience for the end-users :( This causes pain for Maven committers, PMCs, and for end-users who will blame all the issues on maven-plugin (because it was "too old, not releasing soon enough, etc, etc"). It does impact Maven in a negative way (not just the plugin), because, you know, at the end of the day it will be like "-- Maven never works -- Realy? What's that? -- Don't remember, there was a problem with one of the Maven core plugins..." What I mean here is like no-one would file a single issue to Checkstyle team for "breaking backward compatibility". I guess it would be better if they will be responsible for the decisions they make. It looks like there are two ways to proceed here: A) Decline the PR, and keep the method at Checkstyle side. Then no changes to the plugin are needed, and everything is good. B) Ensure the plugin version is exactly the same as the main Checsktyle version. Then end-users could specify a single property like "checkstyle.version", and use it for both plugin and checkstyle (well, they won't need to specify checkstyle then), and it would ensure the plugin is always compatible with Checkstyle engine. C) Accept the PR, and make sure Checkstyle will keep the method available for 5 more years (or so) to ensure everybody has migrated to the new plugin version. I guess "B" implies the plugin must be moved under Checkstyle repository so they could perform simultaneous releases (engine + Maven plugin) I do not think "C" is good because, well, I doubt Checkstyle developers are ready to wait 5 more years, and it looks like they are going to drop the method as soon as the new plugin version is released. Robert>To me moving the plugin would help both projects In practice, Checkstyle developers might be not that familiar with "Maven conventions": ways to resolve ClassPath, ways to configure properties, etc. So it makes a certain sense to review the plugin implementation by someone who understands how Maven should feel. However, "ASF release" overheads seem to be very visible (to PMC), especially when the only things to review are like "oh, an empty method is dropped" :( Vladimir
Apache Wagon vs maven-shade vs embedded licenses
Enrico>(I apologize, I don't want to pollute the vote thread, but this is somehow related) I've altered the subject. Enrico> For binary release (that actually is not part of the official VOTE) I'm not a lawyer, but: > http://www.apache.org/legal/release-policy.html#what > WHAT IS A RELEASE? > Releases are, by definition, anything that is published beyond the group that owns it > http://www.apache.org/legal/release-policy.html#what-must-every-release-contain > Every ASF release must comply with ASF licensing policy release-policy.html does not make a distinction between "part of the official vote" and "not a part of the official vote". It just stays "whatever is released must comply with ASF licensing policy". In other words, the VOTE thread looks to me like "we are about to release Apache Maven Wagon, please check the artifacts". -shaded artifact is a part of the release (because it is "anything that is published beyond the group that owns it"), and -shaded does not comply with jsoup's license ==> I suggest that there's an "utmost importance" issue with the artifacts. >I wonder if we could enhance the pom in the future to report machiene >readable statements like 'the artifact will include a binary copy of this >other third party pom' That would be nice. I'm not sure everything comes from a pom though. For instance, -shaded, -sources, -javadoc and other "classifier-based artifacts" miss their respective poms. However, they all might re-distribute different third-party dependencies. Then people do not always consume artifacts as jar/pom files. For instance, apache-maven-3.6.2-bin.zip does not have a pom file. In my opinion, the licensing conditions should be embedded into each archive if that is possible. There's spdx.org effort, however, I don't think it is ready for use. Vladimir
Re: [VOTE] Release Apache Wagon version 3.3.4
> Are you referring to his statement of the MIT license? I am Vladimir
Re: [VOTE] Release Apache Wagon version 3.3.4
> Staging repo: > https://repository.apache.org/content/repositories/maven-1535/ -1 since https://repository.apache.org/content/repositories/maven-1535/org/apache/maven/wagon/wagon-http/3.3.4/wagon-http-3.3.4-shaded.jar violates licensing terms for the third-party code. One of the violations is org.jsoup:jsoup. I know releases may not be vetoed ( https://www.apache.org/foundation/voting.html#ReleaseVotes ) However, there's > http://www.apache.org/legal/release-policy.html#licensing >Every ASF release MUST comply with ASF licensing policy. This requirement is of utmost importance Vladimir
Re: last review of Reproducible Builds proposal
OpenJDK8 uses hashmap for manifest entries, so jar files are not really reproducible there. Note: there's run-to-run randomization in j.u.HashMap, so the manifest contents is not predictable. Vladimir
Re: Next Generation Integration Testing for Plugins/Core
Karl>on the language features but since JDK8 I don't see any advantage What about Kotlin? There are nice things there: the language is statically compiled, great Java interop, there are extension functions, multiline strings, helpful standard library, default parameters. Kotlin is great for creating DSLs. It can be used with JUnit or other frameworks (e.g. https://github.com/kotlintest/kotlintest ) Vladimir
Re: 3.6.3 release?
Mickael>there was no other critical issues pending I guess https://issues.apache.org/jira/browse/MNG-6771 (licensing issues with binary artifacts) is a blocker for the release. Vladimir
Re: last review of Reproducible Builds proposal
>ISO 8601 is neither aware of zones or >DSTs, just abstract offsets which is a good thing Well. ISO 8601 allows timestamps without "UTC offset" For instance, 2019-10-06T12:30:00 is ISO 8601-compatible timestamp, however it is ambiguous. So I suggest that the property must require Z or +HH:MM part. Vladimir
Re: last review of Reproducible Builds proposal
>but >who really looks at the timestamp of entries in release zips/jars/tar.gz >honestly? Tomcat when it decides on what to send in the "Last-Modified" header. For instance, current Gradle does not allow to configure the timestamp, and for reproducible builds it always sets the timestamp to 0 or so. It breaks Tomcat's assumptions: https://github.com/gradle/gradle/issues/10917 Vladimir
Re: Problems with checkstyle tests on Windows (Was [VOTE] Release Apache Maven Checkstyle Plugin version 3.1.0)
Robert> According to the ASF the -sources.zip are the official release, not the git tag That's right Robert>In this case you don't control the EOL, it is based on the OS of the release manager. Is it? I would disagree here. For instance, Apache JMeter produces both -src.zip (CRLF), and -src.tar.gz with LF-defaults for text files. Of course, one might take -src.tar.gz and try building it on Windows, however that is not like RM's machine controls EOL. On top of that, https://reproducible-builds.org/ is a thing, so it makes sense to be able to reproduce release files independent of the release manager (=producezips with the same sha512 on different OS) So both Git and the release script control the way the release is produced. Input data could contain both variations of the EOL styles by the way, which might be handy for EOL-targeted tests. That is Git repo could just contain a file with CRLF, and a file with LF. So would have the release zip. However I would agree some sort of the treatment should be baked into the test. Vladimir
Re: Problems with checkstyle tests on Windows (Was [VOTE] Release Apache Maven Checkstyle Plugin version 3.1.0)
Eric>In that case, we should generate the test files (to Eric>avoid git interfering), one with linux-style EOLs and one with Eric>Windows-style EOLs and test with both. You'd better have those files under Git control, and you could just specify .gitattributes so the LF file is always LF, and CRLF file is always CRLF. That is way simpler than generation of the file(s), and it is way easier to understand by humans Vladimir
Re: Problems with checkstyle tests on Windows (Was [VOTE] Release Apache Maven Checkstyle Plugin version 3.1.0)
Just in case: I see you seem to be battling against CRLF For instance: https://github.com/apache/maven-checkstyle-plugin/pull/16 Did you consider to add the relevant .gitattributes file to ensure Git converts the files automatically? With a proper .gitattributes file, it is just impossible to commit a file with "a wrong EOL". PS. I've came across this thread when Apache JMeter project noticed that Window-based builds were broken by https://issues.apache.org/jira/browse/INFRA-18383 Vladimir
Re: Problems with checkstyle tests on Windows (Was [VOTE] Release Apache Maven Checkstyle Plugin version 3.1.0)
Robert>A clone from Git succeeds, but the sources.zip fails. Robert>The files in the zip are generated on a unix system, so all EOLs in text files are LF Robert>... Robert>The fix: add a setup.groovy to the IT and rewrite the java files with OS specific EOLs Alternative approaches: A) Provide both Linux (LF) and Windows (CRLF) source distributions (e.g. *.zip and *.tgz). B) Specify "lineSeparator" explicitly. Then you could have both CRLF and LF files at the same time and verify if those work C) Generate file at the build stage. If you generate it into target/ directory, then you could generate the file with appropriate for the platform enconding D) Ensure the file is always in LF or CRLF by adding a relevant .gitattributes entry Vladimir
Re: Enable Travis on Maven Scripting Plugin
Enrico> limited to linux and osx, not windows. Windows can be tested via AppVeyor. The configuration is the same: one adds appveyor.yml (e.g. https://github.com/apache/calcite/blob/master/appveyor.yml ) and asks Infra team to enable AppVeyo-GitHub integration. Vladimir - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: New Committers - community
Enrico> For instance maven-surefire integration tests take 2 hours. In general I see Travis (free edition) does not offer many resources. Each Travis job is limited by 50 minutes, however one can split tests into multiple jobs, so it will run fine. Vladimir
Re: New Committers - community
Enrico> Probably Travis won't be able to run our ITs, at least the free version Why do you think so? Vladimir
Re: New Committers - community
Stephen > Are you sure? I've just created a dummy account and "approval" does not make merge button magically appear. If you have a couple of minutes, please make a PR (I'll approve it shortly) to https://github.com/vlsi/KE-complex_modifications Note: you can click "edit" button in GitHub UI for a random file, so you don't really need to clone the project. I've attached no_merge.png which is a screenshot of GitHub UI for PR author. Apparently, there's no merge button. Vladimir - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: New Committers - community
Karl Heinz Marbaise > The user community is very big but unfortunately the people who are Karl Heinz Marbaise > willing to help is not that big... Can I (ab)use the thread a bit? One of the issues I run into with Maven is "it takes ages for repeated build when nothing has changed". Of course there are details, yet the final perception is like that. I know Apache JMeter team sticks with Ant for a mere reason of fast builds (even though it is way harder to configure in IDE) Once upon a time I was bitten hard enough by Maven which always rebuilds everything even for repeated builds, so I investigated the reason and created https://issues.apache.org/jira/browse/CALCITE-484 The basic case is "jar is rebuilt due to resource was somehow modified". Of course there are multiple offenders, however two cases come from maven-resources-plugin and maven-remote-resources-plugin. I've created a PR (in December 2014) to fix maven-remote-resources-plugin: https://github.com/apache/maven-plugins/pull/40 (Avoid overwrite of the destination file if the produced contents is the same), and it took just FOUR years to release the new version of the plugin in October 2018. The second offender is maven-filtering ( https://issues.apache.org/jira/browse/MSHARED-394 ) In a brief, maven-filtering rewrites the file not matter the contents. For instance, when Maven copies resources, it compares timestamps, so it can skip copy if the file is good enough. However maven-filtering always writes the destination file, thus it triggers jar rebuild, etc. Frankly speaking, I'm proud of a trick in https://github.com/apache/maven-plugins/pull/40 which keeps contents in memory, and if the buffer overflows, then it writes the contents to the destination file. In other words, no temporary files are required, and it can skip file overwrites when the produced contents is the same. I think similar should be done to DefaultMavenFileFilter, however Kristian suggested he wanted to look on his own: https://issues.apache.org/jira/browse/MSHARED-394?focusedCommentId=14449864=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14449864 >Kristian Rosenvold added a comment - 16/Dec/2014 12:19 >Haven given this issue some thought I believe I have come up with a radically better solution than what is suggested here. >There is no point in working on the changes suggested here, >at least not until I give up my other plan. >Since I am currently working on a slightly different task, it will be a couple of weeks before I'll make any activity on this issue. Frankly speaking, I'm very puzzled. The net result is: my first Maven contribution (MRRESOURCES-91) took 4 years to release, and very similar one was rejected with "There is no point in working on the changes suggested here" comment. That is quite sad to be honest. Vladimir
Re: New Committers - community
> And, again, TravisCI also does the same in a decent way, with the benefit > of worrying less about underlying infra and security, and only drawback of > being discoupled from a specific infra (is it really a drawback?) Just in case: Apache Calcite uses both Travis CI and AppVeryor. It just works. It does simplify validation of the changes across various Java versions, javadoc, RAT. AppVeyor validates Windows builds. Both results are automatically displayed at GitHub PR (e.g. https://github.com/apache/calcite/pull/976) with no extra manual actions. Vladimir
Re: New Committers - community
Stephen> Well on other GitHub orgs i’ve seen PR author has Merge button once PR is Stephen> approved by someone with push rights to the repo... until they add a commit Stephen> or the merge result changes It does not work the way you describe. 1) By default GitHub defaults to "Base permissions" == "read". Administrator can set even "base permissions == admin", however that is not something that comes by default. 2) Repository administrator can configure a branch so it requires a PR approval before merge (see https://help.github.com/articles/enabling-required-reviews-for-pull-requests/ ) It might happen so that "author has Merge button" is a case when repository was configured with "default=write" permissions + "require PR approval". Indeed that should produce (I have not checked) the behavior you describe, however it was never a default mode for GiHub. Vladimir - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
Re: New Committers - community
Stephen> Nah. Once committers have approved the PR then the PR can be merged by the Stephen> creator of the PR even if not a committer... at least that’s the default Stephen> way GitHub PRs work By default one needs push rights on the repository to merge PRs. "Approval" changes nothing. "PR approval" does not differ from any other comment. In other words, non-committers can't merge PRs. Vladimir - To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org For additional commands, e-mail: dev-h...@maven.apache.org
MRRESOURCES-91 release request for maven-remote-resources-plugin
Hi, Would anyone be so kind to release maven-remote-resources-plugin ? My patch was merged more than 3 years ago, and the official release is still missing. Here's the JIRA link: https://issues.apache.org/jira/browse/MRRESOURCES-91 The problem is org.apache:apache pom uses maven-remote-resources-plugin to create DEPENDENCIES file, and it causes re-build of the jar, sources-jar every time. Vladimir