Re: ASF Source Assembly now contains .gitignore and .gitattributes

2022-09-16 Thread Vladimir Sitnikov
>“.gitignore” and “.gitattributes” files

.gitignore and .gitattributes can easily be helpful during the build, so it
does sound strange to exclude them.

For instance, Apache Calcite uses .gitattribute to configure that specific
files should have an "executable" bit in the resulting tar.gz archive:
https://github.com/apache/calcite/blob/e41555f568510df04d3883516b2195393a93426d/.gitattributes#L20-L21
If one deletes the .gitattributes file, the resulting tar.gz will be broken.

It does sound strange to me to exclude .gitattributes and .gitignore on the
basis of "SCM specific metadata".

Vladimir


Re: [DISCUSS] Modello release

2022-02-15 Thread Vladimir Sitnikov
>correctly even if the encoding changes, because that will lead to the file
>being overwritten.

Once again: decoding does **not** guarantee you get **invalid** string when
decoding fails.
The replacement string might look like a regular string, and it might even
collide with the input string.

Here's an example:
1. Suppose someone writes out.txt file a Cyrillic letter ф in UTF-8
encoding.
jshell> "ф".getBytes("UTF-8")
$1 ==> byte[2] { -47, -124 }

2. Suppose there's an encoding UTF-Z that replaces unmappable bytes with ?
(ASCII "?")
Suppose UTF-Z treats all "negative bytes" as undecodable.

3. Suppose someone attempts to write "??" ( two ASCII question marks) using
UTF-Z encoding to the file out.txt
The expected file contents should be like 63, 63.
"CachingWriter" would try to decode file contents using UTF-Z, and it would
end up with "??" (two ASCII question marks because the file contains two
negative bytes).

4. The result of decoding would artificially match to the desired string
contents (remember, that at step 3 we wanted to store "??"),
so the caching writer would **skip** file overwrite even though the file
must be updated.

Note: the example does not work for UTF-8 because UTF-8 uses a special
"illegal" character for unmappable chars.
However, there's no guarantee every encoding would use illegal chars for
undecodable cases.

Vladimir


Re: [DISCUSS] Modello release

2022-02-14 Thread Vladimir Sitnikov
>the result string will be corrupted when read

The javadoc does not guarantee the string will be "corrupted" :-/

The "default replacement string" is quite a vague definition,
so there might be an encoding where "default replacement string"
means regular ASCII question mark "?".

All in all, String->byte is way easier to reason about.

WDYT on having something like that in Maven core?

Vladimir


Re: [DISCUSS] Modello release

2022-02-13 Thread Vladimir Sitnikov
I believe the added CachingWriter is might become a cause of silent
failures.

What CachingWriter does is an attempt to read the file and decode it with
the provided encoding.
Apparently, the decoding might fail since the file might be written in
another encoding or it might be corrupted.

A better approach would be to convert the created String into `byte[]`, and
then compare the bytes with file contents.
Then you never really need to decode the file

Could someone with commit privileges just fix it?
The fix is just to remove readString and writeString methods, and use
Arrays.equals to compare the contents.
Then, you can compare file contents and avoid reading the file if you know
the length differs (it optimizes the "file differs" case).



I remember I faced "Maven recompiling everything" issue multiple times, so
I wonder if Maven itself has a solution for that.
I understand tracking inputs and outputs might take a while to implement, so
what if Maven had a solid API for generating files that skips overwriting
if the contents are the same?

Sample issues on the top of my head:
https://issues.apache.org/jira/browse/MRRESOURCES-91
https://github.com/freemarker/fmpp/issues/11
https://github.com/julianhyde/hydromatic-resource/pull/4

Why does every plugin have to reinvent a half-broken caching wheel?

Vladimir


Re: Release maven anton plugin...sorry

2020-04-05 Thread Vladimir Sitnikov
Enrico>signed by another Apache committer (not strictly from Maven project)

AFAIK that is not required:
https://infra.apache.org/release-distribution.html#sigs-and-sums
The key must be in Maven's KEYS file though.

Vladimir


This project has been banned from the build due to previous failures

2020-03-04 Thread Vladimir Sitnikov
Hi,

Do you think $subj message can be more user-friendly?
Can it include the reasons for the failure and steps to resolve the issue
rather than merely "This project has been banned"?

The message is extremely confusing :((

In case you wonder, here's a sample build log:
https://download.copr.fedorainfracloud.org/results/@pgjdbc/pgjdbc-travis/fedora-rawhide-x86_64/01282986-postgresql-jdbc/builder-live.log.gz

The build is a Fedora Copr =that calls=> Python =that calls=> Maven, and
the wording of "This project has been banned" sounds to me more like
"Fedora banned the project" rather than "oh, you are just trying to execute
a non-existing goal".


PS. Frankly speaking, I have no idea which version of Maven is running,
however, as far as I see, master branch contains the same message:
https://github.com/apache/maven/blob/f2e9afd788de919646717532d26eca38826e9924/maven-embedder/src/main/java/org/apache/maven/cli/event/ExecutionEventLogger.java#L300

Vladimir


Re: is it legal to shade "gson" packages in Maven?

2020-02-23 Thread Vladimir Sitnikov
> Copyright problem in Google sources.

Are you going to copy gson sources to maven source code?
If you do that, you need to keep the copyright.

Are you going to take the jar and optionally shade it?
In that case you need to check if gson.jar is accompanied with NOTICE file.
If notice is there, you need append it to maven's notice.
If notice is not there, updating maven's notice is not needed.

The above assumes gson license is Apache-2.0

Vladimir


Re: is it legal to shade "gson" packages in Maven?

2020-02-23 Thread Vladimir Sitnikov
>Why not using XML and built in java support?

It is no longer built-in:
https://www.oracle.com/technetwork/java/javase/11-relnote-issues-5012449.html#JDK-8190378

Vladimir


Re: is it legal to shade "gson" packages in Maven?

2020-02-23 Thread Vladimir Sitnikov
> Here the GSON library is only one

https://github.com/FasterXML/jackson-jr is 100KiB or so.
Have you checked it?

Vladimir


Re: Maven Wrapper

2020-02-18 Thread Vladimir Sitnikov
Robert>Once the Maven Wrapper has it's final location, one can provide new
PRs on our codebase

Do you think Wrapper script should verify the integrity of the downloaded
file?
AFAIK the current implementation just downloads a file and executes it.
Executing random files from the internet does not sound like a nice idea.

Vladimir


Re: Maven Ant Tasks - cannot connect to Maven Central anymore

2020-01-16 Thread Vladimir Sitnikov
Cristi>Is there any place to contribute the change?

Here:
https://github.com/apache/maven-ant-tasks/blob/ed4d9a3bd1a41613791f2466f19c225898d45519/src/main/java/org/apache/maven/artifact/ant/AbstractArtifactWithRepositoryTask.java#L55

PS. https://issues.apache.org/jira/browse/MANTTASKS-206 seems to specify
the way to override the URL.

Vladimir


Re: More checkstyle API changes

2019-12-23 Thread Vladimir Sitnikov
Robert>last time their argument was that they didn't have enough committers
to also maintain a maven plugin

TL;DR: I'm neither Maven nor Checkstyle committer, and I'm inclined that
the PR should be declined.

The case is as follows: there's a method in Checkstyle which is a no-op for
3 years or so.
For some reasons (aesthetics?), Checkstyle developers decided to drop the
method.
They know the method is used by third parties.

^^^ What is the reason to drop a method you know is used by third parties?

The consequences would be:
1) Maven committers and PMC would have to release a new plugin version
(stage, verify, vote, publish dist)
2) Lots of users will face an error while trying to use never Checkstyle
version.
For example, In 99% of the cases, I just upgrade Checkstyle version, and I
have no clue what is my version of maven-checkstyle-plugin.
3) Of course, their builds will fail. Of course, they will be able to
Google the error and they will learn they need to bump
maven-checkstyle-plugin.

I'm not sure Checkstyle developers understand ASF release process, and I'm
not sure they fully understand that "making a release" is not like a
"pushing Git tag" :(
I'm not sure Checkstyle developers understand they are creating a painful
experience for the end-users :(

This causes pain for Maven committers, PMCs, and for end-users who will
blame all the issues on maven-plugin (because it was "too old, not
releasing soon enough, etc, etc").

It does impact Maven in a negative way (not just the plugin), because, you
know, at the end of the day it will be like "-- Maven never works -- Realy?
What's that? -- Don't remember, there was a problem with one of the Maven
core plugins..."
What I mean here is like no-one would file a single issue to Checkstyle
team for "breaking backward compatibility".
I guess it would be better if they will be responsible for the decisions
they make.

It looks like there are two ways to proceed here:
A) Decline the PR, and keep the method at Checkstyle side. Then no changes
to the plugin are needed, and everything is good.
B) Ensure the plugin version is exactly the same as the main Checsktyle
version. Then end-users could specify a single property like
"checkstyle.version", and use it for both plugin and checkstyle (well, they
won't need to specify checkstyle then), and it would ensure the plugin is
always compatible with Checkstyle engine.
C) Accept the PR, and make sure Checkstyle will keep the method available
for 5 more years (or so) to ensure everybody has migrated to the new plugin
version.

I guess "B" implies the plugin must be moved under Checkstyle repository so
they could perform simultaneous releases (engine + Maven plugin)
I do not think "C" is good because, well, I doubt Checkstyle developers are
ready to wait 5 more years, and it looks like they are going to drop the
method as soon as the new plugin version is released.

Robert>To me moving the plugin would help both projects

In practice, Checkstyle developers might be not that familiar with "Maven
conventions": ways to resolve ClassPath, ways to configure properties, etc.
So it makes a certain sense to review the plugin implementation by someone
who understands how Maven should feel.

However, "ASF release" overheads seem to be very visible (to PMC),
especially when the only things to review are like "oh, an empty method is
dropped" :(

Vladimir


Apache Wagon vs maven-shade vs embedded licenses

2019-11-06 Thread Vladimir Sitnikov
Enrico>(I apologize, I don't want to pollute the vote thread, but this is
somehow
related)

I've altered the subject.

Enrico> For binary release (that actually is not part of the official VOTE)

I'm not a lawyer, but:

> http://www.apache.org/legal/release-policy.html#what
> WHAT IS A RELEASE?
> Releases are, by definition, anything that is published beyond the group
that owns it

>
http://www.apache.org/legal/release-policy.html#what-must-every-release-contain
> Every ASF release must comply with ASF licensing policy

release-policy.html does not make a distinction between "part of the
official vote" and "not a part of the official vote".
It just stays "whatever is released must comply with ASF licensing policy".

In other words, the VOTE thread looks to me like "we are about to release
Apache Maven Wagon, please check the artifacts".
-shaded artifact is a part of the release (because it is "anything that is
published beyond the group that owns it"),
and -shaded does not comply with jsoup's license ==> I suggest that there's
an "utmost importance" issue with the artifacts.

>I wonder if we could enhance the pom in the future to report machiene
>readable statements like 'the artifact will include a binary copy of this
>other third party pom'

That would be nice. I'm not sure everything comes from a pom though.
For instance, -shaded, -sources, -javadoc and other "classifier-based
artifacts" miss their respective poms.
However, they all might re-distribute different third-party dependencies.

Then people do not always consume artifacts as jar/pom files.
For instance, apache-maven-3.6.2-bin.zip does not have a pom file.

In my opinion, the licensing conditions should be embedded into each
archive if that is possible.

There's spdx.org effort, however, I don't think it is ready for use.

Vladimir


Re: [VOTE] Release Apache Wagon version 3.3.4

2019-11-05 Thread Vladimir Sitnikov
> Are you referring to his statement of the MIT license?

I am

Vladimir


Re: [VOTE] Release Apache Wagon version 3.3.4

2019-11-05 Thread Vladimir Sitnikov
> Staging repo:
> https://repository.apache.org/content/repositories/maven-1535/

-1 since
https://repository.apache.org/content/repositories/maven-1535/org/apache/maven/wagon/wagon-http/3.3.4/wagon-http-3.3.4-shaded.jar
violates licensing terms for the third-party code.
One of the violations is org.jsoup:jsoup.

I know releases may not be vetoed (
https://www.apache.org/foundation/voting.html#ReleaseVotes )
However, there's

> http://www.apache.org/legal/release-policy.html#licensing
>Every ASF release MUST comply with ASF licensing policy. This requirement
is of utmost importance

Vladimir


Re: last review of Reproducible Builds proposal

2019-11-01 Thread Vladimir Sitnikov
OpenJDK8 uses hashmap for manifest entries, so jar files are not really
reproducible there.

Note: there's run-to-run randomization in j.u.HashMap, so the manifest
contents is not predictable.

Vladimir


Re: Next Generation Integration Testing for Plugins/Core

2019-10-31 Thread Vladimir Sitnikov
Karl>on the language features but since JDK8 I don't see any advantage

What about Kotlin?

There are nice things there: the language is statically compiled, great
Java interop, there are extension functions, multiline strings, helpful
standard library, default parameters.
Kotlin is great for creating DSLs.

It can be used with JUnit or other frameworks (e.g.
https://github.com/kotlintest/kotlintest )

Vladimir


Re: 3.6.3 release?

2019-10-24 Thread Vladimir Sitnikov
Mickael>there was no other critical issues pending

I guess https://issues.apache.org/jira/browse/MNG-6771 (licensing issues
with binary artifacts) is a blocker for the release.

Vladimir


Re: last review of Reproducible Builds proposal

2019-10-06 Thread Vladimir Sitnikov
>ISO 8601 is neither aware of zones or
>DSTs, just abstract offsets which is a good thing

Well. ISO 8601 allows timestamps without "UTC offset"
For instance, 2019-10-06T12:30:00 is ISO 8601-compatible timestamp, however
it is ambiguous.

So I suggest that the property must require Z or +HH:MM part.

Vladimir


Re: last review of Reproducible Builds proposal

2019-10-05 Thread Vladimir Sitnikov
>but
>who really looks at the timestamp of entries in release zips/jars/tar.gz
>honestly?

Tomcat when it decides on what to send in the "Last-Modified" header.
For instance, current Gradle does not allow to configure the timestamp, and
for reproducible builds it always sets the timestamp to 0 or so.
It breaks Tomcat's assumptions:
https://github.com/gradle/gradle/issues/10917

Vladimir


Re: Problems with checkstyle tests on Windows (Was [VOTE] Release Apache Maven Checkstyle Plugin version 3.1.0)

2019-07-22 Thread Vladimir Sitnikov
Robert> According to the ASF the -sources.zip are the official release, not
the git tag

That's right

Robert>In this case you don't control the EOL, it is based on the OS of the
release manager.

Is it? I would disagree here.

For instance, Apache JMeter produces both -src.zip (CRLF), and -src.tar.gz
with LF-defaults for text files.
Of course, one might take -src.tar.gz and try building it on Windows,
however that is not like RM's machine controls EOL.

On top of that, https://reproducible-builds.org/ is a thing, so it makes
sense to be able to reproduce release files independent of the release
manager (=producezips with the same sha512 on different OS)

So both Git and the release script control the way the release is produced.

Input data could contain both variations of the EOL styles by the way,
which might be handy for EOL-targeted tests. That is Git repo could just
contain a file with CRLF, and a file with LF. So would have the release
zip.

However I would agree some sort of the treatment should be baked into the
test.

Vladimir


Re: Problems with checkstyle tests on Windows (Was [VOTE] Release Apache Maven Checkstyle Plugin version 3.1.0)

2019-07-18 Thread Vladimir Sitnikov
Eric>In that case, we should generate the test files (to
Eric>avoid git interfering), one with linux-style EOLs and one with
Eric>Windows-style EOLs and test with both.

You'd better have those files under Git control, and you could just specify
.gitattributes so the LF file is always LF, and CRLF file is always CRLF.

That is way simpler than generation of the file(s), and it is way easier to
understand by humans

Vladimir


Re: Problems with checkstyle tests on Windows (Was [VOTE] Release Apache Maven Checkstyle Plugin version 3.1.0)

2019-07-17 Thread Vladimir Sitnikov
Just in case: I see you seem to be battling against CRLF
For instance: https://github.com/apache/maven-checkstyle-plugin/pull/16
Did you consider to add the relevant .gitattributes file to ensure Git
converts the files automatically?
With a proper .gitattributes file, it is just impossible to commit a file
with "a wrong EOL".

PS. I've came across this thread when Apache JMeter project noticed that
Window-based builds were broken by
https://issues.apache.org/jira/browse/INFRA-18383

Vladimir


Re: Problems with checkstyle tests on Windows (Was [VOTE] Release Apache Maven Checkstyle Plugin version 3.1.0)

2019-07-17 Thread Vladimir Sitnikov
Robert>A clone from Git succeeds, but the sources.zip fails.
Robert>The files in the zip are generated on a unix system, so all EOLs in
text files are LF
Robert>...
Robert>The fix: add a setup.groovy to the IT and rewrite the java files
with OS specific EOLs

Alternative approaches:
A) Provide both Linux (LF) and Windows (CRLF) source distributions (e.g.
*.zip and *.tgz).
B) Specify "lineSeparator" explicitly. Then you could have both CRLF and LF
files at the same time and verify if those work
C) Generate file at the build stage. If you generate it into target/
directory, then you could generate the file with appropriate for the
platform enconding
D) Ensure the file is always in LF or CRLF by adding a relevant
.gitattributes entry

Vladimir


Re: Enable Travis on Maven Scripting Plugin

2019-01-06 Thread Vladimir Sitnikov
Enrico> limited to linux and osx, not windows.

Windows can be tested via AppVeyor.
The configuration is the same: one adds appveyor.yml (e.g.
https://github.com/apache/calcite/blob/master/appveyor.yml ) and asks
Infra team to enable AppVeyo-GitHub integration.

Vladimir

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



Re: New Committers - community

2019-01-02 Thread Vladimir Sitnikov
Enrico> For instance maven-surefire
integration tests take 2 hours.
In general I see Travis (free edition) does not offer many resources.

Each Travis job is limited by 50 minutes, however one can split tests into
multiple jobs, so it will run fine.

Vladimir


Re: New Committers - community

2019-01-02 Thread Vladimir Sitnikov
Enrico> Probably Travis won't be able to run our ITs, at least the free
version

Why do you think so?

Vladimir


Re: New Committers - community

2018-12-29 Thread Vladimir Sitnikov
Stephen > Are you sure?

I've just created a dummy account and "approval" does not make merge button
magically appear.

If you have a couple of minutes, please make a PR (I'll approve it shortly)
to https://github.com/vlsi/KE-complex_modifications
Note: you can click "edit" button in GitHub UI for a random file, so you
don't really need to clone the project.

I've attached no_merge.png which is a screenshot of GitHub UI for PR
author. Apparently, there's no merge button.

Vladimir

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org

Re: New Committers - community

2018-12-29 Thread Vladimir Sitnikov
Karl Heinz Marbaise > The user community is very big but unfortunately the
people who are
Karl Heinz Marbaise > willing to help is not that big...

Can I (ab)use the thread a bit?

One of the issues I run into with Maven is "it takes ages for repeated
build when nothing has changed".
Of course there are details, yet the final perception is like that.

I know Apache JMeter team sticks with Ant for a mere reason of fast builds
(even though it is way harder to configure in IDE)

Once upon a time I was bitten hard enough by Maven which always rebuilds
everything even for repeated builds, so I investigated the reason and
created https://issues.apache.org/jira/browse/CALCITE-484
The basic case is "jar is rebuilt due to resource was somehow modified".
Of course there are multiple offenders, however two cases come from
maven-resources-plugin and maven-remote-resources-plugin.

I've created a PR (in December 2014) to fix maven-remote-resources-plugin:
https://github.com/apache/maven-plugins/pull/40 (Avoid overwrite of the
destination file if the produced contents is the same), and it took just
FOUR years to release the new version of the plugin in October 2018.

The second offender is maven-filtering (
https://issues.apache.org/jira/browse/MSHARED-394 )
In a brief, maven-filtering rewrites the file not matter the contents. For
instance, when Maven copies resources, it compares timestamps, so it can
skip copy if the file is good enough.
However maven-filtering always writes the destination file, thus it
triggers jar rebuild, etc.

Frankly speaking, I'm proud of a trick in
https://github.com/apache/maven-plugins/pull/40 which keeps contents in
memory, and if the buffer overflows, then it writes the contents to the
destination file. In other words, no temporary files are required, and it
can skip file overwrites when the produced contents is the same.

I think similar should be done to DefaultMavenFileFilter, however Kristian
suggested he wanted to look on his own:

https://issues.apache.org/jira/browse/MSHARED-394?focusedCommentId=14449864=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14449864

>Kristian Rosenvold added a comment - 16/Dec/2014 12:19
>Haven given this issue some thought I believe I have come up with a
radically better solution than what is suggested here.
>There is no point in working on the changes suggested here,
>at least not until I give up my other plan.
>Since I am currently working on a slightly different task, it will be a
couple of weeks before I'll make any activity on this issue.


Frankly speaking, I'm very puzzled.
The net result is: my first Maven contribution (MRRESOURCES-91) took 4
years to release, and very similar one was rejected with "There is no point
in working on the changes suggested here" comment.
That is quite sad to be honest.

Vladimir


Re: New Committers - community

2018-12-29 Thread Vladimir Sitnikov
> And, again, TravisCI also does the same in a decent way, with the benefit
> of worrying less about underlying infra and security, and only drawback of
> being discoupled from a specific infra (is it really a drawback?)

Just in case: Apache Calcite uses both Travis CI and AppVeryor. It just
works.

It does simplify validation of the changes across various Java versions,
javadoc, RAT.
AppVeyor validates Windows builds.

Both results are automatically displayed at GitHub PR (e.g.
https://github.com/apache/calcite/pull/976) with no extra manual actions.

Vladimir


Re: New Committers - community

2018-12-29 Thread Vladimir Sitnikov
Stephen> Well on other GitHub orgs i’ve seen PR author has Merge
button once PR is
Stephen> approved by someone with push rights to the repo... until
they add a commit
Stephen> or the merge result changes

It does not work the way you describe.

1) By default GitHub defaults to "Base permissions" == "read".
Administrator can set even "base permissions == admin", however that
is not something that comes by default.
2) Repository administrator can configure a branch so it requires a PR
approval before merge (see
https://help.github.com/articles/enabling-required-reviews-for-pull-requests/
)

It might happen so that "author has Merge button" is a case when
repository was configured with "default=write" permissions + "require
PR approval".
Indeed that should produce (I have not checked) the behavior you
describe, however it was never a default mode for GiHub.

Vladimir

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



Re: New Committers - community

2018-12-29 Thread Vladimir Sitnikov
Stephen> Nah. Once committers have approved the PR then the PR can be
merged by the
Stephen> creator of the PR even if not a committer... at least that’s
the default
Stephen> way GitHub PRs work

By default one needs push rights on the repository to merge PRs.
"Approval" changes nothing. "PR approval" does not differ from any
other comment.
In other words, non-committers can't merge PRs.

Vladimir

-
To unsubscribe, e-mail: dev-unsubscr...@maven.apache.org
For additional commands, e-mail: dev-h...@maven.apache.org



MRRESOURCES-91 release request for maven-remote-resources-plugin

2018-09-16 Thread Vladimir Sitnikov
Hi,

Would anyone be so kind to release maven-remote-resources-plugin ?

My patch was merged more than 3 years ago, and the official release is
still missing.
Here's the JIRA link: https://issues.apache.org/jira/browse/MRRESOURCES-91

The problem is org.apache:apache pom uses maven-remote-resources-plugin to
create DEPENDENCIES file, and it causes re-build of the jar, sources-jar
every time.

Vladimir