Re: [compress] cut 1.16.1 very soon?

[Simon Spero]

Writing tests for osgi modules is painless... the second time :-P

The key to making tests easy is to use pax-exam, which handles most of the
tedious container setup and junit / test-NG test execution.

https://ops4j1.jira.com/wiki/spaces/PAXEXAM4/pages/54263870/Documentation

I can implement this if OSGI related changes are wanted.

Simon

On Feb 7, 2018 5:10 AM, "Stefan Bodewig"  wrote:

> On 2018-02-07, Jochen Wiedmann wrote:
>
> > On Wed, Feb 7, 2018 at 9:54 AM, Stefan Bodewig 
> wrote:
>
> >> I know I asked before (I really am not an OSGi guy at all) and was
> >> pointed into a direction for writing a test that would start a small
> >> container and load the bundle. This should help, but unfortunately the
> >> very idea slipped further and further down my priority list.
>
> > Well, if *you* are having trouble here, that's bound to happen for
> > others, too.  And, I don't know how they would implement that,
> > internally, but as it's their topic, I clearly have the expectation,
> > that *they* provide something useful.
>
> I'd be content with having a regression test, although I'm not sure how
> involved it would get. This time the Import-Package value was
> syntactically incorrect, this might get caught by static analysis. Last
> time I stripped an Import-Package (the one for XZ when I added the
> Brotli dependency IIRC), so the manifest was valid but you have been
> unable to use the algorithms provided by XZ. This is a runtime issue
> that really requires a test, I guess.
>
> > Btw: Would org.apache.felix:maven-bundle-plugin:3.5.0:verify have
> > spotted the problem?
>
> At least not when I use it the naive way (this is based on the rel/1.16
> tag:
>
> $ mvn org.apache.felix:maven-bundle-plugin:3.5.0:verify
> [INFO] Scanning for projects...
> [INFO]
> [INFO] 
> 
> [INFO] Building Apache Commons Compress 1.16
> [INFO] 
> 
> [INFO]
> [INFO] --- maven-bundle-plugin:3.5.0:verify (default-cli) @
> commons-compress ---
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time: 0.625 s
> [INFO] Finished at: 2018-02-07T11:04:54+01:00
> [INFO] Final Memory: 15M/605M
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.felix:maven-bundle-plugin:3.5.0:verify
> (default-cli) on project commons-compress: Execution default-cli of goal
> org.apache.felix:maven-bundle-plugin:3.5.0:verify failed.:
> NullPointerException -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the
> -e switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
>
> with -X I only get a Maven internal stack trace. Same for the 3.4.0
> version that is requested by commons-parent 43.
>
> TBH I don't want to dig deeper here as I really think we need runtime
> verification over static analysis.
>
> Stefan
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [LANG] Add module-info.java?

[Simon Spero]

[In regards to original question, -0.0  (harmless, but pointless, since
applications should not use lang as a jpms module. To be usable as a jpms
module, EVERY  release that has ANY api change must use new module and
package names).[1]  ]

Since all dependencies have to be shad(e|ow)ed (and not exported) when
using jigsaw, it's not too much of a hassle to do a  ½-assed job. The only
thing that might take a bit more work is if you want to have extra support
for services.

I haven't checked if there's a non-beta release, but ASM 6 has had support
for the new module-info class format stuff for a year or so (Remy Forax
being on the  jpms EG).

For custom tooling there's no real win in using module-info.java as input
(there was no real point in putting the metadata in .java and .class files
in the first place 臘‍♂️).

JSON would be one good lazy mode; this could also be used to generate the
.java file.

Another possibility is to derive  module-info data from  bundles. The
metadata can be extracted fairly easily using bndlib. This would be more
complicated if there were imports, but since the only modules commons
projects can safely import are the non-default jdk ones (and possibly not
even all of those), it's essy to use a static map (which can be pretty much
generated from jmod/jdep).

Simon

[1] A module can only appear in the modpath once. A package can only come
from one module. If an application uses multiple third party dependencies,
which use different versions of a fourth party module,  "*Lasciate ogne
speranza, voi ch'intrate".*


On Oct 16, 2017 5:57 AM, "sebb"  wrote:

> On 16 October 2017 at 10:34, Emmanuel Bourg  wrote:
> > Le 14/10/2017 à 14:43, Benedikt Ritter a écrit :
> >
> >> I’d like to hear opinions on this change.
> >
> > I wonder if we could somehow write a module-info.java compiler that
> > works with older version of Java. The syntax doesn't look terribly
> > complicated after all. It could be invoked from a plugin hooked to the
> > compile phase.
>
> Or skip the files if compiling with an older Java version?
>
> > Emmanuel Bourg
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> > For additional commands, e-mail: dev-h...@commons.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: Java 8 curiousity

[Simon Spero]

Could be gc, could be hotspot compiler, or could be library

There was a massive improvement in the performance of the G1 GC in u60 when
dealing with "humongous"  objects; however I assume you wouldn't have been
using G1 before u60 for anything real (that was when it first became
usable.

If you attach visualvm and sample, you should get a better idea of were the
performance wins are coming from.

wrapping the app as a pseudo jmh benchmark and running using perfasm as the
profiler (needs hsdis, Linux, and perf) may be more precise (e.g. better
inlining, different instruction use, etc).

Simon

On Oct 9, 2017 5:21 PM, "Rob Tompkins"  wrote:

Hey all,

At my day job we saw a 60% performance improvement (cpu utilization)
between 1.8.0_40 and 1.8.0_121, and I was wondering if anyone else out
there has seen anything like that before or if anyone might know what could
cause that given that the release notes  don’t directly point to anything.

Cheers,
-Rob
-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [Math][Numbers] Move Field, etc. to numbers?

[Simon Spero]

On Sep 15, 2017 5:30 PM, "Raymond DeCampo"  wrote:


Re terminology a field is an abstract system of numbers so I don't see an
issue there.


Let E = {Dunhill, Camel},

Let  Dunhill * Dunhill = Dunhill,
Dunhill * Camel = Dunhill,
Camel * Dunhill = Dunhill,
Camel * Camel = Camel

Let Dunhill + Dunhill = Dunhill,
   Dunhill + Camel = Camel,
   Camel + Dunhill = Camel,
   Camel + Camel = Dunhill

Then  is a Field, but of brands of cigarette.

 So I vote Camel or Dunhill depending.

Simon

Re: [CSV] CSVMutableRecord

[Simon Spero]

On Aug 15, 2017 8:01 PM, "Gilles"  wrote:

Saying that making record mutable is "breaking" is a bit unfair when we do
> NOT document the mutability of the class in the first place.
>

I'm stating a fact: class is currently immutable, change would make it
mutable; it is functionally breaking.
I didn't say that you are forbidden to do it; just that it would be unwise,
particularly if it would be to save a few bytes.


Exactly.

TL;DR. This is almost always a breaking semantic change; the safest  ways
of implementing it are binary breaking; it's unlikely to have a major
performance impact; it might be better to create a new API module for
enhancements, with current package as legacy or implementation.

If a class previously exposed no mutators, adding one is usually a major
change. This is especially true for final classes, but it still affects use
cases where an instance is owned by another class, which may rely on the
lack of mutability to avoid making defensive copies.
Of course, a final class that has a  package-private getter  to a shared
copy of its backing array could be considered to be sending mixed
messages...

It is possible that a mutable class might have significant performance
advantages over an immutable one beyond saving a few bytes. For example, if
the updates are simple, and depend on the previous value of the cell, then
a mutable version might have better cache behavior. If there's other
sources of cache pressure this might have a higher than expected impact.
The costs of copying the original values might also be relatively
significant.

For an ETL use case these issues are unlikely to be limiting factors; for a
start, there's a non-zero chance that a  CSVRecord was extracted  by
parsing a CSV file. Also a transform will require conversion to some sort
of Number (or String allocation).

The current API doesn't easily support adding alternate implementations of
the relevant types. Implementation classes are final, and important  return
types are concrete.

One solution might be to treat the current code as almost an implementation
module, define a separate API module, and add extra interfaces and
alternate  implementations to support  the target use case (mutable
records, streams, reactivex, transform functions or what have you).

Simon

Re: Ready for JDK 9 ?

[Simon Spero]

Compress HEAD is tested against the equivalent of RC. The main issues were
with tests; some types of mocking (especially of concrete classes) don't
work. This might have been fixed by now.
I believe that the latest jacoco is 9 compatible.

[The biggest problem was caused by a bug in the zip code handling a
particular kind of timestamp; massive changes to the jdk implementation of
zip caused tests that had been passing (but shouldn't have) to fail
properly.]

NOTE:

Adding a Module name manifest header asserts that the code is tested
against Java 9. This is documented in the minutes of the armistice talks.

jigsaw modules are pretty useless for most of Commons (consumers pretty
much have to shade dependencies). [ subliminal whisper about benefits of
having correct OSGI headers]

Simon

On Aug 8, 2017 11:08 AM, "Jörg Schaible" 
wrote:

> Hi,
>
> Gilles wrote:
>
> > Hi.
> >
> > On Tue, 8 Aug 2017 11:09:01 +0100, Rory O'Donnell wrote:
> >> Hi Benedikt,
> >>
> >> Thank you very much for all your testing of JDK 9 during its
> >> development! Such contributions have significantly helped shape and
> >> improve JDK 9.
> >>
> >> Now that we have reached the JDK 9 Final Release Candidate phase [1]
> >> , I would like to ask if your project can be considered to be 'ready
> >> for JDK 9',
> >
> > Is there some simple thing to do in order to be able to answer
> > that question?
>
> IMHO no. Definitelly not in general for all components. Practically we
> would
> have to checkout the latest releases from source (or use the source
> tarballs) and run at least the unit tests with this Java 9 RC.
>
> Cheers,
> Jörg
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [RNG] Generating for [e]nums

[Simon Spero]

On Aug 4, 2017 2:11 PM, "Gary Gregory"  wrote:

For example, I have a enum like:

public enum CardinalDirection (NORTH,SOUTH,EAST,WEST)

public CardinalDirection nextRandomDirection() {
   return rng.next(CardinalDirection.class);
}


This approach may be a bit slow if the random values are near an inner
loop.

In order to select a random enum value, you need to get hold of an array
containing all the possible values (equivalent to what gets returned by the
generated  "public static CardinalDirection[] values()".

This requires calling Class::getEnumConstants, which does a little
reflection, caches the result, then returns a clone of the resultant array.

It may be better to construct a random enum value generator object that
holds a copy of the values array, plus an RNG with the desired
characteristics.

A more general method would take an array of values, or a Supplier
function (e.g. CardinalDirection::values).

You could use a cache in the RNG, keyed by the class, but that's still
going to be a bit expensive in the middle of a tight loop.

Simon

Simon

Language level, Class files, MRJARs & versioning (was Re: [collections][proposal] Java 7)

[Simon Spero]

Class file format is not treated as a breaking change under most versioning
approaches, including the JLS.

The  checkers I looked at that reported on class file format   changes
consider it a micro level version change (+0.0.1)

The past few major version bumps for projects I've worked happened to
coincide with Java version bumps, but they involved major uses of new
features like annotations / try-with-resources / lambda + streams.

These changes were of necessity Major, since there was no default method
support until 8.

Now... MRJARs ( http://openjdk.java.net/jeps/238 )
TL;DR; MRJARs are a mess. Temperature to be determined.

With Java 9, there is now technically the capability to have multiple
versions of classes, targeting  different language levels,  in the same jar
file. This was supposed to be back ported to 8, but that got lost along the
way, so currently it's only usable for 9/ <9.
There's only minimal support at the tool level (threads with maven / gradle
 samples, with authors' apologies :).

There's also no easy way to write code with conditional compilation blocks
(e.g. in jdk9 there's a new interface method in added to j.u.zip.Checksum
that is pulled up from CRC32; there's no easy way to write source  code to
use the method when compiling for jdk 9 but do  instanceof CRC32 / call
virtual if jdk8. Templates or CPP are less than ideal (see ByteBuffer and
friends).

There are some places in commons where a JDK 9 library addition is
important. For example, codec has a CRC32C implementation, and now so does
jdk 9; both are pure Java slice-by-eight... except the jdk version has
annotations marking critical methods as hotspot intrinsic candidates, which
means that on most major machines the Java code will be ignored in favor of
hardware supported carryless multiplication if available.

Coding for multi release jars is something that might be worth developing
code & policies for; some of this waiting on  maven plugin  changes, but
arranging source trees, minimizing copy & paste,  and making sure that the
right tests are run in the right environments is more awkward (e.g. do
some/all unit tests now have to run after package, as if they were
integration tests? Are tests multi-versioned? ).

And returning to the original topic, what should happen to version numbers
if the Java 9 class  needs a minor version bump, but the Java < 9 version
doesn't.


On Jul 28, 2017 8:41 AM, "Rob Tompkins"  wrote:


> On Jul 27, 2017, at 12:53 PM, Gary Gregory  wrote:
>
> We have not in the past forced a major version and Maven coordinate change
> when updating the Java platform. We could do that of course.

I’m a tad surprised by this. It feels like updating the minimum
accommodated java version would necessitate a major version change.

-Rob

>
> Gary
>
> On Jul 27, 2017 06:47, "Jochen Wiedmann" 
wrote:
>
>> Wouldn't that make it 5.0? (Binary incompatible change)
>>
>> Note: I am not opposing the change, I just propose an additional
>> change in the version number.
>>
>> Jochen
>>
>>
>> On Wed, Jul 26, 2017 at 12:18 AM, Gary Gregory 
>> wrote:
>>> Hi All:
>>>
>>> I propose we make Java 7 the minimum for Commons Collection 4.2.
>>>
>>> Gary
>>
>>
>>
>> --
>> The next time you hear: "Don't reinvent the wheel!"
>>
>> http://www.keystonedevelopment.co.uk/wp-content/uploads/2014/10/
>> evolution-of-the-wheel-300x85.jpg
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
>> For additional commands, e-mail: dev-h...@commons.apache.org
>>
>>


-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [lang][collections] SortedProperties

[Simon Spero]

On Jul 18, 2017 2:25 PM, "Rob Tompkins"  wrote:

I'm stuck in the in-between here with the following thought: HashTable
certainly feels like a collection of objects, but it clearly extends
Dictionary and isn't in the collections family. But we are in java.util
here and not in java.lang, so that feels more like it could be in
collections. Plus properties are essentially a map, despite they're being
implemented on top of hashtable.


HashTable extends Dictionary implements Map.

Of course,  Properties extends HashTable  because legacy.

In jdk 9, Properties still extends HashTable.
 However, HashTable now has a magic package private constructor
"HashTable(Void)", which is used by Properties to tell HashTable to go sit
in a corner, with all fields left set to default values.

Properties now uses a ConcurrentHashMap internally, with only a few methods
remaining synchronized. Everything gets delegated to the chm.

Still full of unchecked goodness.

And a great example of  how sometimes inheritance is so useless you just
compose and delegate anyway.

Simon

Re: [lang][collections] SortedProperties

[Simon Spero]

On Jul 18, 2017 11:39 AM, "Gary Gregory"  wrote:

My use case is to write a Properties object in sorted key order, nothing
fancy. The simplest way is to subclass Properties and override keys().

Loading a SortedProperties with a Map is not on my to-do list.




Would your use case be served by one or more static methods?
These methods might need as  arguments:
1) Some form of output sink
2) An instance of Map m satisfying the conditions:
   a) every key in m is an instance of String.
   b) every value in m is an instance of String.

The constraints 2a and 2b   are required   be satisfied by any Properties
object  any time at  it is output.

(Because of the legacy derivation from Hashtable, these  conditions may be
violated at other times without an error being signaled. This also makes
the concept of a Sorted Properties instance problematic, as these
keys/values may be comparable with String.)

There might be different entry points for sorted output, and for output in
the natural order of the map (which might  be sorted).

There may need to be a specialized default comparator if parts of the keys
are to be compared in non lexicographic order (for example, if  a key
contains sequences of digits that should be compared as numbers, or if keys
are hierarchical, with components  separated by periods).

Values need to be encoded wrt the special escape format defined for
properties files.

As a separate set of functionality, a corresponding method or set of
methods could be provided to support populating a Map from an input source
containing contents formatted as specified in j.u.Properties.

So,  io with a touch of codec?

Simon

Re: [daemon] moving to git ? and bump java version.

[Simon Spero]

On Jul 16, 2017 11:49 AM, "Matt Sicker"  wrote:

C quality somewhat depends on which version of C you're trying to remain
compatible with (I'm guessing C89 due to Windows, though I could be wrong).
Valgrind and other tracing tools are typically used.


Valgrind is a default choice (though then you still have plenty of
configuration choices to make :)

A similar set of functionality is available using the various sanitizers
available in clang (and recent versions of gcc, though I haven't tried
that). These are described in the clang user manual:

http://clang.llvm.org/docs/UsersManual.html#controlling-code-generation

Clang also supports static analysis, which can, um, detect errors
statically. This can sometimes generate a bunch of false positives,
depending on the coding  style. The analyzer typically produces an html
report.  These analyzers can be run using scan build.

There are quite a lot more static analyzers available via clang-tidy, which
may or  may not  require installing an extra package.

CppUnit is a sensible choice for unit tests, but does require that tests be
written in C++, which might be a problem if you are sent back in time (tip:
write to Bjarne at AT Bell Laboratories, and ask for a tape).

Re: [compress] HasCharset is a bad name

[Simon Spero]

Since it's an interface, I could change it to IHasACharset?

Or If you prefer I  could rename it to YouGiveLove?  (Lucky Millenials- you
aren't headsonged)

The name follows a pattern for interfaces of this sort, which are basically
retrofit markers for the presence of property, (with associated getters).
I've seen it used in other projects, but it might be bleed through from
some  OWL / RDF patterns.

I'm not in love with the pattern but it required the least thought :)

(java 8 makes this sort of interface moot, as the accessor can be added as
a default method. It is even be a candidate for the proper use of Optional.
)

On Jul 5, 2017 1:14 PM, "Gary Gregory"  wrote:

Hi All,

The new interface name HasCharset is pretty bad IMO.

CharsetProvider is the obvious (to me) better name even though there
already is a class called java.nio.charset.spi.CharsetProvider.

Alternatives could be CharsetContainer, CharsetAccessor, CharsetGetter, ?

Gary

Re: [compress] differences in implementation of Zip ibm vs. oracle?

[Simon Spero]

Is this jdk 8 vs.  java SDK 8? The oracle code picked up some fairly big
changes along the way that aren't present in the IBM library.  (jdk 9 is
really different from both. I haven't seen what the IBM Java 9 is like
yet).

I will see if anything looks blatantly obvious.

Simon


On Jul 10, 2017 3:32 PM, "Allison, Timothy B."  wrote:

Compress colleagues,

  Over on https://bz.apache.org/bugzilla/show_bug.cgi?id=61275, a user
submitted two .xlsx files generated with Apache POI, one by IBM's jvm and
one by Oracle's jvm.  The file generated with Oracle's jvm opens without
issue; however, MSOffice complains but can fix the file generated by IBM's
jvm.  Winzip opens both without complaining.

  Does this ring a bell?  Have you seen this before?  Anything we can do on
our (POI's) side to fix this?

   Thank you.

 Best,

   Tim

Re: Fwd: [jira] [Created] (DISCOVERY-23)

[Simon Spero]

The clue is in the issue id.

Disco-very. Disco-rdia.

Part of commons, they share a common prefix.

What is the prefix? Disco.

How many letters in that prefix? 5 -  2 + 3 !

Disco →sf-raves →hyperreal →Apache.

Coincidence? Then how do you explain the fnords? Or the Twitter? What
better way to hide the conspiracy than in the heartblood of the conspiracy
theorists.

All hail Eris!

Simon // Ok, it could spam, but that just doesn't seem as plausible.

Re: Note on Coveralls and JDK9

[Simon Spero]

It was failing when I got here officer

On Jul 4, 2017 1:46 PM, "Pascal Schumacher" <pascalschumac...@gmx.net>
wrote:

> Am 04.07.2017 um 16:17 schrieb Simon Spero:
>
>> It doesn't work.  :-)
>>
>> More specifically, it doesn't work because coveralls is trying to use
>> JAXB,
>> which is not available by default in JDK9.  (It ships. It's just not
>> available without command line arguments.)
>>
>> The coveralls plugin is using JAXB for a single static method, to convert
>> a
>> byte[] digest into a hex string.  That's such a common form of encoding -
>> if only there was some sort of library to do that.
>>
>> 樂
>>
>> PR submitted.
>>
>
> Pull request build failed ;-)
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Note on Coveralls and JDK9

[Simon Spero]

It doesn't work.  :-)

More specifically, it doesn't work because coveralls is trying to use JAXB,
which is not available by default in JDK9.  (It ships. It's just not
available without command line arguments.)

The coveralls plugin is using JAXB for a single static method, to convert a
byte[] digest into a hex string.  That's such a common form of encoding -
if only there was some sort of library to do that.

樂

PR submitted.

Things in a POM file that are not probably incorrect

[Simon Spero]

1.5
1.5
org.apache.commons.functor

JDK 9 cannot generate or parse class files compiled with -target 1.5  [1].

Per the JPMS Armistice talks,   "Automatic-Module-Name" header should only
be added to jars that have been tested under JDK 9 [2].

Also, javac now supports the new "--release" argument as an alternative to
source + target.
Roughly speaking, this option combines -source & -target with a little bit
of animal sniffing.  JDK 9 includes  signatures for JDK 6,7,8, and 9 (i.e.
all the releases which jdk9 can compile for).
This option is supported by maven-compiler-plugin [4] via a 
element in the plugin  configuration, or by settting the property
*maven.compiler.release* .

Simon

[1] https://bugs.openjdk.java.net/browse/JDK-8011044
[2]
http://openjdk.java.net/projects/jigsaw/spec/minutes/2017-05-18#AutomaticModuleNames--ModuleNameInManifest
[3] http://openjdk.java.net/jeps/247
[4]
https://maven.apache.org/plugins/maven-compiler-plugin/compile-mojo.html#release

Re: [FUNCTOR] Why do we have an API module?

[Simon Spero]

On Jun 25, 2017 5:54 AM, "Benedikt Ritter"  wrote:

Hi,

> Am 24.06.2017 um 20:19 schrieb Matt Benson :
>
> TBH, I don't know that I think there's a reason to do such a
library unless it's really a game changer.

Currently functor is somehow in limbo state. It has been promoted to
proper, but there has never been a release. So I wonder what we want to do
with it.


I think Matt has the right of it.

If there's no current need for Functor, and it's never had a  release, does
it need to make a sound?

A lot of the current snapshot is redundant under plain old JDK 8;  much of
the rest is subsumed by various functional and/or reactive libraries (of
which there are quite a few).  AOL's Cyclops [1] bridges a bunch of these
(e.g. [2],[3].[4]). Is there anything that isn't provided for in cyclops or
one of the things it bridges?

Simon

[1] https://github.com/aol/cyclops-react/wiki
[2] https://www.vavr.io/vavr-docs/
[3] https://github.com/jOOQ/jOOL
[4] https://github.com/reactivex/rxjava

Re: [configuration] checkstyle fails build

[Simon Spero]

On Jun 24, 2017 12:31 PM, "Oliver Heger" 
wrote:

Should changes related to the setup and handling of checkstyle then be
done for Commons as a whole?

(No P.S.)


It is the sort of thing fits nearly I  a parent. Maven is one example (not
sure how much customization is needed).

Also, in Java 9 you can get rid of the NoPS with the spin-wait hinting.

Simon

Some ideas for improving Travis build performance improvements:

[Simon Spero]

COMPRESS-413 has some examples  which include:

   - Switching to the trusty container, which doesn't require installing as
   many updates.
   - Use only two JDKs: openjdk7 and openjdk8.
  - JDK 7 is obsolete -
  - Oracle EOL'ed JDK7  over two years ago; openjdk7 occasionally gets
  patched (though it's not available from the default source in recent
  versions of ubuntu, including xenial (16.0.4), which is the current LTS.
  - Java 7 is mostly relevant only for older versions of Android. Some
  JLS 8 features are not supported for versions of Android prior
to 7.0 (e.g.
  Repeatable annotations, and interface default/ static methods).  However,
  this environment is not tested for under oracle or openjdk 7.  It's also
  painful to build for under maven.
   - Run the entire CI process in the build phase. This avoids compiling
   and testing two or three times over. However, any errors will fail the
   build.
   - Cache build dependencies (this is another reason to move plugin work
   to the build phase, since the cache is saved before after_success is run).
   - Add a mvnw setup. 3.5.0 may be  needed if jdk9ea is added to the build
   matrix.
   - The pull request also has random test coverage increases to pacify a
   confused coveralls.

See: https://github.com/apache/commons-compress/pull/39

which combined with previous changes from~

 Ran for 5 min 4 sec
 Total time 13 min 26 sec

to:

 Ran for 1 min 56 sec
 Total time 1 min 50 sec


[Note that some variation comes from primed/unprimed caches]

Re: [configuration] checkstyle fails build

[Simon Spero]

On Jun 23, 2017 11:03 AM, "Oliver Heger" 
wrote:

However, letting the build fail because of checkstyle error is too
restrictive IMHO. My approach is to work through the errors before creating
a new release. This has the disadvantage that errors might accumulate; but
from one release to the next one there is typically not that much.


That's still my gut feeling- checkstyle build fails are infuriating - but
annoyingly there are issues doing things at the release stage that are
worse  

The big problem is related to recent discussions about code styles and
commits, and the diff bloats and blame/praise shifting that happen when
formatting drifts during a development cycle.  Things turn out much better
if formatting / style checking is done before a commit, or at least fails
the build so that things can be fixed and disappeared if a PR is merged as
a SquashBase (so a change  + a revert cancels out ).

IntelliJ has a checkstyle plugin, which, um, checks for checkstyle
violations. This can run as a real time inspection, or during pre-commit
analysis. It can also adjust code formatting options to match the
checkstyle profile,which helps avoid many issues in the first place.
Eclipse has  similar support, but I am not an Eclipse user so I don't know
the details.

The validate phase checkstyle execution then becomes more of a backstop
(validation is usually the right phase, since that is prior to
compilation).  It's a lot less annoying if it doesn't have too many
"violations".

Since checkstyle can be configured to allow a small number of violations,
and to only treat rule-breakings above a certain severity  as violations
(error by default, but a pre-release execution could increase fussiness to
include warnings).

One thing that is tricky, but which can be worth it, is having a pseudo
flag-day reformatting, where you use an ide to apply the code style to the
entire project, then apply the changes all at once (possibly using a
disposable committer, though this isn't as important if the annotate view
you use shows a bit of commit message).

It is a bit more effort if there a bunch of unmerged branches, but since
the formatting changes  are automated, they can be applied on the branch so
it's not *that* horrible to get a mergeable branch back.

History checking for a few lines may require going one revision back, but
having all the format fixes in a single commit is a lot better than having
them mixed in with real changes.

Simon
 P. S.

I prefer the builtin /google_checks.xml styleguide to the older sun_checks,
partly because it's more up to date, and partly because Google provides
native  ide configuration files for Eclipse and intellij so there's no need
to import the checkstyle file.
The Checkstyle plugin  tracks the Google style guide pretty closely, so as
long as the plugin is up to date, there won't be much divergence. The most
significant changes happen when there are new constructs to consider (e.g.
lambda).

P.P.S.
I do think the google rules are wrong about horizontal alignment (at least
for continuation lines). It makes a huge difference when you have to use
obnoxious amounts of chained methods (can't say fluent without saying flu).


P.P.S.
I do like their tip on the appropriate use of finalize:

*Tip:* Don't do it. If you absolutely must, first read and understand
*Effective
Java* Item 7,  "Avoid
Finalizers," very carefully, and *then* don't do it.

I say finalizers are OK as long as they  warn, and don't actually  do
anything ("you didn't commit a batch insert of 10M items. Were you sure?
Yes / Tough").

Re: OSGi Version at Package Level

[Simon Spero]

On Tue, Jun 20, 2017 at 10:18 AM, Stefan Bodewig  wrote:

>
> Interesting. This means OSGi only provides a way for consumers to say
> which version of an API they want, but not which implementation. This means
> as a consumer you can't say "I want version 1.19 of Compress because I know
> it contains an important fix". This feels more limited in expressiveness
> than I had expected.
>

Bundles can specify all sorts of Requirements, including implementations,
and bugfix version ranges  (by default, the bugfix component is not
specified in the lower bound, but that is just a default).   It can be a
little too expressive :-)

If a new version becomes available, it can often be swapped in dynamically
(e.g. when using Declarative Services (née Felix SCR)  with Greedy dynamic
reference, an inject  service will be replaced whenever a newer version is
available).

For more on updating and refreshing bundles see:
https://stackoverflow.com/questions/4330927/how-does-osgi-bundle-update-work


I do feel that it ought to be possible to have  more aggressive updates for
critical (security) release, though  that's a little orthogonal to the
version range issue.

Re: [compress][all]Fwd: Build failed in Jenkins: Commons-Compress PullRequest Builder #56

[Simon Spero]

Hudkins is alive!

On Tue, Jun 20, 2017 at 12:49 AM, Dave Fisher  wrote:

> Check with Infra weird errors are happening.
>
> Regards,
> Dave
>
> Sent from my iPhone
>
> > On Jun 19, 2017, at 8:53 PM, Stefan Bodewig  wrote:
> >
> >> On 2017-06-19, Gary Gregory wrote:
> >>
> >> Is it just me or have there been Jenkins failures left and right for the
> >> last week or so?
> >
> > That's the Jenkins job I've created for github pull requests and it
> > fails whenever there are merge conflicts, for example.
> >
> > Right now it looks as if the workspace was caught in an unhealthy state,
> > I'll wipe it and reconfigure the job to perform clean checkouts.
> >
> > Stefan
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> > For additional commands, e-mail: dev-h...@commons.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [all] dev@ traffic

[Simon Spero]

On Jun 19, 2017 9:56 AM, "Stefan Bodewig" <bode...@apache.org> wrote:

On 2017-06-19, Simon Spero wrote:

with all parenteses properly nested, wow ;-)

No need to apologize, really.


(apply #'append '((not) (a problem)))

Code formatting for COMMONS (was Re: [GitHub] commons-text issue #52: Test: Improved testcase coverage for StrBuilder)

[Simon Spero]

Is there a set of commons code formatting conventions? More particularly,
is there a machine readable file of same? I use intellij, so native,
checkstyle, or Eclipse is fine.

I've been having problems where I am doing the pre-commit diff check, and
find that I have made unintentional changes to whitespace, and which
intellij and I haven't figured the pattern to.

Checkstyle as plugin is still annoying, but only when we disagree :)

Simon

On Jun 18, 2017 2:33 AM, "PascalSchumacher"  wrote:

Github user PascalSchumacher commented on the issue:

https://github.com/apache/commons-text/pull/52

Thanks!

Please separate the formatting fixes and the test additions into
different commits.

Please stick to the existing code style for the new tests (no new line
at the beginning and at the end of a test method; single new line between
test methods).



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [all] dev@ traffic

[Simon Spero]

(this was prompted by me interrupting a quasi-appropriate JIRA
meta-discussion about JIRA workflow & permissions (on an issue I had opened
because I encountered it whilst working another issue, and had fixed but
not PRed, which Stefan then also fixed)  to further apologize to Stefan for
not replying to an email from him (on dev)  that I'd missed because of
coveralls) .

So +1.
Also, I need to reply to that email :)

Simon

On Jun 19, 2017 6:33 AM, "Stefan Bodewig"  wrote:

Hi all

apart from dev@ and user@ we've got three lists that have been created
for specific notifications that people may be interested in or not:
issues, commits and notifications. My MUA is configured to merge those
three with dev so I didn't really notice we are getting more messages
(and for most people more noise) to dev than I would have expected.

JIRA seems to be set up to send to issues@ for all components, and the
git and svn commit messages of components go to commits@ while svn
commit messages for the website go to notifications@.

Things are different between components for at least

* github integration (some components like Compress send mails to dev,
  others like Numbers send them to notifications)

* CI messages (at least Compress uses dev for Jenkins mails)

IMHO neither should go to dev@ and using notifications@ seems more
logical to me. I'm going to change this for Compress and would likt to
ask other components to check their settings as well so we can raise the
signal ratio for dev@ again.

Stefan

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

CharSequence vs. String (was Re: [GitHub] commons-text pull request #46: TEXT-85:Added CaseUtils class with camel case...)

[Simon Spero]

On Jun 12, 2017 10:47 AM, "arunvinudss"  wrote:

Github user arunvinudss commented on a diff in the pull request:

I am a bit biased towards using String instead of CharSequence . Yes
CharSequence allows us to pass String Buffers and builders and other types
as input potentially increasing the scope of the function but considering
the nature of work we do in this particular method it may not necessarily
be a good idea. My basic contention is that the minute we call toString()
on a charSequence  to do any sort of manipulation it becomes a costly
operation and we may lose performance .


True if the particular CharSequence is not in fact an instance of String.
String::toString returns this.

The bigger problem is that too many methods use String as a parameter or
return type, when  CharSequence would serve just as well. This indeed
requires the invocation of Object::toString.

For methods that use String as the return type, changing the result to
CharSequence is source and binary incompatible, and properly so (since at
some point the user may actually need a String).

A  generic method with Type parameter with CharSequence as bound (T extends
CharSequence) can sometimes be useful, and can be added in addition to
methods taking String arguments, but can't replace them.

There are some places in javac that have special treatment for String - for
example, the + operator , but jdk9 reduces that particular win by indyfying
concat.
If a method doesn't intrinsically require a String, then I prefer
CharSequence. It's probable that sooner or later something is going to
demand a String, but that's not a good reason to be "that guy" :-)

Note:
Strings can be an incredible waste of memory; 40 +  ⌈length/4⌉  bytes
(reduced to a mere  40 + ⌈length/8⌉ bytes in jdk9 when compact strings can
be used).

 This is incredibly painful if you have a vast number of small "strings",
which may not all need to be materialized simultaneously. See e.g. [1]
(~50MiB of UTF-8 chars becomes ~250MiB of Strings. And since there's no
individual humongous object  they all get to make the journey from TLAB to
Old Space the hard way. Note this predates jdk 9,but illustrates some of
the win from compact strings)

Storing the character data in a shared byte array is a huge win. Someone
should tell the jdk implementors to look at applications that do this.
Like, um, javac :-)

Materializing these strings as possibly transient  CharSequence's  is
really convenient... until some method just has to have a String

Also, wouldn't  some sort of low-space-overhead string storage be a good
fit for text?

Simon
[1]  Spero,S. (2015). Time And Relative Dimensions In Semantics: Is OWL
Bigger On The Inside? OWLED 2015. Available at
http://cgi.csc.liv.ac.uk/~valli/OWLED2015/OWLED_2015_paper_12.pdf

Re: [CLI] Please review CLI-277

[Simon Spero]

But BC, per initial response. I said I needed more coffee :)

On Wed, Jun 14, 2017 at 8:55 AM, Simon Spero <sesunc...@gmail.com> wrote:

> Oops -I missed that the method was already generic. Need bigger
> tablet/more coffee / bigger caffeine tablet
>
>
> On Jun 14, 2017 8:46 AM, "Simon Spero" <sesunc...@gmail.com> wrote:
>
> On Jun 14, 2017 4:28 AM, "Benedikt Ritter" <brit...@apache.org> wrote:
>
> I’d like to have feedback for CLI-277 [1], especially whether it will
> affect BC. Clirr is happy with this changes, to I suppose it is fine to
> merge this?
>
> [1] https://github.com/apache/commons-cli/pull/13 <
> https://github.com/apache/commons-cli/pull/13>
>
>
> The changes look like they ought to be ok, but I need to  check on
> something that isn't a tablet.
>
> In general, going from raw to generic  is easy, since the type erasure are
> the same, but changing things once they are generic gets harder. As des
> Rivers et. al. (2007,2017) note:
>
> But, also bear in mind that there are severe constraints on how a type or
> method that already is generic can be compatibly evolved with respect to
> its type parameters (see the tables above). So if you plan to generify an
> API, remember that you only get one chance (release), to get it right. In
> particular, if you change a type in an API signature from the raw type "
> List" to "List" or "List", you will be locked into that
> decision. The moral is that generifying an existing API is something that
> should be considered from the perspective of the API as a whole rather than
> piecemeal on a method-by-method or class-by-class basis.
>
>
> https://wiki.eclipse.org/Evolving_Java-based_APIs_2#Turning_
> non-generic_types_and_methods_into_generic_ones
>
> These guidelines are one of the best reference sources for information on
> compatibility. Definitely more comprehensible than the JLS.
>
>
>

Re: [CLI] Please review CLI-277

[Simon Spero]

Oops -I missed that the method was already generic. Need bigger tablet/more
coffee / bigger caffeine tablet

On Jun 14, 2017 8:46 AM, "Simon Spero" <sesunc...@gmail.com> wrote:

On Jun 14, 2017 4:28 AM, "Benedikt Ritter" <brit...@apache.org> wrote:

I’d like to have feedback for CLI-277 [1], especially whether it will
affect BC. Clirr is happy with this changes, to I suppose it is fine to
merge this?

[1] https://github.com/apache/commons-cli/pull/13 <
https://github.com/apache/commons-cli/pull/13>


The changes look like they ought to be ok, but I need to  check on
something that isn't a tablet.

In general, going from raw to generic  is easy, since the type erasure are
the same, but changing things once they are generic gets harder. As des
Rivers et. al. (2007,2017) note:

But, also bear in mind that there are severe constraints on how a type or
method that already is generic can be compatibly evolved with respect to
its type parameters (see the tables above). So if you plan to generify an
API, remember that you only get one chance (release), to get it right. In
particular, if you change a type in an API signature from the raw type "List"
to "List" or "List", you will be locked into that decision. The
moral is that generifying an existing API is something that should be
considered from the perspective of the API as a whole rather than piecemeal
on a method-by-method or class-by-class basis.


https://wiki.eclipse.org/Evolving_Java-based_APIs_2#Turning_
non-generic_types_and_methods_into_generic_ones

These guidelines are one of the best reference sources for information on
compatibility. Definitely more comprehensible than the JLS.

[CLI] Please review CLI-277

[Simon Spero]

On Jun 14, 2017 4:28 AM, "Benedikt Ritter"  wrote:

I’d like to have feedback for CLI-277 [1], especially whether it will
affect BC. Clirr is happy with this changes, to I suppose it is fine to
merge this?

[1] https://github.com/apache/commons-cli/pull/13 <
https://github.com/apache/commons-cli/pull/13>


The changes look like they ought to be ok, but I need to  check on
something that isn't a tablet.

In general, going from raw to generic  is easy, since the type erasure are
the same, but changing things once they are generic gets harder. As des
Rivers et. al. (2007,2017) note:

But, also bear in mind that there are severe constraints on how a type or
method that already is generic can be compatibly evolved with respect to
its type parameters (see the tables above). So if you plan to generify an
API, remember that you only get one chance (release), to get it right. In
particular, if you change a type in an API signature from the raw type "List"
to "List" or "List", you will be locked into that decision. The
moral is that generifying an existing API is something that should be
considered from the perspective of the API as a whole rather than piecemeal
on a method-by-method or class-by-class basis.


https://wiki.eclipse.org/Evolving_Java-based_APIs_2#Turning_
non-generic_types_and_methods_into_generic_ones

These guidelines are one of the best reference sources for information on
compatibility. Definitely more comprehensible than the JLS.

Re: OSGI Version at Package Level : some more details about the COMPRESS-399 pull request

[Simon Spero]

On Jun 13, 2017 1:48 PM, "Gary Gregory" <garydgreg...@gmail.com> wrote:

I think that the best way to see how this works is to provide a patch...
then we can see how it goes.


Gary-
  Can you clarify a few things:

What would you like to see patches for - the commons parent / related mojo
one or more  commons projects, the Felix maven-bundle-plugin, or some
combination thereof?

The changes which I submitted as a pull-request for COMPRESS-399¹ (as
amended) are on
commons-compress only. https://github.com/apache/commons-compress/pull/26

The net changes:

1. Update the version of the maven-bundle-plugin used.
2. Add the bundle:baseline goal to the verify phase.
3. Add a "provided" scope maven dependency on org.osgi:org.osgi.annotation:
for the  @Version annotation.
4. Override the parent bundle:manifest configuration, with no version in
the export-package (this would override any other source of version info).
5. Add initial package-info.java files to each non-empty package,
initialized to the previously released version.
6. Change the travis.yml configuration to execute mvn verify as the build
task.
7. Add  bundle:baseline-report goal to the site phase.

An example of additional work after  a subsequent api change is
https://github.com/apache/commons-compress/pull/27/commits/82405c13bd2688817108d3f2854387b3417a764d

Some of these changes can be lifted to the parent; the initial setup of
package-info.java @Version annotations requires some code.

Simon

¹ OK, I made the changes, created the issue, then submitted the
pull-request.


On Mon, Jun 12, 2017 at 10:18 AM, Simon Spero <sesunc...@gmail.com> wrote:

> On Mon, Jun 12, 2017 at 10:53 AM, Matt Sicker <boa...@gmail.com> wrote:
>
> > I'd love to have a semantic versioning plugin like that which can
suggest
> > what the version number is for the entire project. Elm (programming
> > language) has a tool like that, and it works rather well in practice
> (even
> > though there are some popular libraries that are at version 5.0.0
already
> > because of it).
>
>
> The bundle:baseline goal does this (it's at the start of the report).
> Having  frequent major version bumps is a good thing if they are necessary
> and the libraries are small (definitely better than having breaking
changes
> without the bump).
> It's not as good as having *correct* semantic-version numbers of 1.4.0 :-)
>

Re: OSGI Version at Package Level : some more details about the COMPRESS-399 pull request

[Simon Spero]

On Mon, Jun 12, 2017 at 10:53 AM, Matt Sicker  wrote:

> I'd love to have a semantic versioning plugin like that which can suggest
> what the version number is for the entire project. Elm (programming
> language) has a tool like that, and it works rather well in practice (even
> though there are some popular libraries that are at version 5.0.0 already
> because of it).


The bundle:baseline goal does this (it's at the start of the report).
Having  frequent major version bumps is a good thing if they are necessary
and the libraries are small (definitely better than having breaking changes
without the bump).
It's not as good as having *correct* semantic-version numbers of 1.4.0 :-)

Re: OSGI Version at Package Level : some more details about the COMPRESS-399 pull request

[Simon Spero]

On Sun, Jun 11, 2017 at 7:37 PM, Gary Gregory 
wrote:

> Is one upshot of this is that we should base the version number based on
> this OSGi report tool?
>

There are a few choices so the SEF upshot selector

should
be set to Fun :-)

One group of choices revolve around the term "version number".   There are
a few different version numbers involved here.

   1. The Maven version number  (i.e. ${project.version} )
   2. The OSGI bundle version (i.e. the one in the  "Bundle-Version"
   manifest header).
   3. The package version for each exported package (e.g. the value of
   version in the manifest header below)
  - Export-Package: org.apache.common.weasels;version="1.1.0"

Another group of choices revolve around interpretations of the term "base".

   1. automatically applying all suggested version numbers, without further
   review (e.g. for maven, writing  goals for the packaging phases).
   2. executing the baseline goal during the verify stage, possibly failing
   the build if the versions contain errors (or warnings), possibly only
   reporting.

Another level  of choice  is whether to have  different policies for
different levels of change.

   -  For example, if the maven version is taken as a given, code could be
   written to restrict changes to the level implied by  the maven version
   change. Thus, micro releases would be failed if any API changes, minor
   versions would be failed if no API changes, or any  Major changes, were
   found.

Some specific decisions:

   1. Setting the maven-bump match the maximum API change.
  - Advantages: This gives appropriate behavior when maven version
  ranges are used (e.g. [1.1,2)).
  - Disadvantages: If the artifact contains a lot of unrelated
  functionality, this may unnecessarily affect dependents that
don't need to
  change. This is especially problematic if it's a major-level change.
   2. Requiring *all* package version numbers to match.
  - This either requires (1), or the addition of a new, manually set
  property
  - Advantages: Require-Bundle imports with ranges will work (however,
  these are not considered good practice)
  - Disadvantages: Packages with changes at a lower level than the
  maximum will nevertheless be treated as if they had the higher level of
  changes.
 - This can require redundant  versions of identical classes to be
 loaded (if the imported packages are not used in any exported packages)
 - This can cause a system to ignore an updated artifact containing
 fixes to packages it uses, solely because of changes to
package it does not
 use.
 - In the worst case, cause systems that would otherwise have been
 resolvable to fail (e.g if a bundle wants to import two
bundles that would
 otherwise have had compatible  ranges for a used package to
no longer have
 a compatible range).
  3. Having per-package versions:
  - This requires (e.g.) incrementing the version number in a
  package-info.java @Version annotation for the first change of a
given level
  for each release.
  - Advantages:
 - most precise package versions (obviously :-).
 - Avoids disadvantages in (2)
 - Lays groundwork for use of other package annotations
- e.g.:  @Export annotation to indicate package is for export,
with other packages considered private (could also be used
for jig-slaw)
 - Disadvantages:
 - Requires updating one or more version number  per release,
 either manually or automatically
 - If NOT (1), OSGI systems that make use of pax mvn / aether URLs
  (e.g. Apache Kafka ) may not work optimally (if a repository index is
 used, this is not a problem)
  4. Automatically incrementing per-package versions:
  - Advantages:
 - Does not require developer to edit package-info.java @Version
 annotation.
  - Disadvantages:
 - Does not require developer to edit package-info.java @Version
 annotation.  "*Wait - that's not backwards compatible*?"
 - Requires some Mojo (most suitable place would be in Apache Felix
 maven-bundle-plugin).
  5. Automatically initializing per-package versions:
  - Advantages:
 - Avoids having to manually create or edit a large number of
 packages. Allows a uniform policy to be set.
  - Disadvantages:
 - Requires a small amount of code (could be new goal, could be
 groovy or bash/perl script).

A big part of the decision comes down to how to handle breaking changes.
If there is a policy to reduce unintended API changes, then manually
bumping package versions if the API change is required is probably best.  A
failing  bundle:baseline can be thought of like a failing unit  

Re: OSGI Version at Package Level : some more details about the COMPRESS-399 pull request

[Simon Spero]

On Sun, Jun 11, 2017 at 3:16 PM, Gary Gregory 
wrote:

> My POV is that I only see a possible use cases for this in multi-module
> components where one module defines an API and others different
> implementation. Our release process is enough of a pain. Asking everyone
> to consider if a change warrants a package version change is a pain. I still
> do not see what practical and concrete problem this would solve for a
> single module component. Granted COMPRESSA defines an API and
> implementations all is one jar. But we work for 100% BC within a minor
> release, so no problem there right? For BC breaks, we use a new version and
> new package name, so no problem either, right?
>
> Can you show us a real problem that this would have solved? If not, write
> up a realistic imaginary problem?
>

See: e.g http://www.sciencedirect.com/science/article/pii/S1571066111000399

Note that the specific versions of *org.apache.commons.fileupload *are not
completely on point, since as far as I know the first version of
commons-fileupload to include bundle headers was 1.2.1

However, we don't have to go much further to find more examples in that
series.
The bundles org.apache.commons.fileupload , version 1.2.*1*, and
 org.apache.commons.fileupload, version 1.2.*2*  use the  same  version
numbers for all packages, are two *micro* (aka bug-fix) version numbers
within the same micro version.  Under  semantic versioning rules, micro
versions must not have any API changes.

However, between those 1.2.1 and 1.2.2, there are *minor* level changes to
two of the five packages (the other three remain unchanged.
Moving from version 1.2.2 to version 1.3.0,  there are *major* breaking
binary changes to four packages (meaning they should have version 2.0.0).
>From version 1.3.0 to 1.3.1 there are minor (backwards compatible changes)
to one package, with the other four having no API changes.
>From 1.3.1 to 1.3.2 has no API changes, so is the micro version bump is
correct.

 Looking at commons-math, there were major breaking changes to 6 packages
between versions  2.0 and 2.1.
There were five more packages with major level changes between 2.1 and
2.2.  This was the second set of breaking changes for three of  of these
packages; their correct version number should have been 4.0.   Note that
this is *before* the packages prefix got changes to org.apache.commons.math3

The nature of the major changes in commons-math (mostly changing the return
types of methods) suggests that the developers might have used a different
approach rather than breaking binary compatibility.  To me, it seems that
automatically bumping the major version would have been the wrong response.


If you like, I can post a list of what the package version should have
been, assuming that every compatibility change was intentional, and
versioning was properly applied,  for  every bundle series under
org.apache.commons and commons-* that was on maven-central as of mid-may?

Simon

Re: OSGI Version at Package Level : some more details about the COMPRESS-399 pull request

[Simon Spero]

So what's the current thinking on using compatibility-verified package
versioning  (for commons-* in general, but  COMPRESS-399 in particular?)

It doesn't take much work, and incorrect package versions cause real
problems.

*You added the headers and here I am. You tell 'em I'M coming... and Jar
Hell's coming with me, you hear?  **Jar Hell's coming with me!*


Simon

On Wed, Jun 7, 2017 at 5:02 PM, Gary Gregory <garydgreg...@gmail.com> wrote:

> On Wed, Jun 7, 2017 at 10:28 AM, Simon Spero <sesunc...@gmail.com> wrote:
>
> > As Bertrand mentioned earlier, the bundle:baseline goal  is built in to
> the
> > maven-bundle-plugin already in use!
> >
> > The pull-request adds that goal to the verify phase of the maven build
> (and
> > changes the travis config to run 'mvn verify'  instead of 'mvn test' ).
> The
> > baseline goal is set to fail the build if the versioning contains errors.
> >
>
> We should run mvn verify anyway to pick up any other checks like
> integration tests.
>
> Gary
>
>
> >
> > It takes very little work to bump the package version numbers when an API
> > change is made.  The first time you change a package in a way that
> requires
> > a new version number, the build will simply break in the verify phase,
> with
> > a list of changes, and a suggested new version number.
> >
> > Any further API changes to the package at the same level or below will
> pass
> > the verify stage without a problem. For example, if you have already
> bumped
> > the package by a minor version, any further minor changes won't require
> any
> > more changes.
> >
> > However, a subsequent major (breaking) changes will fail when verifying.
> > The author of the change can then consider whether the breaking change is
> > the best way forward, or whether their goals can be achieved in different
> > way.Studies have shown that there is considerable confusion about what is
> > and isn't a binary compatible change in java (for example, changing the
> > return type of a method from Collection to List is a
> > binary-incompatible change (but is source-compatible).  Breaking changes
> > need careful consideration.
> >
> > In principle, the major and minor components of the bundle & artifact
> > version should be bumped to match the most significant change in any
> > package in the bundle/artifact, so that maven version-range dependencies
> > don't break, but that ship has sunk.
> >
> > Simon
> > p.s.
> > The reason I'd been making changes to commons-compress was to support
> > creating and using random access tar.xz files in order to store release
> > series of  bundles from maven-central to investigate this issue (and see
> > how many problems can be fixed statically, and how many by dynamic
> > rewriting and/or fibbing.
> > p.p.s.
> > Compressing series of maven artifact is quite interesting ;
> >  for example, the ten releases of common-math3 (to 3.6) contain 40.9 MiB
> >  of uncompressed contents:
> >
> > 17.8 MiB as jars.
> >   6.4 MiB when the jars are packed in version order in .tar.xz format,
> >   4.4 MiB if the jars are run through pack200 and xz'ed individually.
> >   1.6 MiB if the compressed entries of the jar files are reflated first.
> >   1.5 MiB if run through pack200 then tar.xz
> >
>

Re: Testing enhanced @Deprecation vs. jdk 8

[Simon Spero]

Does Animal Sniffer check this? It wasn't clear from eyeballing the code if
it paid much attention to annotations apart from its own, but it all
depends on a possible  twisty maze of delegation in as, so I'm uncertain.

On Jun 9, 2017 11:55 AM, "Gary Gregory" <garydgreg...@gmail.com> wrote:

> That would require that the animal sniffer plugin be setup for components
> that do want to go that way. Doable if there is a sig file for the right
> Java version for that component.
>
> Gary
>
> On Jun 9, 2017 8:13 AM, "Simon Spero" <sesunc...@gmail.com> wrote:
>
> > On Thu, Jun 8, 2017 at 12:19 PM, Gary Gregory <garydgreg...@gmail.com>
> > wrote:
> >
> > > > 3. JDK9 adds some extra parameters to the Deprecated annotation (most
> > > notably  forRemoval=true, which is used to indicate that the annotated
> > item
> > >  is really really deprecated.)   It's not needed in this case, but is
> > > worth thinking about  when jdk9 is eventually released (latest schedule
> > > change : from 7/27/2017 to 9/21/2017).
> > >
> > > I do not think we plan on making Java 9 a requirement for any current
> > > project.
> >
> >
> > I just double checked, and it does seem that the jdk-9 dependency is at
> > library compile time.  Compiling a class with an enhanced deprecation
> > annotation, using the jdk-9 compiler, but with source & target 1.8, and
> > then compiling another class that uses the deprecated class works, and
> > gives the expected messages.
> >
> > Using -Xlint:deprecation shows the detailed deprecation warning.  Using
> > plain -Xlint adds some extra warnings about the ignored annotation bits
> > which require -Xlint:-classfile to disable.
> >
> > Compiling the second class with jdk9 issues the new "removal" warning
> (with
> > -Xlint:-removal required to downgrade the warning to a note).
> >
> > None of this is particularly urgent, but if/when jdk9 is released, it
> might
> > be worth considering
> >
> > Simon
> >
> > ses@snarkive$ javac -version
> > *javac 1.8.0_131*
> >
> > ses@snarkive$ javac  -cp build/classes/main
> depup/src/EyeOfTheWeasel.java
> > *Note: depup/src/EyeOfTheWeasel.java uses or overrides a deprecated API.*
> > *Note: Recompile with -Xlint:deprecation for details.*
> >
> > ses@snarkive$ javac -Xlint:deprecation -cp build/classes/main
> > depup/src/EyeOfTheWeasel.java
> > *depup/src/EyeOfTheWeasel.java:7: warning: [deprecation] Weasel in
> unnamed
> > package has been deprecated*
> > *Weasel.main(new String[] {"UTF-8"});*
> >
> > ses@snarkive$ javac -Xlint -cp build/classes/main
> > depup/src/EyeOfTheWeasel.java
> > *build/classes/main/Weasel.class: warning: Cannot find annotation method
> > 'since()' in type 'Deprecated'*
> > *build/classes/main/Weasel.class: warning: Cannot find annotation method
> > 'forRemoval()' in type 'Deprecated'*
> > *depup/src/EyeOfTheWeasel.java:7: warning: [deprecation] Weasel in
> unnamed
> > package has been deprecated*
> > *Weasel.main(new String[] {"UTF-8"});*
> > *^*
> > *3 warnings*
> >
> > ses@snarkive$ javac -Xlint  -Xlint:-classfile -cp build/classes/main
> > depup/src/EyeOfTheWeasel.java
> > depup/src/EyeOfTheWeasel.java:7: warning: [deprecation] Weasel in
> unnamed
> > package has been deprecated
> > Weasel.main(new String[] {"UTF-8"});
> > ^
> > 1 warning
> >
> > ses@snarkive$ java  -cp build/classes/main/:depup/src/:build/classes
> > EyeOfTheWeasel
> > Emoji Lovin' Hippies!
> > annotation = @java.lang.Deprecated()
> >
> > ses@snarkive$ /usr/lib/jvm/openjdk-9-internal/bin/javac  -cp
> > build/classes/main depup/src/EyeOfTheWeasel.java
> > *depup/src/EyeOfTheWeasel.java:7: warning: [removal] Weasel in unnamed
> > package has been deprecated and marked for removal*
> > *Weasel.main(new String[] {"UTF-8"});*
> > *^*
> > *1 warning*
> >
>

Testing enhanced @Deprecation vs. jdk 8

[Simon Spero]

On Thu, Jun 8, 2017 at 12:19 PM, Gary Gregory 
wrote:

> > 3. JDK9 adds some extra parameters to the Deprecated annotation (most
> notably  forRemoval=true, which is used to indicate that the annotated item
>  is really really deprecated.)   It's not needed in this case, but is
> worth thinking about  when jdk9 is eventually released (latest schedule
> change : from 7/27/2017 to 9/21/2017).
>
> I do not think we plan on making Java 9 a requirement for any current
> project.


I just double checked, and it does seem that the jdk-9 dependency is at
library compile time.  Compiling a class with an enhanced deprecation
annotation, using the jdk-9 compiler, but with source & target 1.8, and
then compiling another class that uses the deprecated class works, and
gives the expected messages.

Using -Xlint:deprecation shows the detailed deprecation warning.  Using
plain -Xlint adds some extra warnings about the ignored annotation bits
which require -Xlint:-classfile to disable.

Compiling the second class with jdk9 issues the new "removal" warning (with
-Xlint:-removal required to downgrade the warning to a note).

None of this is particularly urgent, but if/when jdk9 is released, it might
be worth considering

Simon

ses@snarkive$ javac -version
*javac 1.8.0_131*

ses@snarkive$ javac  -cp build/classes/main depup/src/EyeOfTheWeasel.java
*Note: depup/src/EyeOfTheWeasel.java uses or overrides a deprecated API.*
*Note: Recompile with -Xlint:deprecation for details.*

ses@snarkive$ javac -Xlint:deprecation -cp build/classes/main
depup/src/EyeOfTheWeasel.java
*depup/src/EyeOfTheWeasel.java:7: warning: [deprecation] Weasel in unnamed
package has been deprecated*
*Weasel.main(new String[] {"UTF-8"});*

ses@snarkive$ javac -Xlint -cp build/classes/main
depup/src/EyeOfTheWeasel.java
*build/classes/main/Weasel.class: warning: Cannot find annotation method
'since()' in type 'Deprecated'*
*build/classes/main/Weasel.class: warning: Cannot find annotation method
'forRemoval()' in type 'Deprecated'*
*depup/src/EyeOfTheWeasel.java:7: warning: [deprecation] Weasel in unnamed
package has been deprecated*
*Weasel.main(new String[] {"UTF-8"});*
*^*
*3 warnings*

ses@snarkive$ javac -Xlint  -Xlint:-classfile -cp build/classes/main
depup/src/EyeOfTheWeasel.java
depup/src/EyeOfTheWeasel.java:7: warning: [deprecation] Weasel in unnamed
package has been deprecated
Weasel.main(new String[] {"UTF-8"});
^
1 warning

ses@snarkive$ java  -cp build/classes/main/:depup/src/:build/classes
EyeOfTheWeasel
Emoji Lovin' Hippies!
annotation = @java.lang.Deprecated()

ses@snarkive$ /usr/lib/jvm/openjdk-9-internal/bin/javac  -cp
build/classes/main depup/src/EyeOfTheWeasel.java
*depup/src/EyeOfTheWeasel.java:7: warning: [removal] Weasel in unnamed
package has been deprecated and marked for removal*
*Weasel.main(new String[] {"UTF-8"});*
*^*
*1 warning*

Re: [VOTE] Release Apache Commons Lang 3.6 based on RC3

[Simon Spero]

There is a one other compatibility issue, which can be seen in the attached
code:

import java.nio.charset.StandardCharsets;

public class Weasel {

private static final String US_ASCII = "US-ASCII";
private static final String UTF_8 = "UTF-8";
private static final String STANDARD_US_ASCII =
StandardCharsets.US_ASCII.name();
private static final String STANDARD_UTF_8 = StandardCharsets.UTF_8.name
();

public static void main(String[] args) {

switch (args[0]) {
case US_ASCII:
System.out.println("USA! USA!");
break;
case UTF_8:
System.out.println("Emoji Lovin' Hippies!");
break;
default:
System.out.println("Weirdo.");
}
}
}


This code compiles.
However, if the case labels are changed to STANDARD_US_ASCII and
STANDARD_UTF_8, the compilation will fail, because the  case labels are no
longer constant expressions.
In the actual code, this means that code compiled against an older version
of the library would work against the new code (because the old strings had
already been inlined), but could not be re-compiled.

I don't know why anyone would be doing this...

 (CLIRR pre-dates string switches)

Simon

On Thu, Jun 8, 2017 at 5:10 PM, sebb <seb...@gmail.com> wrote:

> On 8 June 2017 at 18:09, Gary Gregory <garydgreg...@gmail.com> wrote:
> > On Thu, Jun 8, 2017 at 9:57 AM, sebb <seb...@gmail.com> wrote:
> >
> >> On 8 June 2017 at 17:19, Gary Gregory <garydgreg...@gmail.com> wrote:
> >> > On Thu, Jun 8, 2017 at 5:38 AM, Simon Spero <sesunc...@gmail.com>
> wrote:
> >> >
> >> >> [A Note, not a vote :) ]
> >> >>
> >> >> 1. Clirr is generally considered obsolete, as it hadn't been worked
> on
> >> for
> >> >> about ten years.   japicmp is a good replacement, especially for
> report
> >> >> generation, and is used in other commons projects.
> >> >>
> >> >
> >> > IIRC, we've started using japicm here and there. Each component can
> >> decide.
> >> > But yes, Clirr looks pretty dead.
> >> >
> >> >
> >> >> 2. Are the "changes" to the values in CharEncoding really
> necessary[1]
> >> (The
> >> >> deprecation surely is). Technically this is a potentially breaking
> >> binary
> >> >> incompatible change, as constant strings and primitives are inlined
> at
> >> >> compile time. [2]
> >> >> In this particular case, the results of load-time evaluation of the
> new
> >> >> initialization expressions  are identical to the old constants, so
> it's
> >> >> behaviourally compatible; however, since this is the case, and since
> >> it's
> >> >> all deprecated anyway, why not leave the old values in-place?
> >> >>
> >> >
> >> > The changes are not "necessary" that for sure and we do get Clirr
> >> warnings:
> >> >
> >> > Value of field ISO_8859_1 is no longer a compile-time constant
> >> > Value of field US_ASCII is no longer a compile-time constant
> >> > Value of field UTF_16 is no longer a compile-time constant
> >> > Value of field UTF_16BE is no longer a compile-time constant
> >> > Value of field UTF_16LE is no longer a compile-time constant
> >> > Value of field UTF_8 is no longer a compile-time constant
> >> >
> >> > It's source compatible. What is the issue at runtime that could hurt
> >> users?
> >>
> >> As the OP wrote, constants are inlined by the compiler.
> >> So unless all code is recompiled, if it referenced the constant it may
> >> have a stale value.
> >> That is not binary compatible.
> >>
> >
> > But in this case the actual values are the same are they not? So what is
> > the difference? Would this only be a problem if we changed the string
> > values?
>
> AFAIK the compiler only inlines compile-time constants.
> So going forward, the values won't be inlined.
> I assume the behaviour will be unaffected since the runtime value
> should be the same as the constant.
>
> The release notes really ought to explain why the Clirr report items
> aren't a problem.
>
> > Which we can't since these are defined in the JRE. And the JRE is
> > unlikely to change those.
> >
> > Gary
> >
> >
> >>
> >> >
> >> >> 3. JDK9 adds some extra parameters to the Depreca

Re: [VOTE] Release Apache Commons Lang 3.6 based on RC3

[Simon Spero]

[A Note, not a vote :) ]

1. Clirr is generally considered obsolete, as it hadn't been worked on for
about ten years.   japicmp is a good replacement, especially for report
generation, and is used in other commons projects.

2. Are the "changes" to the values in CharEncoding really necessary[1] (The
deprecation surely is). Technically this is a potentially breaking binary
incompatible change, as constant strings and primitives are inlined at
compile time. [2]
In this particular case, the results of load-time evaluation of the new
initialization expressions  are identical to the old constants, so it's
behaviourally compatible; however, since this is the case, and since  it's
all deprecated anyway, why not leave the old values in-place?

3. JDK9 adds some extra parameters to the Deprecated annotation (most
notably  forRemoval=true, which is used to indicate that the annotated item
is really really deprecated.)   It's not needed in this case, but is worth
thinking about  when jdk9 is eventually released (latest schedule change :
from 7/27/2017 to 9/21/2017).

Simon

[1]  https://github.com/apache/commons-lang/commit/7c19a1ff4c217
f03c0be62baf1169d689f566825#diff-820a48456e11e85bf6cf5356c1aed4baR48

[2] https://docs.oracle.com/javase/specs/jls/se8/html/jls-13.html#jls-13.4.9

On Jun 8, 2017 4:48 AM, "Benedikt Ritter"  wrote:

> Hello,
>
> we have fixed quite a few bugs and added some nice new features since
> Commons Lang 3.5 was released, so I would like to release Commons Lang 3.6
> based on RC3.
> The following issues have been fixed since RC2:
>
> - Site build now works from source distribution
> - IBM JDK test failures have been fixed
> - Automatic-Module-Name MANIFEST entry has been added for Java 9
> compatibility
>
> Commons Lang 3.6 R3 is available for review here:
>  https://dist.apache.org/repos/dist/dev/commons/lang (svn revision 19928)
>
> The tag is here:
> https://git-wip-us.apache.org/repos/asf?p=commons-lang.git;a=tag;h=
> e454e79463ffbbd9114db43019dd211debbcc105
>
> Commit ID the tag points at:
>  360198dfb6a2d68f1702f616dfacee34ae0541bb
>
> Maven Artifacts:
>  https://repository.apache.org/content/repositories/orgapachecommons-1250
>
> These are the Maven artifacts and their hashes:
>
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6-javadoc.jar
> (SHA1: c8adadb6c0b56c73f2cc2b4c77a09bfe34ec3a66)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6-sources.jar.asc
> (SHA1: 46347c179ca9246d146d653bdc7363bda6f17d44)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6.pom.asc
> (SHA1: 1309d4f3dd41a01ff9dd1f3c1a6eee10dad1ef77)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6.pom
> (SHA1: 0fb4499188c94c63b3cba44f12481e193708c4a8)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6.jar.asc
> (SHA1: e67e7d44751f1e346c2fda496193cbe251cfe098)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6-javadoc.jar.asc
> (SHA1: 6b19fa12d319ced69c0f8a27001569514711f9dc)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6-sources.jar
> (SHA1: f89c1df082d6f67cb7c956715c56d7e7a0508d0a)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6.jar
> (SHA1: e58ba08b01d37a023746f0987dcd910036a63021)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6-tests.jar.asc
> (SHA1: af050e8c29a801a5d6783268c56230b814f56240)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6-
> test-sources.jar.asc
> (SHA1: 71e4c11efb9e3b2eff402ba4648d21822fb8da4a)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6-test-sources.jar
> (SHA1: 04a0fc8293d4ed64a54dcc9ba5f996776a4657ea)
> /org/apache/commons/commons-lang3/3.6/commons-lang3-3.6-tests.jar
> (SHA1: 87993a16c14a251062e3fe860fa53b5ef5304a0f)
>
> I have tested this with JDK 7, JDK 8 and JDK 9 EA b172 using Maven 3.5.0.
>
> Details of changes since 3.5 are in the release notes:
>https://dist.apache.org/repos/dist/dev/commons/lang/RELEASE-NOTES.txt
>http://home.apache.org/~britter/commons/lang/LANG_3_6_
> RC3/changes-report.html
>
> Site:
>  http://home.apache.org/~britter/commons/lang/LANG_3_6_RC3
>  (note some *relative* links are broken and the 3.6 directories are
>  not yet created - these will be OK once the site is deployed)
>
> Clirr Report (compared to 3.5):
>http://home.apache.org/~britter/commons/lang/LANG_3_6_
> RC3/clirr-report.html
>
> RAT Report:
>http://home.apache.org/~britter/commons/lang/LANG_3_6_
> RC3/rat-report.html
>
> KEYS:
>  https://www.apache.org/dist/commons/KEYS
>
> Please review the release candidate and vote.
> This vote will close no sooner that 72 hours from now,
> i.e. sometime after 11:00 CEST (UTC+2) 11-June 2017
>
>  [ ] +1 Release these artifacts
>  [ ] +0 OK, but...
>  [ ] -0 OK, but really should fix...
>  [ ] -1 I oppose this release because...
>
> Thanks!
> Benedikt
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: 

OSGI Version at Package Level : some more details about the COMPRESS-399 pull request

[Simon Spero]

As Bertrand mentioned earlier, the bundle:baseline goal  is built in to the
maven-bundle-plugin already in use!

The pull-request adds that goal to the verify phase of the maven build (and
changes the travis config to run 'mvn verify'  instead of 'mvn test' ). The
baseline goal is set to fail the build if the versioning contains errors.

It takes very little work to bump the package version numbers when an API
change is made.  The first time you change a package in a way that requires
a new version number, the build will simply break in the verify phase, with
a list of changes, and a suggested new version number.

Any further API changes to the package at the same level or below will pass
the verify stage without a problem. For example, if you have already bumped
the package by a minor version, any further minor changes won't require any
more changes.

However, a subsequent major (breaking) changes will fail when verifying.
The author of the change can then consider whether the breaking change is
the best way forward, or whether their goals can be achieved in different
way.Studies have shown that there is considerable confusion about what is
and isn't a binary compatible change in java (for example, changing the
return type of a method from Collection to List is a
binary-incompatible change (but is source-compatible).  Breaking changes
need careful consideration.

In principle, the major and minor components of the bundle & artifact
version should be bumped to match the most significant change in any
package in the bundle/artifact, so that maven version-range dependencies
don't break, but that ship has sunk.

Simon
p.s.
The reason I'd been making changes to commons-compress was to support
creating and using random access tar.xz files in order to store release
series of  bundles from maven-central to investigate this issue (and see
how many problems can be fixed statically, and how many by dynamic
rewriting and/or fibbing.
p.p.s.
Compressing series of maven artifact is quite interesting ;
 for example, the ten releases of common-math3 (to 3.6) contain 40.9 MiB
 of uncompressed contents:

17.8 MiB as jars.
  6.4 MiB when the jars are packed in version order in .tar.xz format,
  4.4 MiB if the jars are run through pack200 and xz'ed individually.
  1.6 MiB if the compressed entries of the jar files are reflated first.
  1.5 MiB if run through pack200 then tar.xz