Re: [rng] Releasing 1.3

[Rob Tompkins]

I would cd into ./target/commons-release-plugin/scm and try the svn commit from 
there manually. That’s the directory where everything gets staged.

I also think your svn password gets cached generally, but that’s predicated on 
having checked something into svn previously. Given our recent migration to all 
git based projects i could see that not working the first time. You can also 
pass “-Duser.password=...” at the command step, but that’s ugly in my opinion 
and can be avoided by putting that password down in your settings.xml. 

Keep me posted and I’ll try to answer quickly,
-Rob 

> On Nov 4, 2019, at 2:31 PM, Alex Herbert  wrote:
> 
> 
> 
>> On 4 Nov 2019, at 18:38, Rob Tompkins  wrote:
>> 
>> Also keep me posted. I’ll be near my keyboard all day. -Rob
>> 
 On Nov 4, 2019, at 1:21 PM, Rob Tompkins  wrote:
>>> 
>>> 
>>> 
 On Nov 4, 2019, at 1:16 PM, Alex Herbert  wrote:
 
 
>> On 04/11/2019 17:36, Alex Herbert wrote:
> 
> 
>> On 04/11/2019 15:40, Alex Herbert wrote:
>> 
>> On 04/11/2019 12:43, Gilles Sadowski wrote:
>>> Hello.
>>> 
>>> Le lun. 4 nov. 2019 à 13:05, Alex Herbert  a 
>>> écrit :
 
 On 04/11/2019 00:42, Gilles Sadowski wrote:
> Hi.
> 
>> [...]
>> 
>> So there are at least two options:
>> 
>> 1. Release using JDK 9
> +1
 OK. I'll run through the release with JDK 9. I do not think we have any
 NIO classes so the issue raised by Gary should not be present.
>>> Here
>>> $ JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 mvn
>>> -Pcommons-rng-examples clean package install site site:stage
>>> built fine (but I did not check the reports).
>> 
>> Good to know.
>> 
>> I've been through a dry run and the outputs look as expected.
> 
> The commons-release-plugin:1.7 has failed on the final step to stage the 
> files in Nexus.
> 
> All the files appear to be present but the plugin fails when processing 
> the parent for the multi-module project.
> 
> Here is the command that is currently failing:
> 
> mvn -X -e -Duser.name=aherbert -Pcommons-rng-examples -Prelease clean 
> package site site:stage deploy -rf :commons-rng
> 
> [DEBUG] -- end configuration --
> [INFO] Checking out dist from: 
> scm:svn:https://dist.apache.org/repos/dist/dev/commons/rng
> Executing: /bin/sh -c cd 
> '/scratch/commons-rng/dist-archive/target/commons-release-plugin' && 
> 'svn' '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 
> 'checkout' 'https://dist.apache.org/repos/dist/dev/commons/rng@' 
> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
> Executing: /bin/sh -c cd 
> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>  && 'svn' 'remove' '--targets' 
> '/tmp/maven-scm-5448861436738372728-targets'
> [INFO] Cleaning distribution area for: commons-rng
> Executing: /bin/sh -c cd 
> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>  && 'svn' '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 
> 'commit' '--file' '/tmp/maven-scm-1469264231.commit' '--encoding' 'UTF-8' 
> '--targets' '/tmp/maven-scm-14546727756421455038-targets'
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time:  7.376 s
> [INFO] Finished at: 2019-11-04T17:19:15Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.commons:commons-release-plugin:1.7:clean-staging 
> (clean-staging) on project commons-rng: Failed to commit files: null 
> [null] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.commons:commons-release-plugin:1.7:clean-staging 
> (clean-staging) on project commons-rng: Failed to commit files: null 
> [null]
>  at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:215)
>  at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:156)
>  at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:148)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
>  at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
>  at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:56)
>  at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> 

Re: [Numbers] Arrays of "Complex" objects and RAM

[Eric Barnhill]

That's interesting. The JTransforms library performs Fourier transforms
that can take complex input, output, or both. They do this with interleaved
double[] arrays, which I suppose is much more space efficient, and the
status of a number as real or imaginary is implicit by its location being
odd or even.

The MultidimensionalCounter functionality you mention is for example known
in Matlab as ind2sub() and sub2ind() . It allows for 1d vectorizing of
certain functions which can improve performance. Some people swear by it. I
always found it an additional mental exercise that I didn't want to occupy
myself with unless I absolutely had to. So, maybe it makes sense to
"advertise" this approach like you say, but some users may just want the 3D
arrays for more rapid prototyping-ish applications.

I wonder if there isn't some streaming solution for this -- the numbers are
stored as an interleaved 1D array, but are streamed through a Complex
constructor before any needed operations are performed.

And I guess that leads to my last question -- suppose someone wants to call
a trig function on a series of Complex numbers. Now let's imagine the
primitives have been stored in some maximally efficient way. It seems like,
to use any of the functionality in Complex, these numbers would have to be
unpacked, cast as Complex, operated on, then cast back to how they are
being stored. I wonder if that would prove to be more efficient in the end.

On Sat, Nov 2, 2019 at 7:14 PM Gilles Sadowski  wrote:

> Hello.
>
> The class "ComplexUtils" deal with multi-dimensional arrays that hold
> instances of the "Complex" class.
> I've recently encountered a use-case where it was pointed out that storing
> many "Complex" instances (as seems the purpose of these utilities) is quite
> inefficient memory-wise as each instance will take 32 bytes[1] while the
> real and imaginary parts would only take 16 bytes as 2 primitive "double"s.
> This is compounded by multi-dimensional array where each sub-dimensional
> element is an array object (and thus takes another additional 16 bytes).
> For example, a
> double[10][5][4]
> would take
> 16 * (1 + 10 * 5) + 10 * 5 * 4 * 8
>   = 2416 bytes.
> Assuming that in the above array, the last dimension holds 2 complex
> numbers,
> the same data can be represented as
> Complex[10][5][2]
> that would take
> 16 * ((1 + 10 * 5) + (10 * 5 * 2)) + 10 * 5 * 2 * 2 * 8
>   = 4016 bytes.
> In both cases, the payload (10 * 5 * 2 complex numbers) is
> 10 * 5 * 2 * 2 * 8
>   = 1600 bytes.
> If stored in a one-dimensional array, the size in memory would be 1616
> bytes.
> Thus in the case of a data cube holding 100 complex numbers, a 3D array
> takes 1.5 (primitives) or 2.5 ("Complex" objects) more memory than a 1D
> array.
> If this is correct, I wonder whether we should advertise such a
> "ComplexUtils"
> class.  It would perhaps be preferable to propose a data cube
> abstraction.[2]
>
> WDYT?
>
> Regards,
> Gilles
>
> [1] https://www.baeldung.com/java-size-of-object
> [2] Based on
> http://commons.apache.org/proper/commons-math/apidocs/org/apache/commons/math4/util/MultidimensionalCounter.html
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
> For additional commands, e-mail: dev-h...@commons.apache.org
>
>

Re: [rng] Releasing 1.3

[Alex Herbert]

> On 4 Nov 2019, at 19:31, Alex Herbert  wrote:
> 
> 
> 
>> On 4 Nov 2019, at 18:38, Rob Tompkins > > wrote:
>> 
>> Also keep me posted. I’ll be near my keyboard all day. -Rob
>> 
>>> On Nov 4, 2019, at 1:21 PM, Rob Tompkins >> > wrote:
>>> 
>>> 
>>> 
 On Nov 4, 2019, at 1:16 PM, Alex Herbert >>> > wrote:
 
 
>> On 04/11/2019 17:36, Alex Herbert wrote:
> 
> 
>> On 04/11/2019 15:40, Alex Herbert wrote:
>> 
>> On 04/11/2019 12:43, Gilles Sadowski wrote:
>>> Hello.
>>> 
>>> Le lun. 4 nov. 2019 à 13:05, Alex Herbert >> > a écrit :
 
 On 04/11/2019 00:42, Gilles Sadowski wrote:
> Hi.
> 
>> [...]
>> 
>> So there are at least two options:
>> 
>> 1. Release using JDK 9
> +1
 OK. I'll run through the release with JDK 9. I do not think we have any
 NIO classes so the issue raised by Gary should not be present.
>>> Here
>>>  $ JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 mvn
>>> -Pcommons-rng-examples clean package install site site:stage
>>> built fine (but I did not check the reports).
>> 
>> Good to know.
>> 
>> I've been through a dry run and the outputs look as expected.
> 
> The commons-release-plugin:1.7 has failed on the final step to stage the 
> files in Nexus.
> 
> All the files appear to be present but the plugin fails when processing 
> the parent for the multi-module project.
> 
> Here is the command that is currently failing:
> 
> mvn -X -e -Duser.name=aherbert -Pcommons-rng-examples -Prelease clean 
> package site site:stage deploy -rf :commons-rng
> 
> [DEBUG] -- end configuration --
> [INFO] Checking out dist from: 
> scm:svn:https://dist.apache.org/repos/dist/dev/commons/rng 
> 
> Executing: /bin/sh -c cd 
> '/scratch/commons-rng/dist-archive/target/commons-release-plugin' && 
> 'svn' '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 
> 'checkout' 'https://dist.apache.org/repos/dist/dev/commons/rng@' 
> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup
>  
> '
> Executing: /bin/sh -c cd 
> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>  && 'svn' 'remove' '--targets' 
> '/tmp/maven-scm-5448861436738372728-targets'
> [INFO] Cleaning distribution area for: commons-rng
> Executing: /bin/sh -c cd 
> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>  && 'svn' '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 
> 'commit' '--file' '/tmp/maven-scm-1469264231.commit' '--encoding' 'UTF-8' 
> '--targets' '/tmp/maven-scm-14546727756421455038-targets'
> [INFO] 
> 
> [INFO] BUILD FAILURE
> [INFO] 
> 
> [INFO] Total time:  7.376 s
> [INFO] Finished at: 2019-11-04T17:19:15Z
> [INFO] 
> 
> [ERROR] Failed to execute goal 
> org.apache.commons:commons-release-plugin:1.7:clean-staging 
> (clean-staging) on project commons-rng: Failed to commit files: null 
> [null] -> [Help 1]
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
> goal org.apache.commons:commons-release-plugin:1.7:clean-staging 
> (clean-staging) on project commons-rng: Failed to commit files: null 
> [null]
>   at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:215)
>   at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:156)
>   at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
> (MojoExecutor.java:148)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:117)
>   at 
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
> (LifecycleModuleBuilder.java:81)
>   at 
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>  (SingleThreadedBuilder.java:56)
>   at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
> (LifecycleStarter.java:128)
>   at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
>   at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
>   at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
>   

Re: [rng] Releasing 1.3

[Alex Herbert]

> On 4 Nov 2019, at 18:38, Rob Tompkins  wrote:
> 
> Also keep me posted. I’ll be near my keyboard all day. -Rob
> 
>> On Nov 4, 2019, at 1:21 PM, Rob Tompkins  wrote:
>> 
>> 
>> 
>>> On Nov 4, 2019, at 1:16 PM, Alex Herbert  wrote:
>>> 
>>> 
> On 04/11/2019 17:36, Alex Herbert wrote:
 
 
> On 04/11/2019 15:40, Alex Herbert wrote:
> 
> On 04/11/2019 12:43, Gilles Sadowski wrote:
>> Hello.
>> 
>> Le lun. 4 nov. 2019 à 13:05, Alex Herbert  a 
>> écrit :
>>> 
>>> On 04/11/2019 00:42, Gilles Sadowski wrote:
 Hi.
 
> [...]
> 
> So there are at least two options:
> 
> 1. Release using JDK 9
 +1
>>> OK. I'll run through the release with JDK 9. I do not think we have any
>>> NIO classes so the issue raised by Gary should not be present.
>> Here
>>  $ JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 mvn
>> -Pcommons-rng-examples clean package install site site:stage
>> built fine (but I did not check the reports).
> 
> Good to know.
> 
> I've been through a dry run and the outputs look as expected.
 
 The commons-release-plugin:1.7 has failed on the final step to stage the 
 files in Nexus.
 
 All the files appear to be present but the plugin fails when processing 
 the parent for the multi-module project.
 
 Here is the command that is currently failing:
 
 mvn -X -e -Duser.name=aherbert -Pcommons-rng-examples -Prelease clean 
 package site site:stage deploy -rf :commons-rng
 
 [DEBUG] -- end configuration --
 [INFO] Checking out dist from: 
 scm:svn:https://dist.apache.org/repos/dist/dev/commons/rng
 Executing: /bin/sh -c cd 
 '/scratch/commons-rng/dist-archive/target/commons-release-plugin' && 'svn' 
 '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 'checkout' 
 'https://dist.apache.org/repos/dist/dev/commons/rng@' 
 '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
 Executing: /bin/sh -c cd 
 '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
  && 'svn' 'remove' '--targets' '/tmp/maven-scm-5448861436738372728-targets'
 [INFO] Cleaning distribution area for: commons-rng
 Executing: /bin/sh -c cd 
 '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
  && 'svn' '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 
 'commit' '--file' '/tmp/maven-scm-1469264231.commit' '--encoding' 'UTF-8' 
 '--targets' '/tmp/maven-scm-14546727756421455038-targets'
 [INFO] 
 
 [INFO] BUILD FAILURE
 [INFO] 
 
 [INFO] Total time:  7.376 s
 [INFO] Finished at: 2019-11-04T17:19:15Z
 [INFO] 
 
 [ERROR] Failed to execute goal 
 org.apache.commons:commons-release-plugin:1.7:clean-staging 
 (clean-staging) on project commons-rng: Failed to commit files: null 
 [null] -> [Help 1]
 org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
 goal org.apache.commons:commons-release-plugin:1.7:clean-staging 
 (clean-staging) on project commons-rng: Failed to commit files: null [null]
   at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
 (MojoExecutor.java:215)
   at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
 (MojoExecutor.java:156)
   at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
 (MojoExecutor.java:148)
   at 
 org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
 (LifecycleModuleBuilder.java:117)
   at 
 org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
 (LifecycleModuleBuilder.java:81)
   at 
 org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
  (SingleThreadedBuilder.java:56)
   at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
 (LifecycleStarter.java:128)
   at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
   at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
   at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
   at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
   at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
   at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
   at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
   at jdk.internal.reflect.NativeMethodAccessorImpl.invoke 
 (NativeMethodAccessorImpl.java:62)
   at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke 
 (DelegatingMethodAccessorImpl.java:43)
   at 

Re: [rng] Releasing 1.3

[Rob Tompkins]

Also keep me posted. I’ll be near my keyboard all day. -Rob

> On Nov 4, 2019, at 1:21 PM, Rob Tompkins  wrote:
> 
> 
> 
>> On Nov 4, 2019, at 1:16 PM, Alex Herbert  wrote:
>> 
>> 
 On 04/11/2019 17:36, Alex Herbert wrote:
>>> 
>>> 
 On 04/11/2019 15:40, Alex Herbert wrote:
 
 On 04/11/2019 12:43, Gilles Sadowski wrote:
> Hello.
> 
> Le lun. 4 nov. 2019 à 13:05, Alex Herbert  a 
> écrit :
>> 
>> On 04/11/2019 00:42, Gilles Sadowski wrote:
>>> Hi.
>>> 
 [...]
 
 So there are at least two options:
 
 1. Release using JDK 9
>>> +1
>> OK. I'll run through the release with JDK 9. I do not think we have any
>> NIO classes so the issue raised by Gary should not be present.
> Here
>   $ JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 mvn
> -Pcommons-rng-examples clean package install site site:stage
> built fine (but I did not check the reports).
 
 Good to know.
 
 I've been through a dry run and the outputs look as expected.
>>> 
>>> The commons-release-plugin:1.7 has failed on the final step to stage the 
>>> files in Nexus.
>>> 
>>> All the files appear to be present but the plugin fails when processing the 
>>> parent for the multi-module project.
>>> 
>>> Here is the command that is currently failing:
>>> 
>>> mvn -X -e -Duser.name=aherbert -Pcommons-rng-examples -Prelease clean 
>>> package site site:stage deploy -rf :commons-rng
>>> 
>>> [DEBUG] -- end configuration --
>>> [INFO] Checking out dist from: 
>>> scm:svn:https://dist.apache.org/repos/dist/dev/commons/rng
>>> Executing: /bin/sh -c cd 
>>> '/scratch/commons-rng/dist-archive/target/commons-release-plugin' && 'svn' 
>>> '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 'checkout' 
>>> 'https://dist.apache.org/repos/dist/dev/commons/rng@' 
>>> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>>> Executing: /bin/sh -c cd 
>>> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>>>  && 'svn' 'remove' '--targets' '/tmp/maven-scm-5448861436738372728-targets'
>>> [INFO] Cleaning distribution area for: commons-rng
>>> Executing: /bin/sh -c cd 
>>> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>>>  && 'svn' '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 
>>> 'commit' '--file' '/tmp/maven-scm-1469264231.commit' '--encoding' 'UTF-8' 
>>> '--targets' '/tmp/maven-scm-14546727756421455038-targets'
>>> [INFO] 
>>> 
>>> [INFO] BUILD FAILURE
>>> [INFO] 
>>> 
>>> [INFO] Total time:  7.376 s
>>> [INFO] Finished at: 2019-11-04T17:19:15Z
>>> [INFO] 
>>> 
>>> [ERROR] Failed to execute goal 
>>> org.apache.commons:commons-release-plugin:1.7:clean-staging (clean-staging) 
>>> on project commons-rng: Failed to commit files: null [null] -> [Help 1]
>>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
>>> goal org.apache.commons:commons-release-plugin:1.7:clean-staging 
>>> (clean-staging) on project commons-rng: Failed to commit files: null [null]
>>>at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
>>> (MojoExecutor.java:215)
>>>at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
>>> (MojoExecutor.java:156)
>>>at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
>>> (MojoExecutor.java:148)
>>>at 
>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
>>> (LifecycleModuleBuilder.java:117)
>>>at 
>>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
>>> (LifecycleModuleBuilder.java:81)
>>>at 
>>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>>>  (SingleThreadedBuilder.java:56)
>>>at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
>>> (LifecycleStarter.java:128)
>>>at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
>>>at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
>>>at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
>>>at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
>>>at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
>>>at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
>>>at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>>>at jdk.internal.reflect.NativeMethodAccessorImpl.invoke 
>>> (NativeMethodAccessorImpl.java:62)
>>>at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke 
>>> (DelegatingMethodAccessorImpl.java:43)
>>>at java.lang.reflect.Method.invoke (Method.java:564)
>>>at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
>>> (Launcher.java:289)
>>>at 

Re: [rng] Releasing 1.3

[Rob Tompkins]

> On Nov 4, 2019, at 1:16 PM, Alex Herbert  wrote:
> 
> 
>> On 04/11/2019 17:36, Alex Herbert wrote:
>> 
>> 
>>> On 04/11/2019 15:40, Alex Herbert wrote:
>>> 
>>> On 04/11/2019 12:43, Gilles Sadowski wrote:
 Hello.
 
 Le lun. 4 nov. 2019 à 13:05, Alex Herbert  a 
 écrit :
> 
> On 04/11/2019 00:42, Gilles Sadowski wrote:
>> Hi.
>> 
>>> [...]
>>> 
>>> So there are at least two options:
>>> 
>>> 1. Release using JDK 9
>> +1
> OK. I'll run through the release with JDK 9. I do not think we have any
> NIO classes so the issue raised by Gary should not be present.
 Here
$ JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 mvn
 -Pcommons-rng-examples clean package install site site:stage
 built fine (but I did not check the reports).
>>> 
>>> Good to know.
>>> 
>>> I've been through a dry run and the outputs look as expected.
>> 
>> The commons-release-plugin:1.7 has failed on the final step to stage the 
>> files in Nexus.
>> 
>> All the files appear to be present but the plugin fails when processing the 
>> parent for the multi-module project.
>> 
>> Here is the command that is currently failing:
>> 
>> mvn -X -e -Duser.name=aherbert -Pcommons-rng-examples -Prelease clean 
>> package site site:stage deploy -rf :commons-rng
>> 
>> [DEBUG] -- end configuration --
>> [INFO] Checking out dist from: 
>> scm:svn:https://dist.apache.org/repos/dist/dev/commons/rng
>> Executing: /bin/sh -c cd 
>> '/scratch/commons-rng/dist-archive/target/commons-release-plugin' && 'svn' 
>> '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 'checkout' 
>> 'https://dist.apache.org/repos/dist/dev/commons/rng@' 
>> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>> Executing: /bin/sh -c cd 
>> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>>  && 'svn' 'remove' '--targets' '/tmp/maven-scm-5448861436738372728-targets'
>> [INFO] Cleaning distribution area for: commons-rng
>> Executing: /bin/sh -c cd 
>> '/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
>>  && 'svn' '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 
>> 'commit' '--file' '/tmp/maven-scm-1469264231.commit' '--encoding' 'UTF-8' 
>> '--targets' '/tmp/maven-scm-14546727756421455038-targets'
>> [INFO] 
>> 
>> [INFO] BUILD FAILURE
>> [INFO] 
>> 
>> [INFO] Total time:  7.376 s
>> [INFO] Finished at: 2019-11-04T17:19:15Z
>> [INFO] 
>> 
>> [ERROR] Failed to execute goal 
>> org.apache.commons:commons-release-plugin:1.7:clean-staging (clean-staging) 
>> on project commons-rng: Failed to commit files: null [null] -> [Help 1]
>> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute 
>> goal org.apache.commons:commons-release-plugin:1.7:clean-staging 
>> (clean-staging) on project commons-rng: Failed to commit files: null [null]
>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
>> (MojoExecutor.java:215)
>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
>> (MojoExecutor.java:156)
>> at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
>> (MojoExecutor.java:148)
>> at 
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
>> (LifecycleModuleBuilder.java:117)
>> at 
>> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
>> (LifecycleModuleBuilder.java:81)
>> at 
>> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
>>  (SingleThreadedBuilder.java:56)
>> at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
>> (LifecycleStarter.java:128)
>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
>> at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
>> at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
>> at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
>> at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
>> at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
>> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
>> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke 
>> (NativeMethodAccessorImpl.java:62)
>> at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke 
>> (DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke (Method.java:564)
>> at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
>> (Launcher.java:289)
>> at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
>> (Launcher.java:229)
>> at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
>> (Launcher.java:415)
>> at 

Re: [rng] Releasing 1.3

[Alex Herbert]

On 04/11/2019 17:36, Alex Herbert wrote:



On 04/11/2019 15:40, Alex Herbert wrote:


On 04/11/2019 12:43, Gilles Sadowski wrote:

Hello.

Le lun. 4 nov. 2019 à 13:05, Alex Herbert  
a écrit :


On 04/11/2019 00:42, Gilles Sadowski wrote:

Hi.


[...]

So there are at least two options:

1. Release using JDK 9

+1
OK. I'll run through the release with JDK 9. I do not think we have 
any

NIO classes so the issue raised by Gary should not be present.

Here
   $ JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 mvn
-Pcommons-rng-examples clean package install site site:stage
built fine (but I did not check the reports).


Good to know.

I've been through a dry run and the outputs look as expected.


The commons-release-plugin:1.7 has failed on the final step to stage 
the files in Nexus.


All the files appear to be present but the plugin fails when 
processing the parent for the multi-module project.


Here is the command that is currently failing:

mvn -X -e -Duser.name=aherbert -Pcommons-rng-examples -Prelease clean 
package site site:stage deploy -rf :commons-rng


[DEBUG] -- end configuration --
[INFO] Checking out dist from: 
scm:svn:https://dist.apache.org/repos/dist/dev/commons/rng
Executing: /bin/sh -c cd 
'/scratch/commons-rng/dist-archive/target/commons-release-plugin' && 'svn' 
'--username' 'aherbert' '--no-auth-cache' '--non-interactive' 'checkout' 
'https://dist.apache.org/repos/dist/dev/commons/rng@' 
'/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
Executing: /bin/sh -c cd 
'/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup' 
&& 'svn' 'remove' '--targets' '/tmp/maven-scm-5448861436738372728-targets'
[INFO] Cleaning distribution area for: commons-rng
Executing: /bin/sh -c cd 
'/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup' 
&& 'svn' '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 'commit' 
'--file' '/tmp/maven-scm-1469264231.commit' '--encoding' 'UTF-8' '--targets' 
'/tmp/maven-scm-14546727756421455038-targets'
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time:  7.376 s
[INFO] Finished at: 2019-11-04T17:19:15Z
[INFO] 
[ERROR] Failed to execute goal 
org.apache.commons:commons-release-plugin:1.7:clean-staging (clean-staging) on 
project commons-rng: Failed to commit files: null [null] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.commons:commons-release-plugin:1.7:clean-staging (clean-staging) on 
project commons-rng: Failed to commit files: null [null]
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:215)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:156)
     at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:148)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
     at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
     at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:56)
     at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
     at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
     at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
     at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
     at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
     at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
     at jdk.internal.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
     at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
     at java.lang.reflect.Method.invoke (Method.java:564)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
     at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
     at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
(Launcher.java:415)
     at org.codehaus.plexus.classworlds.launcher.Launcher.main 
(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoFailureException: Failed to commit 
files: null [null]
     at 
org.apache.commons.release.plugin.mojos.CommonsStagingCleanupMojo.execute 
(CommonsStagingCleanupMojo.java:193)
     at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
(DefaultBuildPluginManager.java:137)
     at 

Re: [rng] Releasing 1.3

[Alex Herbert]

On 04/11/2019 15:40, Alex Herbert wrote:


On 04/11/2019 12:43, Gilles Sadowski wrote:

Hello.

Le lun. 4 nov. 2019 à 13:05, Alex Herbert  
a écrit :


On 04/11/2019 00:42, Gilles Sadowski wrote:

Hi.


[...]

So there are at least two options:

1. Release using JDK 9

+1

OK. I'll run through the release with JDK 9. I do not think we have any
NIO classes so the issue raised by Gary should not be present.

Here
   $ JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 mvn
-Pcommons-rng-examples clean package install site site:stage
built fine (but I did not check the reports).


Good to know.

I've been through a dry run and the outputs look as expected.


The commons-release-plugin:1.7 has failed on the final step to stage the 
files in Nexus.


All the files appear to be present but the plugin fails when processing 
the parent for the multi-module project.


Here is the command that is currently failing:

mvn -X -e -Duser.name=aherbert -Pcommons-rng-examples -Prelease clean 
package site site:stage deploy -rf :commons-rng


[DEBUG] -- end configuration --

[INFO] Checking out dist from: 
scm:svn:https://dist.apache.org/repos/dist/dev/commons/rng
Executing: /bin/sh -c cd 
'/scratch/commons-rng/dist-archive/target/commons-release-plugin' && 'svn' 
'--username' 'aherbert' '--no-auth-cache' '--non-interactive' 'checkout' 
'https://dist.apache.org/repos/dist/dev/commons/rng@' 
'/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup'
Executing: /bin/sh -c cd 
'/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup' 
&& 'svn' 'remove' '--targets' '/tmp/maven-scm-5448861436738372728-targets'
[INFO] Cleaning distribution area for: commons-rng
Executing: /bin/sh -c cd 
'/scratch/commons-rng/dist-archive/target/commons-release-plugin/scm-cleanup' 
&& 'svn' '--username' 'aherbert' '--no-auth-cache' '--non-interactive' 'commit' 
'--file' '/tmp/maven-scm-1469264231.commit' '--encoding' 'UTF-8' '--targets' 
'/tmp/maven-scm-14546727756421455038-targets'
[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time:  7.376 s
[INFO] Finished at: 2019-11-04T17:19:15Z
[INFO] 
[ERROR] Failed to execute goal 
org.apache.commons:commons-release-plugin:1.7:clean-staging (clean-staging) on 
project commons-rng: Failed to commit files: null [null] -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal 
org.apache.commons:commons-release-plugin:1.7:clean-staging (clean-staging) on 
project commons-rng: Failed to commit files: null [null]
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:215)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:156)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:148)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:117)
    at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject 
(LifecycleModuleBuilder.java:81)
    at 
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build
 (SingleThreadedBuilder.java:56)
    at org.apache.maven.lifecycle.internal.LifecycleStarter.execute 
(LifecycleStarter.java:128)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
    at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
    at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
    at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
    at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
    at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
    at jdk.internal.reflect.NativeMethodAccessorImpl.invoke 
(NativeMethodAccessorImpl.java:62)
    at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke 
(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke (Method.java:564)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced 
(Launcher.java:289)
    at org.codehaus.plexus.classworlds.launcher.Launcher.launch 
(Launcher.java:229)
    at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode 
(Launcher.java:415)
    at org.codehaus.plexus.classworlds.launcher.Launcher.main 
(Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoFailureException: Failed to commit 
files: null [null]
    at 
org.apache.commons.release.plugin.mojos.CommonsStagingCleanupMojo.execute 
(CommonsStagingCleanupMojo.java:193)
    at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo 
(DefaultBuildPluginManager.java:137)
    at org.apache.maven.lifecycle.internal.MojoExecutor.execute 
(MojoExecutor.java:210)
    at 

Re: [rng] Releasing 1.3

[Alex Herbert]

On 04/11/2019 12:43, Gilles Sadowski wrote:

Hello.

Le lun. 4 nov. 2019 à 13:05, Alex Herbert  a écrit :


On 04/11/2019 00:42, Gilles Sadowski wrote:

Hi.


[...]

So there are at least two options:

1. Release using JDK 9

+1

OK. I'll run through the release with JDK 9. I do not think we have any
NIO classes so the issue raised by Gary should not be present.

Here
   $ JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 mvn
-Pcommons-rng-examples clean package install site site:stage
built fine (but I did not check the reports).


Good to know.

I've been through a dry run and the outputs look as expected.

This command does not work with Java 9:

mvn -N -Pcommons-rng-examples commons-build:download-page

You have to build the download page using Java 8. Then switch to Java 9 
again.






I am looking at the clirr report to check and add @since tags.

Q. How to annotate implementation of a new interface on an existing class?

Should all the new interface methods have @since tags?

Isn't it good enough to have the tag in the class Javadoc?


For new classes. This is new methods added to existing classes due to 
implementation of a new interface.


I've taken the safe option to put @since on all the new methods.

I also added @since to all the new enum values in RandomSource.

Alex



Gilles


[...]

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [rng] Releasing 1.3

[Gilles Sadowski]

Hello.

Le lun. 4 nov. 2019 à 13:05, Alex Herbert  a écrit :
>
>
> On 04/11/2019 00:42, Gilles Sadowski wrote:
> > Hi.
> >
> >> [...]
> >>
> >> So there are at least two options:
> >>
> >> 1. Release using JDK 9
> > +1
>
> OK. I'll run through the release with JDK 9. I do not think we have any
> NIO classes so the issue raised by Gary should not be present.

Here
  $ JAVA_HOME=/usr/lib/jvm/java-9-openjdk-amd64 mvn
-Pcommons-rng-examples clean package install site site:stage
built fine (but I did not check the reports).

> I am looking at the clirr report to check and add @since tags.
>
> Q. How to annotate implementation of a new interface on an existing class?
>
> Should all the new interface methods have @since tags?

Isn't it good enough to have the tag in the class Javadoc?

Gilles

> >
> > [...]

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [rng] Releasing 1.3

[Alex Herbert]

On 04/11/2019 00:42, Gilles Sadowski wrote:

Hi.


[...]

So there are at least two options:

1. Release using JDK 9

+1


OK. I'll run through the release with JDK 9. I do not think we have any 
NIO classes so the issue raised by Gary should not be present.


I am looking at the clirr report to check and add @since tags.

Q. How to annotate implementation of a new interface on an existing class?

Should all the new interface methods have @since tags?





2. Update the examples project to use source 11 for the Java modules code and 
release using a non-oracle JDK 11

Note that updating to use source 11 will break the Jenkins build which uses JDK 
9 to run a full build including the -P commons-rng-examples profile. This could 
be ignored, updated to JDK 11 or dropped in favour of adding an appropriate JDK 
11 run using Travis.

Another option is to somehow get around the javadoc issue with modules and 
unnamed modules.

Let's not delay because of side issues. :-)
Hopefully, this will be fixed in a future JDK version, or when we
upgrade the target
of all modules.

Best,
Gilles

-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org



-
To unsubscribe, e-mail: dev-unsubscr...@commons.apache.org
For additional commands, e-mail: dev-h...@commons.apache.org

Re: [CODEC] Sign Extension Error in Murmur3

[Claude Warren]

There are a number of issues with the format and potential bugs in the
Codec Murmur3 code. ( See spotbugs, PMD, and codestyle reports)  The one
that tripped me up was the mix of tab/space lines.  Anyway, these issues
can be fixed in later pull requests.  I think is one overarching question
and one specific question:

In general, what is the policy for how do we deal with digest
implementations that have a bug and that have been released into the wild?

Specifically in this case, what do we do to solve the problem?

I was in favor of the following specific action:

1. Deprecate all affected methods and document the issue.
2. Move the deprecated methods into a new class called HiveMurmur3 and have
all deprecation point to this class.
3. Create a new Murmur3 class with method names like hash128_x64 (ones that
do not conflict with the old names).

But the more I think about it the more I think that we should follow KISS
and

1. Create a new Murmur3 class with method names like hash128_x64 (ones that
do not conflict with the old names).
2. Update the javadocs for the defective methods with notes indicating they
have defects.  Perhaps write a readme type document explaining in detail
the issue.

I think in general we should follow the same action with any digest defect:
New methods and Javadoc old.

Claude




On Mon, Nov 4, 2019 at 8:08 AM Alex Herbert 
wrote:

>
>
> > On 4 Nov 2019, at 02:13, sebb  wrote:
> >
> > On Mon, 4 Nov 2019 at 00:53, Claude Warren  cla...@xenei.com>> wrote:
> >>
> >> I think the way to prove they behave correctly is to test the results
> >> against the original "C" implementation.  With this in mind I proposed
> the
> >> changes.
> >>
> >> I agree that there should be a way to revert to the old code as there is
> >> code in the wild that depends on the earlier implementation.
> >>
> >> I know that there are broken implementations in other Apache projects,
> but
> >> those are not libraries (i.e. Cassandra).
> >>
> >> I would suggest that perhaps the changes that I provided be renamed as
> >> hash128Standard and hash32Standard (or some other designator to show
> that
> >> it is different from the original CODEC implementation).
> >>
> >> Alternatively hash128 and hash32 could be deprecated  and replaced with
> >> hashx86_32Old, and hashx64_128Old and provide the corrected versions for
> >> hashx86_32 and hashx64_128.  Obviously this leaves the door open to a
> >> hashx86_128 implementation.
> >>
> >> My opinion is that we should provide Murmur3 hash implementations that
> >> produce the same results as the "C"/C++ implementations for cross
> >> application compatibility.
> >
> > I agree: but the original implementation has far fewer methods.
> > So how does one check the Codec methods that have no corresponding C/C++
> method?
>
> AFAICT the methods that accept a short, int, or long are convenience
> methods to avoid creating a byte[] for the data. The JUnit tests
> demonstrate this by passing the same values to a ByteBuffer and then
> calling the corresponding byte[] methods. The methods do assume the endian
> order of the primitive datatypes as processed by a default ByteBuffer (Big
> Endian). So perhaps this should be added to the Javadoc with an example for
> the equivalent call to the method using the byte[].
>
> Note that these methods will not be broken. The bug in the code is for
> input byte[] lengths that are not a factor of 4. The fix is for any of the
> leftover 1, 2, or 3 bytes when computing the hash.
>
> The broken methods are:
>
> public static int hash32(final byte[] data)
> public static int hash32(final String data)
> public static int hash32(final byte[] data, final int length)
> public static int hash32(final byte[] data, final int length, final int
> seed)
> public static int hash32(final byte[] data, final int offset, final int
> length, final int seed)
>
> This class is also broken:
> public static class IncrementalHash32
>
> >
> > Maybe the Codec implementation needs to be drastically pruned as well.
> >
> >> Claude
> >>
> >> On Sun, Nov 3, 2019 at 11:34 PM sebb  wrote:
> >>
> >>> As I see it, there are two use cases here.
> >>>
> >>> 1) Commons Codec code is used exclusively, in which case it is
> >>> essential that the output does not change for a given seed.
> >>>
> >>> 2) Commons Codec is used in conjunction with other implementations, in
> >>> which case it is essential that Codec is aligned with other
> >>> implementations
> >>>
> >>> Both use cases seem valid to me, so I think it's important to allow
> >>> both old and new algorithms to co-exist going forward.
> >>>
> >>> However:
> >>> Should the default behaviour change (with option to revert)?
> >>> Or would it be better to make the corrected behaviour optional?
> >>>
> >>> Or deliberately break the API to force users to choose between old and
> >>> new behaviour?
> >>>
> >>> Note that Wikipedia says that the x86 and x64 versions of the
> >>> algorithm generate different results in the case of the 128 

Re: [CODEC] Sign Extension Error in Murmur3

[Alex Herbert]

> On 4 Nov 2019, at 02:13, sebb  wrote:
> 
> On Mon, 4 Nov 2019 at 00:53, Claude Warren  > wrote:
>> 
>> I think the way to prove they behave correctly is to test the results
>> against the original "C" implementation.  With this in mind I proposed the
>> changes.
>> 
>> I agree that there should be a way to revert to the old code as there is
>> code in the wild that depends on the earlier implementation.
>> 
>> I know that there are broken implementations in other Apache projects, but
>> those are not libraries (i.e. Cassandra).
>> 
>> I would suggest that perhaps the changes that I provided be renamed as
>> hash128Standard and hash32Standard (or some other designator to show that
>> it is different from the original CODEC implementation).
>> 
>> Alternatively hash128 and hash32 could be deprecated  and replaced with
>> hashx86_32Old, and hashx64_128Old and provide the corrected versions for
>> hashx86_32 and hashx64_128.  Obviously this leaves the door open to a
>> hashx86_128 implementation.
>> 
>> My opinion is that we should provide Murmur3 hash implementations that
>> produce the same results as the "C"/C++ implementations for cross
>> application compatibility.
> 
> I agree: but the original implementation has far fewer methods.
> So how does one check the Codec methods that have no corresponding C/C++ 
> method?

AFAICT the methods that accept a short, int, or long are convenience methods to 
avoid creating a byte[] for the data. The JUnit tests demonstrate this by 
passing the same values to a ByteBuffer and then calling the corresponding 
byte[] methods. The methods do assume the endian order of the primitive 
datatypes as processed by a default ByteBuffer (Big Endian). So perhaps this 
should be added to the Javadoc with an example for the equivalent call to the 
method using the byte[].

Note that these methods will not be broken. The bug in the code is for input 
byte[] lengths that are not a factor of 4. The fix is for any of the leftover 
1, 2, or 3 bytes when computing the hash.

The broken methods are:

public static int hash32(final byte[] data)
public static int hash32(final String data)
public static int hash32(final byte[] data, final int length)
public static int hash32(final byte[] data, final int length, final int seed)
public static int hash32(final byte[] data, final int offset, final int length, 
final int seed)

This class is also broken:
public static class IncrementalHash32

> 
> Maybe the Codec implementation needs to be drastically pruned as well.
> 
>> Claude
>> 
>> On Sun, Nov 3, 2019 at 11:34 PM sebb  wrote:
>> 
>>> As I see it, there are two use cases here.
>>> 
>>> 1) Commons Codec code is used exclusively, in which case it is
>>> essential that the output does not change for a given seed.
>>> 
>>> 2) Commons Codec is used in conjunction with other implementations, in
>>> which case it is essential that Codec is aligned with other
>>> implementations
>>> 
>>> Both use cases seem valid to me, so I think it's important to allow
>>> both old and new algorithms to co-exist going forward.
>>> 
>>> However:
>>> Should the default behaviour change (with option to revert)?
>>> Or would it be better to make the corrected behaviour optional?
>>> 
>>> Or deliberately break the API to force users to choose between old and
>>> new behaviour?
>>> 
>>> Note that Wikipedia says that the x86 and x64 versions of the
>>> algorithm generate different results in the case of the 128 bit hash.
>>> However Commons Codec does not have different implementations for x86
>>> and x64; it is not clear which has been implemented.
>>> The JavaDoc also fails to mention that there is a 64bit hash.

And lacks  and  tags in the header as appropriate.

>>> 
>>> The class clearly needs additional test data; it's possible that there
>>> are discrepancies in other hash sizes as well.
>>> 
>>> The code has multiple implementations for each hash size depending on
>>> the parameter types, which increases the number of tests needed.
>>> 
>>> Note that the original C++ implementation only has 3 hash methods:
>>> MurmurHash3_x86_32
>>> MurmurHash3_x86_128
>>> MurmurHash3_x64_128
>>> 
>>> AFAICT these all take the same parameters:
>>> - unsigned byte array
>>> - length
>>> - unsigned 32 bit seed
>>> - pointer to output buffer
>>> 
>>> No idea why the Codec version allows additional input such as one (or
>>> two) longs, etc.
>>> Unless there is a defined standard for how these should be handled, it
>>> will be impossible to prove that they behave correctly.
>>> 
>>> On Sun, 3 Nov 2019 at 22:01, Alex Herbert 
>>> wrote:
 
 
 
> On 3 Nov 2019, at 21:45, Gary Gregory  wrote:
> 
> I feel like I am missing something basic in the assumption of this
>>> issue:
> there is no such thing as an unsigned int in Java and the ticket talks
> about (C?) unsigned ints. Please help me understand how or why we
>>> should
> care about C vs. Java ints. Are we comparing