Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-07-01 Thread Shai Erera
About the encoders package - there are several encoders there besides VInt, so I wouldn't dispose of it so quickly. That said, I think we should definitely explore consolidating VInt with the core classes, and maybe write an encoder which delegate to them. Or, come up w/ a different approach for a

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-07-01 Thread Michael McCandless
On Fri, Jul 1, 2011 at 2:33 AM, Uwe Schindler wrote: > Hi, > > I don't understand the whole discussion here, so please compare these two > implementations and tell me which one is faster. Please don't hurt me, if you > don't want to see src.jar code from OpenJDK Java6 - just delete this mail if

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-07-01 Thread Michael McCandless
On Fri, Jul 1, 2011 at 1:47 AM, Simon Willnauer wrote: > Mike I think we should do that but the real issue here is what if > somebody comes up with any arbitrary method in the future claiming its > slow we back out and use the "we think safe way" what if it is > actually the other way around and

RE: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Uwe Schindler
Hi, I don't understand the whole discussion here, so please compare these two implementations and tell me which one is faster. Please don't hurt me, if you don't want to see src.jar code from OpenJDK Java6 - just delete this mail if you don’t want to (the code here is licensed under GPL): Arra

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
On Fri, Jul 1, 2011 at 12:19 AM, Michael McCandless wrote: > On Thu, Jun 30, 2011 at 4:45 PM, Simon Willnauer > wrote: >> On Thu, Jun 30, 2011 at 8:50 PM, Dawid Weiss >> wrote: I don't seen any evidence that this is any slower though. >>> >>> You need to run with -client (if the machine is

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Michael McCandless
On Thu, Jun 30, 2011 at 4:45 PM, Simon Willnauer wrote: > On Thu, Jun 30, 2011 at 8:50 PM, Dawid Weiss > wrote: >>> I don't seen any evidence that this is any slower though. >> >> You need to run with -client (if the machine is a beast this is tricky >> because x64 will pick -server regardless of

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
On Thu, Jun 30, 2011 at 8:50 PM, Dawid Weiss wrote: >> I don't seen any evidence that this is any slower though. > > You need to run with -client (if the machine is a beast this is tricky > because x64 will pick -server regardless of the command-line setting) > and you need to be copying generic a

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Michael McCandless
Fair enough, and I agree. Though the least we could do is rotate in a Windows env, where Java runs with -client, to our Jenkins. But simple-to-follow rules like "Don't use Arrays.copyOf; use System.arraycopy instead" (if indeed System.arraycopy seems to generally not be slower) seem like a no-bra

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Dawid Weiss
> I think it's important Lucene keeps good performance on "ordinary" > machines/envs. Not that this voice will help in anything, but I think the above is virtually impossible to achieve unless you have a bunch of machines, OSs and VMs to continually test on and a consistent set of benchmarks plott

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Michael McCandless
I think it's important Lucene keeps good performance on "ordinary" machines/envs. It's really quite dangerous that the active Lucene devs all use beasts for development/testing. We draw false conclusions. So we really should be testing with -client and if indeed generified Arrays.copyOf (and any

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Dawid Weiss
> I don't seen any evidence that this is any slower though. You need to run with -client (if the machine is a beast this is tricky because x64 will pick -server regardless of the command-line setting) and you need to be copying generic arrays. I think this can be shown -- a caliper benchmark would

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
On Thu, Jun 30, 2011 at 4:44 PM, Robert Muir wrote: > I think Dawid is correct here... so we should change this back? still > personally when I see array clone() or copyOf() it makes me concerned, I > know these are as fast as arraycopy in recent versions, but depending on > which variant is used,

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Robert Muir
I think Dawid is correct here... so we should change this back? still personally when I see array clone() or copyOf() it makes me concerned, I know these are as fast as arraycopy in recent versions, but depending on which variant is used, and whether you use -server, can be slower... in general I j

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
On Thu, Jun 30, 2011 at 3:26 PM, Uwe Schindler wrote: > We had an issue about this with FST's array growing in Mike's code, in facts > ist *much* slower for generic Arrays' T[] copyOf(T[]...), with T extends > Object (uses slow reflection). > > For primitives it can only get faster in later JVMs

RE: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Uwe Schindler
We had an issue about this with FST's array growing in Mike's code, in facts ist *much* slower for generic Arrays' T[] copyOf(T[]...), with T extends Object (uses slow reflection). For primitives it can only get faster in later JVMs, this is why we want to change all ArrayUtils.grow() to use th

RE: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Uwe Schindler
Robert, as noted in my other eMail, ist only slow for the generic Object[] method (as it uses j.l.reflect.Array.newInstance(Class componentType)). We are talking here about byte[], and the Arrays method is implemented with the same 3 lines of code, Simon replaced. The only difference is a Math

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Dawid Weiss
Arrays.copyOf(primitive) is actually System.arraycopy by default. If intrinsics are used it can only get faster. For object types it will probably be a bit slower for -client because of a runtime check for the component type. Dawid On Thu, Jun 30, 2011 at 3:05 PM, Robert Muir wrote: > because on

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
Robert I agree but doesn't that apply to Arrays.copyOf(Object[],int) only? here we use a specialized primitive version? simon On Thu, Jun 30, 2011 at 3:05 PM, Robert Muir wrote: > because on windows 32bit at least, -client is still the default on > most jres out there. > > i realize people don't

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Robert Muir
because on windows 32bit at least, -client is still the default on most jres out there. i realize people don't care about -client, but i will police foo[].clone() / arrays.copyOf etc to prevent problems. There are comments about this stuff on the relevant bug reports (oracle's site is down, sorry

RE: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Uwe Schindler
Hi Robert, you reverted a use of Arrays.copyOf() on native types which is *exactly* implemented like this in Arrays.java code! The slow ones are T[] copyOf(T[] array, int newlen) (because they use reflection). Uwe - Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.d

Re: svn commit: r1141510 - /lucene/dev/trunk/modules/facet/src/java/org/apache/lucene/util/UnsafeByteArrayOutputStream.java

2011-06-30 Thread Simon Willnauer
hmm are you concerned about the extra Math.min that happens in the copyOf method? I don't how that relates to "intrinsic" and java 1.7 I don't have strong feelings here just checking if you mix something up in the comment you put there... I am happy to keep the old and now current code simon On