Hi again,
Looks like everybody is busy with stuff on a somewhat lower level of
Maslov's Pyramid? ;)
So, I will simply answer myself to my questions:
> It looks to me like Grape resp. ivy is not thread safe.
>
> Question 1: Is that a bug or just how things currently are? (If it's
a bug, rather Grape bug or an ivy bug, resp. where would I best file it?)
I consider that a bug, even though it seems to me that fixing it might
possibly be not so simple...
I created an account for the Groovy Apache JIRA, resp. seems like my
user had been already transferred there (or maybe I created an account
in the past and forgotten about it), with the intent to file a bug with
Groovy, but apparently I don't have the rights to do so?
If its possible without jumping through too many hoops, I would still be
available to file the bug (how?), else this post will have to suffice... ;)
> Question 2: If it's not a bug, is that a general issue, i.e. if I
compile two sets of sources with the GroovyCompiler (say I create a
CompilationUnit instance and add sources) will there be similar issues
if both sets of sources have the same Grape dependencies? (Yes, I know,
I could just try, but maybe the answer obvious to someone who knows the
implementation.)
Yes, it is definitely a general issue, see unit test further below which
uses CompilationUnit directly.
Question 3 (new): Am I stuck?
No, I am not, at least not with Grengine: http://grengine.ch
I can simply override DefaultGroovyCompiler with a class that
synchronizes around CompilationUnit.compile(), since Grengine is not
using GroovyClassLoader, GroovyShell or GroovyScriptEngine, except a
GroovyClassLoader during compilation. (Of course, in order to use Grape,
a GroovyClassLoader has to be in the classpath at runtime (or a
RootLoader, actually is there any real reason for that?), but that is
also possible to do by configuring Grengine.)
Or is there a workaround that works with Groovy itself, on the level of
the compiler or GroovyClassLoader/GroovyShell/GroovyScriptEngine, except
synchronizing each an every call that might compile anything?
If not, I will probably enhance Grengine to make the workaround simpler,
independently of whether this will be fixed in the future, in order to
provide a workaround for older Groovy versions.
Best wishes,
Alain
Here is the Java unit test:
import static org.junit.Assert.assertTrue;
import java.util.Collections;
import java.util.LinkedList;
import java.util.List;
import org.codehaus.groovy.control.CompilationUnit;
import org.codehaus.groovy.control.Phases;
import org.junit.Test;
public class GroovyCompileConcurrencyTest {
@Test
public void testConcurrencyLowLevel() throws Exception {
final List<Throwable> errors = Collections.synchronizedList(new
LinkedList<Throwable>());
final int nLoops = 1000;
final int nThreads = 10;
for (int i=0; i<nLoops; i++) {
System.out.print(".");
List<Thread> scriptThreads = new LinkedList<Thread>();
for (int j=0; j<nThreads; j++) {
final CompilationUnit cu = new CompilationUnit();
cu.addSource("Whatever",
"@Grab('com.google.guava:guava:18.0')\n" +
"import com.google.common.base.Ascii\n");
Thread scriptThread = new Thread(
new Runnable() {
public void run() {
try {
cu.compile(Phases.CLASS_GENERATION);
} catch (Throwable t) {
errors.add(t);
t.printStackTrace();
}
}
});
scriptThreads.add(scriptThread);
}
for (Thread scriptThread : scriptThreads) {
scriptThread.start();
}
for (Thread scriptThread : scriptThreads) {
scriptThread.join();
}
assertTrue("must be true", errors.size() == 0);
}
}
}
On 26.04.15 17:52, Alain Stalder wrote:
Hi there,
In the Java unit test listed further below, I create 100 GroovyShell
instances and add the same directory to the classpath of the
GroovyClassLoader of each GroovyShell. This directory contains 100
script files, named Util0.groovy to Util99.groovy, containing the
following script (with XX replaced by 0..99):
@Grab('com.google.guava:guava:18.0')
import com.google.common.base.Ascii
class UtilXX {
static boolean isUpperCase(def c) {
return Ascii.isUpperCase(c)\n"
}
}
The unit test then runs
shell.evaluate("return Util" + j + ".isUpperCase('C' as char)");
in 100 separate threads (with j from 0..99).
This test does not always fail, but often. Most of the time, a
ConcurrentModificationException occurs down in ivy (which is used by
Grape). In other cases, a MultipleCompilationErrorsException ocurred
at "@Grab('com.google.guava:guava:18.0')". (Full stacktraces at the
bottom.)
It looks to me like Grape resp. ivy is not thread safe.
Question 1: Is that a bug or just how things currently are? (If it's a
bug, rather Grape bug or an ivy bug, resp. where would I best file it?)
Question 2: If it's not a bug, is that a general issue, i.e. if I
compile two sets of sources with the GroovyCompiler (say I create a
CompilationUnit instance and add sources) will there be similar issues
if both sets of sources have the same Grape dependencies? (Yes, I
know, I could just try, but maybe the answer obvious to someone who
knows the implementation.)
(I have tested this with Groovy 2.4.3 and ivy 2.4.0 on MacOS X with
Java 6, but also I have also seen the same issues on CentOS with
several Groovy/ivy versions and Java 7, on a deployed webapp, so it
seems to be fairly independent of the exact environment.)
Best wishes,
Alain
-------------------------------------
Java Unit Test (does not always fail)
-------------------------------------
import static org.junit.Assert.assertEquals;
import static org.junit.Assert.assertTrue;
import groovy.lang.GroovyShell;
import java.io.File;
import java.io.FileNotFoundException;
import java.io.PrintWriter;
import java.io.UnsupportedEncodingException;
import java.util.LinkedList;
import java.util.List;
import org.junit.Before;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.TemporaryFolder;
public class GrapeAndGroovyShellConcurrencyTest {
@Rule
public TemporaryFolder tempFolder = new TemporaryFolder();
private volatile boolean testPassed;
@Before
public void setUp() {
testPassed = true;
}
public static void setFileText(File file, String text)
throws FileNotFoundException, UnsupportedEncodingException {
PrintWriter writer = new PrintWriter(file, "UTF-8");
writer.write(text);
writer.close();
}
@Test
public void testConcurrency() throws Exception {
File dir = tempFolder.getRoot();
final int n = 100;
for (int i=0; i<n; i++) {
File f1 = new File(dir, "Util" + i + ".groovy");
setFileText(f1, "@Grab('com.google.guava:guava:18.0')\n"
+ "import com.google.common.base.Ascii\n"
+ "class Util" + i + " {\n"
+ " static boolean isUpperCase(def c) {\n"
+ " return Ascii.isUpperCase(c)\n"
+ " }\n"
+ "}\n"
);
}
List<Thread> scriptThreads = new LinkedList<Thread>();
for (int i=0; i<n; i++) {
final GroovyShell shell = new GroovyShell();
shell.getClassLoader().addClasspath(dir.getAbsolutePath());
final int j = i;
Thread scriptThread = new Thread(
new Runnable() {
public void run() {
try {
assertEquals(true,
shell.evaluate("return Util" + j + ".isUpperCase('C' as char)"));
System.out.println("Thread " +
Thread.currentThread().getName() + " : OK");
} catch (Throwable t) {
System.out.println("Thread " +
Thread.currentThread().getName() + " : FAILED");
t.printStackTrace();
testPassed = false;
}
}
});
scriptThread.setDaemon(true);
scriptThread.setName("run-" + i);
scriptThread.start();
scriptThreads.add(scriptThread);
}
for (Thread scriptThread : scriptThreads) {
scriptThread.join();
}
assertTrue("must be true", testPassed);
}
}
------------
Stacktrace 1
------------
org.codehaus.groovy.control.MultipleCompilationErrorsException:
startup failed:
General error during conversion:
java.util.ConcurrentModificationException
java.util.ConcurrentModificationException
at java.util.ArrayList$Itr.checkForComodification(ArrayList.java:859)
at java.util.ArrayList$Itr.next(ArrayList.java:831)
at
org.apache.ivy.util.MessageLoggerHelper.sumupProblems(MessageLoggerHelper.java:45)
at
org.apache.ivy.util.MessageLoggerEngine.sumupProblems(MessageLoggerEngine.java:136)
at org.apache.ivy.util.Message.sumupProblems(Message.java:143)
at
org.apache.ivy.core.resolve.ResolveEngine.resolve(ResolveEngine.java:347)
at org.apache.ivy.Ivy.resolve(Ivy.java:523)
at org.apache.ivy.Ivy$resolve$1.call(Unknown Source)
at groovy.grape.GrapeIvy.getDependencies(GrapeIvy.groovy:404)
at sun.reflect.GeneratedMethodAccessor671.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSite.invoke(PogoMetaMethodSite.java:166)
at
org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:56)
at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:563)
at groovy.grape.GrapeIvy$resolve$56.callCurrent(Unknown Source)
at groovy.grape.GrapeIvy.resolve(GrapeIvy.groovy:532)
at groovy.grape.GrapeIvy$resolve$45.callCurrent(Unknown Source)
at
org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:49)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:151)
at
org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:179)
at groovy.grape.GrapeIvy.grab(GrapeIvy.groovy:254)
at groovy.grape.Grape.grab(Grape.java:163)
at
groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:358)
at
org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:319)
at
org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:928)
at
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:590)
at
org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:566)
at
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:543)
at
groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:297)
at
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:692)
at groovy.lang.GroovyShell.parse(GroovyShell.java:704)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:588)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:627)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:598)
at
GrapeAndGroovyShellConcurrencyTest$1.run(GrapeAndGroovyShellConcurrencyTest.java:64)
at java.lang.Thread.run(Thread.java:745)
1 error
at
org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309)
at
org.codehaus.groovy.control.ErrorCollector.addException(ErrorCollector.java:155)
at
org.codehaus.groovy.control.SourceUnit.addException(SourceUnit.java:345)
at
groovy.grape.GrabAnnotationTransformation.visit(GrabAnnotationTransformation.java:367)
at
org.codehaus.groovy.transform.ASTTransformationVisitor$3.call(ASTTransformationVisitor.java:319)
at
org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:928)
at
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:590)
at
org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:566)
at
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:543)
at
groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:297)
at
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:692)
at groovy.lang.GroovyShell.parse(GroovyShell.java:704)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:588)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:627)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:598)
at
GrapeAndGroovyShellConcurrencyTest$1.run(GrapeAndGroovyShellConcurrencyTest.java:64)
at java.lang.Thread.run(Thread.java:745)
------------
Stacktrace 2
------------
org.codehaus.groovy.control.MultipleCompilationErrorsException:
startup failed:
file:/var/folders/38/r0n49vmn7zg5dffk79_tgpl80000gn/T/junit124846036508580912/Util7.groovy:
1: unable to resolve class com.google.common.base.Ascii
@ line 1, column 1.
@Grab('com.google.guava:guava:18.0')
^
1 error
at
org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:309)
at
org.codehaus.groovy.control.CompilationUnit.applyToSourceUnits(CompilationUnit.java:943)
at
org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:590)
at
org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:539)
at
groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:297)
at
groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:267)
at groovy.lang.GroovyShell.parseClass(GroovyShell.java:692)
at groovy.lang.GroovyShell.parse(GroovyShell.java:704)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:588)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:627)
at groovy.lang.GroovyShell.evaluate(GroovyShell.java:598)
at
GrapeAndGroovyShellConcurrencyTest$1.run(GrapeAndGroovyShellConcurrencyTest.java:64)
at java.lang.Thread.run(Thread.java:745)
.
