I would like to add on to this thread that the feature to add locking around
command execution is added on the develop branch, and should be included in the
later release. So after that, you won’t get this error if you create multiple
regions in parallel using gfsh.
From: Jinmei Liao <jil...@vmware.com>
Reply-To: "user@geode.apache.org" <user@geode.apache.org>
Date: Thursday, November 5, 2020 at 8:02 AM
To: "user@geode.apache.org" <user@geode.apache.org>
Subject: Re: Creating multiple regions in parallel using gfsh
Hi, Ante,
Gfsh in general does not support concurrent users. All of its commands are
implemented without locking common resources. That’s an improvement yet to be
added to gfsh.
The exception you are seeing here are due to jaxb xml parser cannot handle
multiple parsing jobs at the same time, that will happen when two threads are
trying to add elements to the cluster configuration xml. One of them will fail,
but the regions are already created on the servers. The index creation later
will fail because it couldn’t find the region element in the xml when updating
the cluster configuration.
One thing you can try is when your script detect a “create region” command
failure, you can issue a “delete region” command to clean the half-done state.
It should work.
Jinmei
From: Ante Orlandini <ante.orland...@ericsson.com>
Reply-To: "user@geode.apache.org" <user@geode.apache.org>
Date: Thursday, November 5, 2020 at 2:43 AM
To: "user@geode.apache.org" <user@geode.apache.org>
Subject: Creating multiple regions in parallel using gfsh
Hi,
When creating multiple regions at once (in parallel) using gfsh interface, some
of region creations fail with exception:
[error 2020/11/05 10:59:29.034 CET <RMI TCP Connection(7)-127.0.0.1> tid=0x52]
Could not execute "create region --name=region1 --type=PARTITION".
java.lang.RuntimeException
at
org.apache.geode.internal.config.JAXBService.unMarshall(JAXBService.java:157)
at
org.apache.geode.management.internal.cli.commands.CreateRegionCommand.createRegion(CreateRegionCommand.java:522)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
at
org.apache.geode.management.internal.cli.remote.CommandExecutor.callInvokeMethod(CommandExecutor.java:139)
at
org.apache.geode.management.internal.cli.remote.CommandExecutor.invokeCommand(CommandExecutor.java:149)
at
org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:77)
at
org.apache.geode.management.internal.cli.remote.CommandExecutor.execute(CommandExecutor.java:61)
at
org.apache.geode.management.internal.cli.remote.OnlineCommandProcessor.executeCommand(OnlineCommandProcessor.java:132)
at
org.apache.geode.management.internal.cli.remote.OnlineCommandProcessor.executeCommandReturningJson(OnlineCommandProcessor.java:138)
at
org.apache.geode.management.internal.beans.MemberMBeanBridge.processCommand(MemberMBeanBridge.java:1237)
at
org.apache.geode.management.internal.beans.MemberMBean.processCommand(MemberMBean.java:424)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)
at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)
at
com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:193)
at
com.sun.jmx.mbeanserver.ConvertingMethod.invokeWithOpenReturn(ConvertingMethod.java:175)
at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:117)
at
com.sun.jmx.mbeanserver.MXBeanIntrospector.invokeM2(MXBeanIntrospector.java:54)
at
com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)
at
com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)
at
com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)
at
com.sun.jmx.remote.security.MBeanServerAccessController.invoke(MBeanServerAccessController.java:468)
at
javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)
at
javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)
at
javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)
at
javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)
at
javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)
at sun.rmi.transport.Transport$1.run(Transport.java:200)
at sun.rmi.transport.Transport$1.run(Transport.java:197)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.Transport.serviceCall(Transport.java:196)
at
sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at
sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: javax.xml.bind.UnmarshalException
- with linked exception:
[org.xml.sax.SAXException: FWK005 parse may not be called while parsing.]
at
javax.xml.bind.helpers.AbstractUnmarshallerImpl.createUnmarshalException(AbstractUnmarshallerImpl.java:335)
at
com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.createUnmarshalException(UnmarshallerImpl.java:548)
at
com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:234)
at
com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:206)
at
com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal(UnmarshallerImpl.java:258)
at
org.apache.geode.internal.config.JAXBService.unMarshall(JAXBService.java:155)
... 55 more
Caused by: org.xml.sax.SAXException: FWK005 parse may not be called while
parsing.
at
com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1253)
at
org.xml.sax.helpers.XMLFilterImpl.parse(XMLFilterImpl.java:357)
at
com.sun.xml.bind.v2.runtime.unmarshaller.UnmarshallerImpl.unmarshal0(UnmarshallerImpl.java:228)
... 58 more
Also regions that were created unsuccessfully remain in database and you can
put and query entries from it, but for example creating indexes fails:
gfsh>list regions
List of regions
---------------
region1
region2
region3
gfsh>put --region=region1 --key="1" --value="1"
Result : true
Key Class : java.lang.String
Key : 1
Value Class : java.lang.String
Old Value : null
gfsh>query --query="select * from /region1"
Result : true
Limit : 100
Rows : 1
Result
------
1
gfsh>create index --name=index --expression=aaa --type=range --region=/region1
Region region1 does not exist.
Does gfsh interface support this use case where you create regions in parallel?
To me this is invalid behavior since even if it doesn’t support it, regions
that were created unsuccessfully shouldn’t be left in database.
Full locator logs are attached, geode version is 13.0.1.
BR,
Ante Orlandini
