[PROPOSAL] backporting GEODE-8764 to 1.13 and 9.10
GEODE-8764 is an enhanced version of GEODE-6930. Lucene functions should only require DATA:READ permission on the specified region, no need to gain permission on other unrelated regions. The fix has no risk. Regards Xiaojian Zhou
Re: [PROPOSAL] Change the default value of conserve-sockets to false
Anil, you wrote: - We need to be thinking about auto setting of configuration values (dynamic) based on the load, resource availability and service agreements. It would be cool to eventually remove this property altogether and auto-configure it. Besides the things you mention, another thing that would need to be considered is features being used. For example, wan requires conserve-sockets=false. This discussion maybe should be moved to a different thread so we don't distract from this one. You also asked: - Will there be dedicated channel for communication from the node where conserve-socket is set to false to the remote nodes? Since the server doing the op has conserve-sockets=false, and an unshared p2p message reader is used on the remote member that means a dedicated (thread-owned) connection is used. ConnectionTable.get decides that. Here is a stack for creating a thread-owned sender: java.lang.Exception: Stack trace at java.lang.Thread.dumpStack(Thread.java:1333) at org.apache.geode.internal.tcp.Connection.(Connection.java:1224) at org.apache.geode.internal.tcp.Connection.createSender(Connection.java:1025) at org.apache.geode.internal.tcp.ConnectionTable.getThreadOwnedConnection(ConnectionTable.java:474) at org.apache.geode.internal.tcp.ConnectionTable.get(ConnectionTable.java:577) at org.apache.geode.internal.tcp.TCPConduit.getConnection(TCPConduit.java:800) at org.apache.geode.distributed.internal.direct.DirectChannel.getConnections(DirectChannel.java:452) at org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:268) at org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:182) at org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:511) Here are the same use cases with additional logging containing thread-owned Connection creation or shared Connection usage: Case 1: The server (server1) that processes the put operation from the client is primary and has conserve-sockets=false. The server (server2) that handles the UpdateWithContextMessage has conserve-sockets=true. 1. A ServerConnection thread in server1 sends the UpdateWithContextMessage: ServerConnection on port 60539 Thread 3: TestDistributionMessageObserver operation=beforeSendMessage; time=1607039519049; message=UpdateOperation$UpdateWithContextMessage(...); recipients=[192.168.1.8(server2:54360):41002] 2. The ServerConnection thread in server1 creates the thread-owned Connection: ServerConnection on port 60539 Thread 3: Connection. sender=192.168.1.8(server2:54360):41002(uid=10); socket=Socket[addr=/192.168.1.8,port=45823,localport=60595]; time=1607039519050 ServerConnection on port 60539 Thread 3: ConnectionTable.get using threadOwnedConnection=192.168.1.8(server2:54360):41002(uid=10); socket=Socket[addr=/192.168.1.8,port=45823,localport=60595]; time=1607039519051 3. A P2P Listener Thread in server2 creates the receiver Connection: P2P Listener Thread /192.168.1.8:45823: Connection. receiver=null(uid=0); socket=Socket[addr=/192.168.1.8,port=60595,localport=45823]; time=1607039519050 4. The unshared P2P message reader in server2 reads the handshake from server1's Connection: P2P message reader for 192.168.1.8(server1:54333):41001 unshared ordered uid=10 dom #1 local port=45823 remote port=60595: Connection.readHandshakeForReceiver receiver=192.168.1.8(server1:54333):41001(uid=10); socket=Socket[addr=/192.168.1.8,port=60595,localport=45823]; time=1607039519050 5. The unshared P2P message reader in server2 handles the UpdateWithContextMessage even though conserve-sockets=true: P2P message reader for 192.168.1.8(server1:54333):41001 unshared ordered uid=10 dom #1 local port=45823 remote port=60595: TestDistributionMessageObserver operation=beforeProcessMessage; time=1607039519051; message=UpdateOperation$UpdateWithContextMessage(...); recipients=[null] P2P message reader for 192.168.1.8(server1:54333):41001 unshared ordered uid=10 dom #1 local port=45823 remote port=60595: TestDistributionMessageObserver operation=afterProcessMessage; time=1607039519052; message=UpdateOperation$UpdateWithContextMessage(...); recipients=[null] Case 2: The server (server2) that processes the put operation from the client is primary and has conserve-sockets=true. The server (server1) that handles the UpdateWithContextMessage has conserve-sockets=false. 1. A ServerConnection thread in server2 sends the UpdateWithContextMessage: ServerConnection on port 60463 Thread 1: TestDistributionMessageObserver operation=beforeSendMessage; time=1607039137587; message=UpdateOperation$UpdateWithContextMessage(...); recipients=[192.168.1.8(server1:53948):41001] 2. The ServerConnection thread in server2 uses the shared Connection to server1: ServerConnection on port 60463 Thread 1: ConnectionTable.get using sharedConnection=192.168.1.8(server1:53948):41001(uid=3); socket=Socket[addr=/192.168.1.8,port=56562,localpo
Re: [PROPOSAL] Change the default value of conserve-sockets to false
+1 for having the default be conserve-sockets=false. Any time there has been trouble and conserve-sockets=true is involved we always suggest changing it to false. On 12/3/20, 6:58 AM, "Anilkumar Gingade" wrote: I was conversing with few of the dev's about requirement of different settings/configuration for set of nodes in the cluster depending on the business/application needs; for example set of nodes serving different kind of application requirement (data store) than other nodes in the cluster (computation heavy). I am calling this as heterogeneous cluster configuration (mostly in large cluster) compared to homogeneous cluster (same config across all the nodes). We need to be thinking both kind of deployment as the business models are moving towards cloud based services more and more for the entire org. We need to be thinking about auto setting of configuration values (dynamic) based on the load, resource availability and service agreements. We should plan taking few of these settings and build a logic where these can be automatically adjusted. Sorry for diverting from the actual email thread subject. Barry, it’s a great find. Will there be dedicated channel for communication from the node where conserve-socket is set to false to the remote nodes. -Anil. On 12/2/20, 3:14 PM, "Barrett Oglesby" wrote: I ran a bunch of tests using the long-running-test code where the servers had a mix of conserve-sockets settings, and they all worked ok. One set of tests had 6 servers - 3 with conserve-sockets=false and 3 with conserve-sockets=true. Another set of tests had 4 servers - 3 with conserve-sockets=false and 1 with conserve-sockets=true. In each case, the multi-threaded client did: - puts - gets - destroys - function updates - oql queries One thing I found interesting was the server where the operation originated dictated which thread was used on the remote server. If the server where the operation originated had conserve-sockets=false, then the remote server used an unshared P2P message reader to process the replication no matter what its conserve-sockets setting was. And if the server where the operation originated had conserve-sockets=true, then the remote server used a shared P2P message reader to process the replication no matter what its conserve-sockets setting was. Here is some logging from a DistributionMessageObserver that shows that behavior. Case 1: The server (server1) that processes the put operation from the client is primary and has conserve-sockets=false. The server (server2) that handles the UpdateWithContextMessage has conserve-sockets=true. 1. A ServerConnection thread in server1 sends the UpdateWithContextMessage: ServerConnection on port 60802 Thread 4: TestDistributionMessageObserver operation=beforeSendMessage; time=1606929894787; message=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[192.168.1.8(server-conserve-sockets1:58995):41002] 2. An unshared P2P message reader in server2 handles the UpdateWithContextMessage even though conserve-sockets=true: P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered uid=11 dom #1 local port=58405 remote port=60860: DistributionMessage.schedule msg=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; newValue=(10485820 bytes)) P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered uid=11 dom #1 local port=58405 remote port=60860: TestDistributionMessageObserver operation=beforeProcessMessage; time=1606929894809; message=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null] P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered uid=11 dom #1 local port=58405 remote port=60860: TestDistributionMessageObserver operation=afterProcessMessage; time=1606929894810; message=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null] Case 2: The server (server1) that processes the put operation from the client is primary and has conserve-sockets=true. The server (server2) that handles the UpdateWithContextMessage has conserve-sockets=false. 1. A ServerConnection thread in server1 sends the UpdateWithContextMessage: ServerConnection on port 61474 Thread 1: TestDistributionMessageObserver operation=beforeSendMessage; time=1606932400283; message=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__da
Re: [PROPOSAL] Change the default value of conserve-sockets to false
I was conversing with few of the dev's about requirement of different settings/configuration for set of nodes in the cluster depending on the business/application needs; for example set of nodes serving different kind of application requirement (data store) than other nodes in the cluster (computation heavy). I am calling this as heterogeneous cluster configuration (mostly in large cluster) compared to homogeneous cluster (same config across all the nodes). We need to be thinking both kind of deployment as the business models are moving towards cloud based services more and more for the entire org. We need to be thinking about auto setting of configuration values (dynamic) based on the load, resource availability and service agreements. We should plan taking few of these settings and build a logic where these can be automatically adjusted. Sorry for diverting from the actual email thread subject. Barry, it’s a great find. Will there be dedicated channel for communication from the node where conserve-socket is set to false to the remote nodes. -Anil. On 12/2/20, 3:14 PM, "Barrett Oglesby" wrote: I ran a bunch of tests using the long-running-test code where the servers had a mix of conserve-sockets settings, and they all worked ok. One set of tests had 6 servers - 3 with conserve-sockets=false and 3 with conserve-sockets=true. Another set of tests had 4 servers - 3 with conserve-sockets=false and 1 with conserve-sockets=true. In each case, the multi-threaded client did: - puts - gets - destroys - function updates - oql queries One thing I found interesting was the server where the operation originated dictated which thread was used on the remote server. If the server where the operation originated had conserve-sockets=false, then the remote server used an unshared P2P message reader to process the replication no matter what its conserve-sockets setting was. And if the server where the operation originated had conserve-sockets=true, then the remote server used a shared P2P message reader to process the replication no matter what its conserve-sockets setting was. Here is some logging from a DistributionMessageObserver that shows that behavior. Case 1: The server (server1) that processes the put operation from the client is primary and has conserve-sockets=false. The server (server2) that handles the UpdateWithContextMessage has conserve-sockets=true. 1. A ServerConnection thread in server1 sends the UpdateWithContextMessage: ServerConnection on port 60802 Thread 4: TestDistributionMessageObserver operation=beforeSendMessage; time=1606929894787; message=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[192.168.1.8(server-conserve-sockets1:58995):41002] 2. An unshared P2P message reader in server2 handles the UpdateWithContextMessage even though conserve-sockets=true: P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered uid=11 dom #1 local port=58405 remote port=60860: DistributionMessage.schedule msg=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; newValue=(10485820 bytes)) P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered uid=11 dom #1 local port=58405 remote port=60860: TestDistributionMessageObserver operation=beforeProcessMessage; time=1606929894809; message=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null] P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered uid=11 dom #1 local port=58405 remote port=60860: TestDistributionMessageObserver operation=afterProcessMessage; time=1606929894810; message=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null] Case 2: The server (server1) that processes the put operation from the client is primary and has conserve-sockets=true. The server (server2) that handles the UpdateWithContextMessage has conserve-sockets=false. 1. A ServerConnection thread in server1 sends the UpdateWithContextMessage: ServerConnection on port 61474 Thread 1: TestDistributionMessageObserver operation=beforeSendMessage; time=1606932400283; message=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[192.168.1.8(server1:63224):41001] 2. The shared P2P message reader in server2 handles the UpdateWithContextMessage and sends the ReplyMessage even though conserve-sockets=false: P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 shared ordered
