subject:"Re\: \[PROPOSAL\] Change the default value of conserve\-sockets to false"

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-09 Thread Donal Evans

Thank you to everyone who participated in this discussion.

After a quick tally of responses, it looks like there are 6 people in favour of 
the change happening right now and 3 who are in favour but would prefer it to 
wait for a major version change. While this isn't a strong consensus, I do 
think that it demonstrates a solid majority in favour of the proposal as part 
of a minor version, so I have opened a pull request[1] with the necessary 
changes and invite everyone to provide feedback on it, particularly with 
regards to updating/expanding documentation around this change.

[1] https://github.com/apache/geode/pull/5832

From: Dan Smith 
Sent: Wednesday, December 9, 2020 11:22 AM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false

I will go ahead and withdraw my objection to this change. Based on some side 
conversations, at least at VMWare it sounds like we don't have customers that 
are not setting this flag. So the scenario I'm worried about where a customer 
upgrades their production cluster and has it crash due to this change seems 
less likely. I do agree false is a better default.

I would also be fine waiting until 2.0 to make this change.

-Dan

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-09 Thread John Blum

I agree with Dan, here, along with the consensus that false is the better 
default (in most cases).

So, I simply want to re-iterate the importance of "documentation" in whatever 
direction we decide.  There are, without a doubt, both pros and cons to each 
configuration arrangement (true of false) where the conserve-sockets setting is 
concerned.  Let's just make sure our old and new users are aware, too.

-j

From: Dan Smith 
Sent: Wednesday, December 9, 2020 11:22 AM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false

I will go ahead and withdraw my objection to this change. Based on some side 
conversations, at least at VMWare it sounds like we don't have customers that 
are not setting this flag. So the scenario I'm worried about where a customer 
upgrades their production cluster and has it crash due to this change seems 
less likely. I do agree false is a better default.

I would also be fine waiting until 2.0 to make this change.

-Dan

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-09 Thread Dan Smith

I will go ahead and withdraw my objection to this change. Based on some side 
conversations, at least at VMWare it sounds like we don't have customers that 
are not setting this flag. So the scenario I'm worried about where a customer 
upgrades their production cluster and has it crash due to this change seems 
less likely. I do agree false is a better default.

I would also be fine waiting until 2.0 to make this change.

-Dan

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-04 Thread Anilkumar Gingade

Barry, Thanks for the detailed response. Very helpful.

On 12/3/20, 4:22 PM, "Barrett Oglesby"  wrote:

Anil, you wrote:

- We need to be thinking about auto setting of configuration values 
(dynamic) based on the load, resource availability and service agreements.

It would be cool to eventually remove this property altogether and 
auto-configure it. Besides the things you mention, another thing that would 
need to be considered is features being used. For example, wan requires 
conserve-sockets=false. This discussion maybe should be moved to a different 
thread so we don't distract from this one.

You also asked:

- Will there be dedicated channel for communication from the node where 
conserve-socket is set to false to the remote nodes?

Since the server doing the op has conserve-sockets=false, and an unshared 
p2p message reader is used on the remote member that means a dedicated 
(thread-owned) connection is used.

ConnectionTable.get decides that. Here is a stack for creating a 
thread-owned sender:

java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1333)
at org.apache.geode.internal.tcp.Connection.(Connection.java:1224)
at 
org.apache.geode.internal.tcp.Connection.createSender(Connection.java:1025)
at 
org.apache.geode.internal.tcp.ConnectionTable.getThreadOwnedConnection(ConnectionTable.java:474)
at 
org.apache.geode.internal.tcp.ConnectionTable.get(ConnectionTable.java:577)
at 
org.apache.geode.internal.tcp.TCPConduit.getConnection(TCPConduit.java:800)
at 
org.apache.geode.distributed.internal.direct.DirectChannel.getConnections(DirectChannel.java:452)
at 
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:268)
at 
org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:182)
at 
org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:511)

Here are the same use cases with additional logging containing thread-owned 
Connection creation or shared Connection usage:

Case 1:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=false.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=true.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 60539 Thread 3: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1607039519049; 
message=UpdateOperation$UpdateWithContextMessage(...); 
recipients=[192.168.1.8(server2:54360):41002]

2. The ServerConnection thread in server1 creates the thread-owned 
Connection:

ServerConnection on port 60539 Thread 3: Connection. 
sender=192.168.1.8(server2:54360):41002(uid=10); 
socket=Socket[addr=/192.168.1.8,port=45823,localport=60595]; time=1607039519050
ServerConnection on port 60539 Thread 3: ConnectionTable.get using 
threadOwnedConnection=192.168.1.8(server2:54360):41002(uid=10); 
socket=Socket[addr=/192.168.1.8,port=45823,localport=60595]; time=1607039519051

3. A P2P Listener Thread in server2 creates the receiver Connection:

P2P Listener Thread /192.168.1.8:45823: Connection. 
receiver=null(uid=0); 
socket=Socket[addr=/192.168.1.8,port=60595,localport=45823]; time=1607039519050

4. The unshared P2P message reader in server2 reads the handshake from 
server1's Connection:

P2P message reader for 192.168.1.8(server1:54333):41001 unshared 
ordered uid=10 dom #1 local port=45823 remote port=60595: 
Connection.readHandshakeForReceiver 
receiver=192.168.1.8(server1:54333):41001(uid=10); 
socket=Socket[addr=/192.168.1.8,port=60595,localport=45823]; time=1607039519050

5. The unshared P2P message reader in server2 handles the 
UpdateWithContextMessage even though conserve-sockets=true:

P2P message reader for 192.168.1.8(server1:54333):41001 unshared 
ordered uid=10 dom #1 local port=45823 remote port=60595: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1607039519051; message=UpdateOperation$UpdateWithContextMessage(...); 
recipients=[null]
P2P message reader for 192.168.1.8(server1:54333):41001 unshared 
ordered uid=10 dom #1 local port=45823 remote port=60595: 
TestDistributionMessageObserver operation=afterProcessMessage; 
time=1607039519052; message=UpdateOperation$UpdateWithContextMessage(...); 
recipients=[null]

Case 2:

The server (server2) that processes the put operation from the client is 
primary and has conserve-sockets=true.
The server (server1) that handles the UpdateWithContextMessage has 
conserve-sockets=false.

1. A ServerConnection thread in server2 sends the UpdateWithContextMessage:

ServerConnection on port 60463 Thread 1: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1607039137587; 
message=UpdateOperation$UpdateWithContextMessage(...);

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-03 Thread Barrett Oglesby

Anil, you wrote:

- We need to be thinking about auto setting of configuration values (dynamic) 
based on the load, resource availability and service agreements.

It would be cool to eventually remove this property altogether and 
auto-configure it. Besides the things you mention, another thing that would 
need to be considered is features being used. For example, wan requires 
conserve-sockets=false. This discussion maybe should be moved to a different 
thread so we don't distract from this one.

You also asked:

- Will there be dedicated channel for communication from the node where 
conserve-socket is set to false to the remote nodes?

Since the server doing the op has conserve-sockets=false, and an unshared p2p 
message reader is used on the remote member that means a dedicated 
(thread-owned) connection is used.

ConnectionTable.get decides that. Here is a stack for creating a thread-owned 
sender:

java.lang.Exception: Stack trace
at java.lang.Thread.dumpStack(Thread.java:1333)
at org.apache.geode.internal.tcp.Connection.(Connection.java:1224)
at org.apache.geode.internal.tcp.Connection.createSender(Connection.java:1025)
at 
org.apache.geode.internal.tcp.ConnectionTable.getThreadOwnedConnection(ConnectionTable.java:474)
at org.apache.geode.internal.tcp.ConnectionTable.get(ConnectionTable.java:577)
at org.apache.geode.internal.tcp.TCPConduit.getConnection(TCPConduit.java:800)
at 
org.apache.geode.distributed.internal.direct.DirectChannel.getConnections(DirectChannel.java:452)
at 
org.apache.geode.distributed.internal.direct.DirectChannel.sendToMany(DirectChannel.java:268)
at 
org.apache.geode.distributed.internal.direct.DirectChannel.sendToOne(DirectChannel.java:182)
at 
org.apache.geode.distributed.internal.direct.DirectChannel.send(DirectChannel.java:511)

Here are the same use cases with additional logging containing thread-owned 
Connection creation or shared Connection usage:

Case 1:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=false.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=true.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 60539 Thread 3: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1607039519049; 
message=UpdateOperation$UpdateWithContextMessage(...); 
recipients=[192.168.1.8(server2:54360):41002]

2. The ServerConnection thread in server1 creates the thread-owned Connection:

ServerConnection on port 60539 Thread 3: Connection. 
sender=192.168.1.8(server2:54360):41002(uid=10); 
socket=Socket[addr=/192.168.1.8,port=45823,localport=60595]; time=1607039519050
ServerConnection on port 60539 Thread 3: ConnectionTable.get using 
threadOwnedConnection=192.168.1.8(server2:54360):41002(uid=10); 
socket=Socket[addr=/192.168.1.8,port=45823,localport=60595]; time=1607039519051

3. A P2P Listener Thread in server2 creates the receiver Connection:

P2P Listener Thread /192.168.1.8:45823: Connection. receiver=null(uid=0); 
socket=Socket[addr=/192.168.1.8,port=60595,localport=45823]; time=1607039519050

4. The unshared P2P message reader in server2 reads the handshake from 
server1's Connection:

P2P message reader for 192.168.1.8(server1:54333):41001 unshared ordered 
uid=10 dom #1 local port=45823 remote port=60595: 
Connection.readHandshakeForReceiver 
receiver=192.168.1.8(server1:54333):41001(uid=10); 
socket=Socket[addr=/192.168.1.8,port=60595,localport=45823]; time=1607039519050

5. The unshared P2P message reader in server2 handles the 
UpdateWithContextMessage even though conserve-sockets=true:

P2P message reader for 192.168.1.8(server1:54333):41001 unshared ordered 
uid=10 dom #1 local port=45823 remote port=60595: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1607039519051; message=UpdateOperation$UpdateWithContextMessage(...); 
recipients=[null]
P2P message reader for 192.168.1.8(server1:54333):41001 unshared ordered 
uid=10 dom #1 local port=45823 remote port=60595: 
TestDistributionMessageObserver operation=afterProcessMessage; 
time=1607039519052; message=UpdateOperation$UpdateWithContextMessage(...); 
recipients=[null]

Case 2:

The server (server2) that processes the put operation from the client is 
primary and has conserve-sockets=true.
The server (server1) that handles the UpdateWithContextMessage has 
conserve-sockets=false.

1. A ServerConnection thread in server2 sends the UpdateWithContextMessage:

ServerConnection on port 60463 Thread 1: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1607039137587; 
message=UpdateOperation$UpdateWithContextMessage(...); 
recipients=[192.168.1.8(server1:53948):41001]

2. The ServerConnection thread in server2 uses the shared Connection to server1:

ServerConnection on port 60463 Thread 1: ConnectionTable.get using 
sharedConnection=192.168.1.8(server1:53948):41001(uid=3);

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-03 Thread Bruce Schuchardt

+1 for having the default be conserve-sockets=false.   Any time there has been 
trouble and conserve-sockets=true is involved we always suggest changing it to 
false.


On 12/3/20, 6:58 AM, "Anilkumar Gingade"  wrote:

I was conversing with few of the dev's about requirement of different 
settings/configuration for set of nodes in the cluster depending on the 
business/application needs; for example set of nodes serving different kind of 
application requirement (data store) than other nodes in the cluster 
(computation heavy). I am calling this as heterogeneous cluster configuration 
(mostly in large cluster) compared to homogeneous cluster (same config across 
all the nodes). We need to be thinking both kind of deployment as the business 
models are moving towards cloud based services more and more for the entire org.
We need to be thinking about auto setting of configuration values (dynamic) 
based on the load, resource availability and service agreements. We should plan 
taking few of these settings and build a logic where these can be automatically 
adjusted.

Sorry for diverting from the actual email thread subject.

Barry, it’s a great find. Will there be dedicated channel for communication 
from the node where conserve-socket is set to false to the remote nodes.

 -Anil.

On 12/2/20, 3:14 PM, "Barrett Oglesby"  wrote:

I ran a bunch of tests using the long-running-test code where the 
servers had a mix of conserve-sockets settings, and they all worked ok.

One set of tests had 6 servers - 3 with conserve-sockets=false and 3 
with conserve-sockets=true.

Another set of tests had 4 servers - 3 with conserve-sockets=false and 
1 with conserve-sockets=true.

In each case, the multi-threaded client did:

- puts
- gets
- destroys
- function updates
- oql queries

One thing I found interesting was the server where the operation 
originated dictated which thread was used on the remote server. If the server 
where the operation originated had conserve-sockets=false, then the remote 
server used an unshared P2P message reader to process the replication no matter 
what its conserve-sockets setting was. And if the server where the operation 
originated had conserve-sockets=true, then the remote server used a shared P2P 
message reader to process the replication no matter what its conserve-sockets 
setting was.

Here is some logging from a DistributionMessageObserver that shows that 
behavior.

Case 1:

The server (server1) that processes the put operation from the client 
is primary and has conserve-sockets=false.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=true.

1. A ServerConnection thread in server1 sends the 
UpdateWithContextMessage:

ServerConnection on port 60802 Thread 4: 
TestDistributionMessageObserver operation=beforeSendMessage; 
time=1606929894787; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); 
recipients=[192.168.1.8(server-conserve-sockets1:58995):41002]

2. An unshared P2P message reader in server2 handles the 
UpdateWithContextMessage even though conserve-sockets=true:

P2P message reader for 192.168.1.8(server1:58984):41001 unshared 
ordered uid=11 dom #1 local port=58405 remote port=60860: 
DistributionMessage.schedule 
msg=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; 
sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; 
newValue=(10485820 bytes))
P2P message reader for 192.168.1.8(server1:58984):41001 unshared 
ordered uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1606929894809; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]
P2P message reader for 192.168.1.8(server1:58984):41001 unshared 
ordered uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=afterProcessMessage; 
time=1606929894810; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]

Case 2:

The server (server1) that processes the put operation from the client 
is primary and has conserve-sockets=true.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=false.

1. A ServerConnection thread in server1 sends the 
UpdateWithContextMessage:

ServerConnection on port 61474 Thread 1: 
TestDistributionMessageObserver operation=beforeSendMessage; 
time=1606932400283; message=UpdateOperation$UpdateWithContextMessage(region

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-03 Thread Anilkumar Gingade

I was conversing with few of the dev's about requirement of different 
settings/configuration for set of nodes in the cluster depending on the 
business/application needs; for example set of nodes serving different kind of 
application requirement (data store) than other nodes in the cluster 
(computation heavy). I am calling this as heterogeneous cluster configuration 
(mostly in large cluster) compared to homogeneous cluster (same config across 
all the nodes). We need to be thinking both kind of deployment as the business 
models are moving towards cloud based services more and more for the entire org.
We need to be thinking about auto setting of configuration values (dynamic) 
based on the load, resource availability and service agreements. We should plan 
taking few of these settings and build a logic where these can be automatically 
adjusted.

Sorry for diverting from the actual email thread subject.

Barry, it’s a great find. Will there be dedicated channel for communication 
from the node where conserve-socket is set to false to the remote nodes.

 -Anil.

On 12/2/20, 3:14 PM, "Barrett Oglesby"  wrote:

I ran a bunch of tests using the long-running-test code where the servers 
had a mix of conserve-sockets settings, and they all worked ok.

One set of tests had 6 servers - 3 with conserve-sockets=false and 3 with 
conserve-sockets=true.

Another set of tests had 4 servers - 3 with conserve-sockets=false and 1 
with conserve-sockets=true.

In each case, the multi-threaded client did:

- puts
- gets
- destroys
- function updates
- oql queries

One thing I found interesting was the server where the operation originated 
dictated which thread was used on the remote server. If the server where the 
operation originated had conserve-sockets=false, then the remote server used an 
unshared P2P message reader to process the replication no matter what its 
conserve-sockets setting was. And if the server where the operation originated 
had conserve-sockets=true, then the remote server used a shared P2P message 
reader to process the replication no matter what its conserve-sockets setting 
was.

Here is some logging from a DistributionMessageObserver that shows that 
behavior.

Case 1:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=false.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=true.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 60802 Thread 4: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1606929894787; 
message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); 
recipients=[192.168.1.8(server-conserve-sockets1:58995):41002]

2. An unshared P2P message reader in server2 handles the 
UpdateWithContextMessage even though conserve-sockets=true:

P2P message reader for 192.168.1.8(server1:58984):41001 unshared 
ordered uid=11 dom #1 local port=58405 remote port=60860: 
DistributionMessage.schedule 
msg=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; 
sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; 
newValue=(10485820 bytes))
P2P message reader for 192.168.1.8(server1:58984):41001 unshared 
ordered uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1606929894809; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]
P2P message reader for 192.168.1.8(server1:58984):41001 unshared 
ordered uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=afterProcessMessage; 
time=1606929894810; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]

Case 2:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=true.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=false.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 61474 Thread 1: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1606932400283; 
message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); 
recipients=[192.168.1.8(server1:63224):41001]

2. The shared P2P message reader in server2 handles the 
UpdateWithContextMessage and sends the ReplyMessage even though 
conserve-sockets=false:

P2P message reader for 
192.168.1.8(server-conserve-sockets1:63240):41002 shared

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-02 Thread Xiaojian Zhou

OK, I double checked, my memory is wrong. It was true as early as 6.0.

From: Xiaojian Zhou 
Date: Wednesday, December 2, 2020 at 3:29 PM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false
+1
I think it’s good to change back the default to be false. It was false before.

From: Barrett Oglesby 
Date: Wednesday, December 2, 2020 at 3:14 PM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false
I ran a bunch of tests using the long-running-test code where the servers had a 
mix of conserve-sockets settings, and they all worked ok.

One set of tests had 6 servers - 3 with conserve-sockets=false and 3 with 
conserve-sockets=true.

Another set of tests had 4 servers - 3 with conserve-sockets=false and 1 with 
conserve-sockets=true.

In each case, the multi-threaded client did:

- puts
- gets
- destroys
- function updates
- oql queries

One thing I found interesting was the server where the operation originated 
dictated which thread was used on the remote server. If the server where the 
operation originated had conserve-sockets=false, then the remote server used an 
unshared P2P message reader to process the replication no matter what its 
conserve-sockets setting was. And if the server where the operation originated 
had conserve-sockets=true, then the remote server used a shared P2P message 
reader to process the replication no matter what its conserve-sockets setting 
was.

Here is some logging from a DistributionMessageObserver that shows that 
behavior.

Case 1:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=false.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=true.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 60802 Thread 4: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1606929894787; 
message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); 
recipients=[192.168.1.8(server-conserve-sockets1:58995):41002]

2. An unshared P2P message reader in server2 handles the 
UpdateWithContextMessage even though conserve-sockets=true:

P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered 
uid=11 dom #1 local port=58405 remote port=60860: DistributionMessage.schedule 
msg=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; 
sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; 
newValue=(10485820 bytes))
P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered 
uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1606929894809; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]
P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered 
uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=afterProcessMessage; 
time=1606929894810; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]

Case 2:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=true.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=false.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 61474 Thread 1: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1606932400283; 
message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); 
recipients=[192.168.1.8(server1:63224):41001]

2. The shared P2P message reader in server2 handles the 
UpdateWithContextMessage and sends the ReplyMessage even though 
conserve-sockets=false:

P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 
shared ordered uid=4 local port=54619 remote port=61472: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1606932400295; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; 
sender=192.168.1.8(server-conserve-sockets1:63240):41002; op=UPDATE; 
key=0; newValue=(10485820 bytes)); recipients=[null]
P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 
shared ordered uid=4 local port=54619 remote port=61472: 
TestDistributionMessageObserver operation=beforeSendMessage; 
time=1606932400296; message=ReplyMessage processorId=42 from null; 
recipients=[192.168.1.8(server-conserve-sockets1:63240):41002]
P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 
shared

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-02 Thread Xiaojian Zhou

+1
I think it’s good to change back the default to be false. It was false before.

From: Barrett Oglesby 
Date: Wednesday, December 2, 2020 at 3:14 PM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false
I ran a bunch of tests using the long-running-test code where the servers had a 
mix of conserve-sockets settings, and they all worked ok.

One set of tests had 6 servers - 3 with conserve-sockets=false and 3 with 
conserve-sockets=true.

Another set of tests had 4 servers - 3 with conserve-sockets=false and 1 with 
conserve-sockets=true.

In each case, the multi-threaded client did:

- puts
- gets
- destroys
- function updates
- oql queries

One thing I found interesting was the server where the operation originated 
dictated which thread was used on the remote server. If the server where the 
operation originated had conserve-sockets=false, then the remote server used an 
unshared P2P message reader to process the replication no matter what its 
conserve-sockets setting was. And if the server where the operation originated 
had conserve-sockets=true, then the remote server used a shared P2P message 
reader to process the replication no matter what its conserve-sockets setting 
was.

Here is some logging from a DistributionMessageObserver that shows that 
behavior.

Case 1:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=false.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=true.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 60802 Thread 4: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1606929894787; 
message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); 
recipients=[192.168.1.8(server-conserve-sockets1:58995):41002]

2. An unshared P2P message reader in server2 handles the 
UpdateWithContextMessage even though conserve-sockets=true:

P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered 
uid=11 dom #1 local port=58405 remote port=60860: DistributionMessage.schedule 
msg=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; 
sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; 
newValue=(10485820 bytes))
P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered 
uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1606929894809; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]
P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered 
uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=afterProcessMessage; 
time=1606929894810; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]

Case 2:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=true.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=false.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 61474 Thread 1: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1606932400283; 
message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); 
recipients=[192.168.1.8(server1:63224):41001]

2. The shared P2P message reader in server2 handles the 
UpdateWithContextMessage and sends the ReplyMessage even though 
conserve-sockets=false:

P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 
shared ordered uid=4 local port=54619 remote port=61472: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1606932400295; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; 
sender=192.168.1.8(server-conserve-sockets1:63240):41002; op=UPDATE; 
key=0; newValue=(10485820 bytes)); recipients=[null]
P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 
shared ordered uid=4 local port=54619 remote port=61472: 
TestDistributionMessageObserver operation=beforeSendMessage; 
time=1606932400296; message=ReplyMessage processorId=42 from null; 
recipients=[192.168.1.8(server-conserve-sockets1:63240):41002]
P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 
shared ordered uid=4 local port=54619 remote port=61472: 
TestDistributionMessageObserver operation=afterProcessMessage; 
time=1606932400296; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48';

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-12-02 Thread Barrett Oglesby

I ran a bunch of tests using the long-running-test code where the servers had a 
mix of conserve-sockets settings, and they all worked ok.

One set of tests had 6 servers - 3 with conserve-sockets=false and 3 with 
conserve-sockets=true.

Another set of tests had 4 servers - 3 with conserve-sockets=false and 1 with 
conserve-sockets=true.

In each case, the multi-threaded client did:

- puts
- gets
- destroys
- function updates
- oql queries

One thing I found interesting was the server where the operation originated 
dictated which thread was used on the remote server. If the server where the 
operation originated had conserve-sockets=false, then the remote server used an 
unshared P2P message reader to process the replication no matter what its 
conserve-sockets setting was. And if the server where the operation originated 
had conserve-sockets=true, then the remote server used a shared P2P message 
reader to process the replication no matter what its conserve-sockets setting 
was.

Here is some logging from a DistributionMessageObserver that shows that 
behavior.

Case 1:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=false.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=true.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 60802 Thread 4: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1606929894787; 
message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); 
recipients=[192.168.1.8(server-conserve-sockets1:58995):41002]

2. An unshared P2P message reader in server2 handles the 
UpdateWithContextMessage even though conserve-sockets=true:

P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered 
uid=11 dom #1 local port=58405 remote port=60860: DistributionMessage.schedule 
msg=UpdateOperation$UpdateWithContextMessage(region path='/__PR/_B__data_48'; 
sender=192.168.1.8(server1:58984):41001; op=UPDATE; key=0; 
newValue=(10485820 bytes))
P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered 
uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1606929894809; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]
P2P message reader for 192.168.1.8(server1:58984):41001 unshared ordered 
uid=11 dom #1 local port=58405 remote port=60860: 
TestDistributionMessageObserver operation=afterProcessMessage; 
time=1606929894810; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; sender=192.168.1.8(server1:58984):41001; 
op=UPDATE; key=0; newValue=(10485820 bytes)); recipients=[null]

Case 2:

The server (server1) that processes the put operation from the client is 
primary and has conserve-sockets=true.
The server (server2) that handles the UpdateWithContextMessage has 
conserve-sockets=false.

1. A ServerConnection thread in server1 sends the UpdateWithContextMessage:

ServerConnection on port 61474 Thread 1: TestDistributionMessageObserver 
operation=beforeSendMessage; time=1606932400283; 
message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; op=UPDATE; key=0; newValue=(10485820 bytes)); 
recipients=[192.168.1.8(server1:63224):41001]

2. The shared P2P message reader in server2 handles the 
UpdateWithContextMessage and sends the ReplyMessage even though 
conserve-sockets=false:

P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 
shared ordered uid=4 local port=54619 remote port=61472: 
TestDistributionMessageObserver operation=beforeProcessMessage; 
time=1606932400295; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; 
sender=192.168.1.8(server-conserve-sockets1:63240):41002; op=UPDATE; 
key=0; newValue=(10485820 bytes)); recipients=[null]
P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 
shared ordered uid=4 local port=54619 remote port=61472: 
TestDistributionMessageObserver operation=beforeSendMessage; 
time=1606932400296; message=ReplyMessage processorId=42 from null; 
recipients=[192.168.1.8(server-conserve-sockets1:63240):41002]
P2P message reader for 192.168.1.8(server-conserve-sockets1:63240):41002 
shared ordered uid=4 local port=54619 remote port=61472: 
TestDistributionMessageObserver operation=afterProcessMessage; 
time=1606932400296; message=UpdateOperation$UpdateWithContextMessage(region 
path='/__PR/_B__data_48'; 
sender=192.168.1.8(server-conserve-sockets1:63240):41002; op=UPDATE; 
key=0; newValue=(10485820 bytes)); recipients=[null]

3. The shared P2P message reader in server1 handles the ReplyMessage:

P2P message reader for 192.168.1.8(server1:63224):41001 shared unordered 
uid=3

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-23 Thread Anthony Baker

Udo, you’re correct that individual servers can set the property independently. 
I was assuming this is more like the ’security-manager` property and others 
that require all cluster members to be in agreement.

I’m not sure I understand the use case to allow this setting to be per-member. 
That makes it pretty challenging to reason about what is happening in a cluster 
when doing root cause analysis. There is even an API to change this value 
dynamically:  
https://geode.apache.org/docs/guide/12/managing/monitor_tune/performance_controls_controlling_socket_use.html

…but I’ve only seen that used to make function threads/sockets follow the 
correct setting.

Anthony


On Nov 20, 2020, at 11:23 AM, Udo Kohlmeyer 
mailto:u...@vmware.com>> wrote:

@Anthony I cannot think of a single reason, why the server should not start up, 
even in a rolling upgrade. This setting should not have an effect on the 
cluster (other than potentially positive). Also, if the Geode were to enforce 
this setting across the cluster, then we have seriously broken our “shared 
nothing value” here..

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-23 Thread Xiaojian Zhou

Passed dunit tests is not enough. It might only mean we don't have enough test 
coverage. 

We need to inspect the code to see what will be the behavior when 2 servers 
configured different conserve-sockets.

On 11/20/20, 3:30 PM, "Donal Evans"  wrote:

Regarding behaviour during RollingUpgrade; I created a draft PR with this 
change to test the feasibility and see what problems, if any, would be caused 
by tests assuming the default setting to be true. After fixing two DUnit tests 
that were not explicitly setting the value of conserve-sockets to true, no test 
failures were observed. I also ran a large suite of proprietary tests that 
include rolling upgrade and observed no problems there. This doesn't mean that 
there would definitely be no problems caused by this change, but I can at least 
say that none of the testing we currently have showed any problems.

From: Anthony Baker 
Sent: Friday, November 20, 2020 8:52 AM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to 
false

Question:  how would this work with a rolling upgrade?  If the user did not 
set this property and we changed the default I believe that we would prevent 
the upgraded member from rejoining the cluster.

Of course the user could explicitly set this property as you point out.

Anthony

> On Nov 20, 2020, at 8:49 AM, Donal Evans  wrote:
>
> While I agree that the potential impact of having the setting changed out 
from a user may be high, the cost of addressing that change is very small. All 
users have to do is explicitly set the conserve-sockets value to true if they 
were previously using the default and they will be back to where they started 
with no change in behaviour or resource requirements. This could be as simple 
as adding a single line to a properties file, which seems like a pretty small 
inconvenience.
>
> Get Outlook for 
Android
>
> 
> From: Anthony Baker 
> Sent: Thursday, November 19, 2020 5:57:33 PM
> To: dev@geode.apache.org 
> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to 
false
>
> I think there are many good reasons to flip the default value for this 
property. I do question whether requiring a user to allocate new hardware to 
support the changed resource requirements is appropriate for a minor version 
bump. In most cases I think that would come as an unwelcome surprise during the 
upgrade.
>
> Anthony
>
>> On Nov 19, 2020, at 10:42 AM, Dan Smith  wrote:
>>
>> Personally, this has caused enough grief in the past (both ways, 
actually!) that I 'd say this is a major version change.
>> I agree with John. Either value of conserve-sockets can crash or hang 
your system depending on your use case.
>>
>> If this was just a matter of slowing down or speeding up performance, I 
think we could change it. But users that are impacted won't just see their 
system slow down. It will crash or hang. Potentially only with production sized 
workloads.
>>
>> With conserve-sockets=false every thread on the server creates its own 
sockets to other servers. With N servers that's N sockets per thread. With our 
default of a max of 800 threads for client connections and a 20 server cluster 
you are looking at a worst case of 800 * 20 = 16K sending sockets per server, 
with another 16K receiving sockets and 16K receiving threads. That's before 
considering function execution threads, WAN receivers, and various other 
executors we have on the server. Users with too many threads will hit their 
file descriptor or thread limits. Or they will run out of memory for thread 
stacks, socket buffers, etc.
>>
>> -Dan
>>
>

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-20 Thread Donal Evans

Regarding behaviour during RollingUpgrade; I created a draft PR with this 
change to test the feasibility and see what problems, if any, would be caused 
by tests assuming the default setting to be true. After fixing two DUnit tests 
that were not explicitly setting the value of conserve-sockets to true, no test 
failures were observed. I also ran a large suite of proprietary tests that 
include rolling upgrade and observed no problems there. This doesn't mean that 
there would definitely be no problems caused by this change, but I can at least 
say that none of the testing we currently have showed any problems.

From: Anthony Baker 
Sent: Friday, November 20, 2020 8:52 AM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false

Question:  how would this work with a rolling upgrade?  If the user did not set 
this property and we changed the default I believe that we would prevent the 
upgraded member from rejoining the cluster.

Of course the user could explicitly set this property as you point out.


Anthony


> On Nov 20, 2020, at 8:49 AM, Donal Evans  wrote:
>
> While I agree that the potential impact of having the setting changed out 
> from a user may be high, the cost of addressing that change is very small. 
> All users have to do is explicitly set the conserve-sockets value to true if 
> they were previously using the default and they will be back to where they 
> started with no change in behaviour or resource requirements. This could be 
> as simple as adding a single line to a properties file, which seems like a 
> pretty small inconvenience.
>
> Get Outlook for 
> Android
>
> 
> From: Anthony Baker 
> Sent: Thursday, November 19, 2020 5:57:33 PM
> To: dev@geode.apache.org 
> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false
>
> I think there are many good reasons to flip the default value for this 
> property. I do question whether requiring a user to allocate new hardware to 
> support the changed resource requirements is appropriate for a minor version 
> bump. In most cases I think that would come as an unwelcome surprise during 
> the upgrade.
>
> Anthony
>
>> On Nov 19, 2020, at 10:42 AM, Dan Smith  wrote:
>>
>> Personally, this has caused enough grief in the past (both ways, actually!) 
>> that I 'd say this is a major version change.
>> I agree with John. Either value of conserve-sockets can crash or hang your 
>> system depending on your use case.
>>
>> If this was just a matter of slowing down or speeding up performance, I 
>> think we could change it. But users that are impacted won't just see their 
>> system slow down. It will crash or hang. Potentially only with production 
>> sized workloads.
>>
>> With conserve-sockets=false every thread on the server creates its own 
>> sockets to other servers. With N servers that's N sockets per thread. With 
>> our default of a max of 800 threads for client connections and a 20 server 
>> cluster you are looking at a worst case of 800 * 20 = 16K sending sockets 
>> per server, with another 16K receiving sockets and 16K receiving threads. 
>> That's before considering function execution threads, WAN receivers, and 
>> various other executors we have on the server. Users with too many threads 
>> will hit their file descriptor or thread limits. Or they will run out of 
>> memory for thread stacks, socket buffers, etc.
>>
>> -Dan
>>
>

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-20 Thread Udo Kohlmeyer

@Anthony I cannot think of a single reason, why the server should not start up, 
even in a rolling upgrade. This setting should not have an effect on the 
cluster (other than potentially positive). Also, if the Geode were to enforce 
this setting across the cluster, then we have seriously broken our “shared 
nothing value” here..

From: Xiaojian Zhou 
Date: Saturday, November 21, 2020 at 5:34 AM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false
1) Conserve-socket will only impact p2p connection. If set to false, that mean 
the p2p connections between 2 servers can be created on request, as many as 
needed.
2) currently the default setting is true (I don't remember when did we change 
it from false to true)
3) For rollingUpgrade, unfortunately, if server1 is set to true and server2 is 
set to false, our server start-up will not check the mismatch automatically so 
far. We have to add some coding to prevent a server with different setting to 
join. And I don't know in current mixed setting environment, what will be the 
behavior. It could be an interesting dunit test scenario.

Regards
Gester

On 11/20/20, 8:53 AM, "Anthony Baker"  wrote:

Question:  how would this work with a rolling upgrade?  If the user did not 
set this property and we changed the default I believe that we would prevent 
the upgraded member from rejoining the cluster.

Of course the user could explicitly set this property as you point out.

Anthony

> On Nov 20, 2020, at 8:49 AM, Donal Evans  wrote:
>
> While I agree that the potential impact of having the setting changed out 
from a user may be high, the cost of addressing that change is very small. All 
users have to do is explicitly set the conserve-sockets value to true if they 
were previously using the default and they will be back to where they started 
with no change in behaviour or resource requirements. This could be as simple 
as adding a single line to a properties file, which seems like a pretty small 
inconvenience.
>
> Get Outlook for 
Android
>
> 
> From: Anthony Baker 
> Sent: Thursday, November 19, 2020 5:57:33 PM
> To: dev@geode.apache.org 
> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to 
false
>
> I think there are many good reasons to flip the default value for this 
property. I do question whether requiring a user to allocate new hardware to 
support the changed resource requirements is appropriate for a minor version 
bump. In most cases I think that would come as an unwelcome surprise during the 
upgrade.
>
> Anthony
>
>> On Nov 19, 2020, at 10:42 AM, Dan Smith  wrote:
>>
>> Personally, this has caused enough grief in the past (both ways, 
actually!) that I 'd say this is a major version change.
>> I agree with John. Either value of conserve-sockets can crash or hang 
your system depending on your use case.
>>
>> If this was just a matter of slowing down or speeding up performance, I 
think we could change it. But users that are impacted won't just see their 
system slow down. It will crash or hang. Potentially only with production sized 
workloads.
>>
>> With conserve-sockets=false every thread on the server creates its own 
sockets to other servers. With N servers that's N sockets per thread. With our 
default of a max of 800 threads for client connections and a 20 server cluster 
you are looking at a worst case of 800 * 20 = 16K sending sockets per server, 
with another 16K receiving sockets and 16K receiving threads. That's before 
considering function execution threads, WAN receivers, and various other 
executors we have on the server. Users with too many threads will hit their 
file descriptor or thread limits. Or they will run out of memory for thread 
stacks, socket buffers, etc.
>>
>> -Dan
>>
>

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-20 Thread Xiaojian Zhou

1) Conserve-socket will only impact p2p connection. If set to false, that mean 
the p2p connections between 2 servers can be created on request, as many as 
needed. 
2) currently the default setting is true (I don't remember when did we change 
it from false to true)
3) For rollingUpgrade, unfortunately, if server1 is set to true and server2 is 
set to false, our server start-up will not check the mismatch automatically so 
far. We have to add some coding to prevent a server with different setting to 
join. And I don't know in current mixed setting environment, what will be the 
behavior. It could be an interesting dunit test scenario. 

Regards
Gester

On 11/20/20, 8:53 AM, "Anthony Baker"  wrote:

Question:  how would this work with a rolling upgrade?  If the user did not 
set this property and we changed the default I believe that we would prevent 
the upgraded member from rejoining the cluster.

Of course the user could explicitly set this property as you point out.


Anthony


> On Nov 20, 2020, at 8:49 AM, Donal Evans  wrote:
> 
> While I agree that the potential impact of having the setting changed out 
from a user may be high, the cost of addressing that change is very small. All 
users have to do is explicitly set the conserve-sockets value to true if they 
were previously using the default and they will be back to where they started 
with no change in behaviour or resource requirements. This could be as simple 
as adding a single line to a properties file, which seems like a pretty small 
inconvenience.
> 
> Get Outlook for 
Android
> 
> 
> From: Anthony Baker 
> Sent: Thursday, November 19, 2020 5:57:33 PM
> To: dev@geode.apache.org 
> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to 
false
> 
> I think there are many good reasons to flip the default value for this 
property. I do question whether requiring a user to allocate new hardware to 
support the changed resource requirements is appropriate for a minor version 
bump. In most cases I think that would come as an unwelcome surprise during the 
upgrade.
> 
> Anthony
> 
>> On Nov 19, 2020, at 10:42 AM, Dan Smith  wrote:
>> 
>> Personally, this has caused enough grief in the past (both ways, 
actually!) that I 'd say this is a major version change.
>> I agree with John. Either value of conserve-sockets can crash or hang 
your system depending on your use case.
>> 
>> If this was just a matter of slowing down or speeding up performance, I 
think we could change it. But users that are impacted won't just see their 
system slow down. It will crash or hang. Potentially only with production sized 
workloads.
>> 
>> With conserve-sockets=false every thread on the server creates its own 
sockets to other servers. With N servers that's N sockets per thread. With our 
default of a max of 800 threads for client connections and a 20 server cluster 
you are looking at a worst case of 800 * 20 = 16K sending sockets per server, 
with another 16K receiving sockets and 16K receiving threads. That's before 
considering function execution threads, WAN receivers, and various other 
executors we have on the server. Users with too many threads will hit their 
file descriptor or thread limits. Or they will run out of memory for thread 
stacks, socket buffers, etc.
>> 
>> -Dan
>> 
>

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-20 Thread Anthony Baker

Question:  how would this work with a rolling upgrade?  If the user did not set 
this property and we changed the default I believe that we would prevent the 
upgraded member from rejoining the cluster.

Of course the user could explicitly set this property as you point out.


Anthony


> On Nov 20, 2020, at 8:49 AM, Donal Evans  wrote:
> 
> While I agree that the potential impact of having the setting changed out 
> from a user may be high, the cost of addressing that change is very small. 
> All users have to do is explicitly set the conserve-sockets value to true if 
> they were previously using the default and they will be back to where they 
> started with no change in behaviour or resource requirements. This could be 
> as simple as adding a single line to a properties file, which seems like a 
> pretty small inconvenience.
> 
> Get Outlook for 
> Android
> 
> 
> From: Anthony Baker 
> Sent: Thursday, November 19, 2020 5:57:33 PM
> To: dev@geode.apache.org 
> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false
> 
> I think there are many good reasons to flip the default value for this 
> property. I do question whether requiring a user to allocate new hardware to 
> support the changed resource requirements is appropriate for a minor version 
> bump. In most cases I think that would come as an unwelcome surprise during 
> the upgrade.
> 
> Anthony
> 
>> On Nov 19, 2020, at 10:42 AM, Dan Smith  wrote:
>> 
>> Personally, this has caused enough grief in the past (both ways, actually!) 
>> that I 'd say this is a major version change.
>> I agree with John. Either value of conserve-sockets can crash or hang your 
>> system depending on your use case.
>> 
>> If this was just a matter of slowing down or speeding up performance, I 
>> think we could change it. But users that are impacted won't just see their 
>> system slow down. It will crash or hang. Potentially only with production 
>> sized workloads.
>> 
>> With conserve-sockets=false every thread on the server creates its own 
>> sockets to other servers. With N servers that's N sockets per thread. With 
>> our default of a max of 800 threads for client connections and a 20 server 
>> cluster you are looking at a worst case of 800 * 20 = 16K sending sockets 
>> per server, with another 16K receiving sockets and 16K receiving threads. 
>> That's before considering function execution threads, WAN receivers, and 
>> various other executors we have on the server. Users with too many threads 
>> will hit their file descriptor or thread limits. Or they will run out of 
>> memory for thread stacks, socket buffers, etc.
>> 
>> -Dan
>> 
>

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-20 Thread Donal Evans

While I agree that the potential impact of having the setting changed out from 
a user may be high, the cost of addressing that change is very small. All users 
have to do is explicitly set the conserve-sockets value to true if they were 
previously using the default and they will be back to where they started with 
no change in behaviour or resource requirements. This could be as simple as 
adding a single line to a properties file, which seems like a pretty small 
inconvenience.

Get Outlook for Android

From: Anthony Baker 
Sent: Thursday, November 19, 2020 5:57:33 PM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false

I think there are many good reasons to flip the default value for this 
property. I do question whether requiring a user to allocate new hardware to 
support the changed resource requirements is appropriate for a minor version 
bump. In most cases I think that would come as an unwelcome surprise during the 
upgrade.

Anthony

> On Nov 19, 2020, at 10:42 AM, Dan Smith  wrote:
>
> Personally, this has caused enough grief in the past (both ways, actually!) 
> that I 'd say this is a major version change.
> I agree with John. Either value of conserve-sockets can crash or hang your 
> system depending on your use case.
>
> If this was just a matter of slowing down or speeding up performance, I think 
> we could change it. But users that are impacted won't just see their system 
> slow down. It will crash or hang. Potentially only with production sized 
> workloads.
>
> With conserve-sockets=false every thread on the server creates its own 
> sockets to other servers. With N servers that's N sockets per thread. With 
> our default of a max of 800 threads for client connections and a 20 server 
> cluster you are looking at a worst case of 800 * 20 = 16K sending sockets per 
> server, with another 16K receiving sockets and 16K receiving threads. That's 
> before considering function execution threads, WAN receivers, and various 
> other executors we have on the server. Users with too many threads will hit 
> their file descriptor or thread limits. Or they will run out of memory for 
> thread stacks, socket buffers, etc.
>
> -Dan
>

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-19 Thread Anthony Baker

I think there are many good reasons to flip the default value for this 
property. I do question whether requiring a user to allocate new hardware to 
support the changed resource requirements is appropriate for a minor version 
bump. In most cases I think that would come as an unwelcome surprise during the 
upgrade.

Anthony

> On Nov 19, 2020, at 10:42 AM, Dan Smith  wrote:
> 
> Personally, this has caused enough grief in the past (both ways, actually!) 
> that I 'd say this is a major version change.
> I agree with John. Either value of conserve-sockets can crash or hang your 
> system depending on your use case.
> 
> If this was just a matter of slowing down or speeding up performance, I think 
> we could change it. But users that are impacted won't just see their system 
> slow down. It will crash or hang. Potentially only with production sized 
> workloads.
> 
> With conserve-sockets=false every thread on the server creates its own 
> sockets to other servers. With N servers that's N sockets per thread. With 
> our default of a max of 800 threads for client connections and a 20 server 
> cluster you are looking at a worst case of 800 * 20 = 16K sending sockets per 
> server, with another 16K receiving sockets and 16K receiving threads. That's 
> before considering function execution threads, WAN receivers, and various 
> other executors we have on the server. Users with too many threads will hit 
> their file descriptor or thread limits. Or they will run out of memory for 
> thread stacks, socket buffers, etc.
> 
> -Dan
>

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-19 Thread Dan Smith

Personally, this has caused enough grief in the past (both ways, actually!) 
that I 'd say this is a major version change.
I agree with John. Either value of conserve-sockets can crash or hang your 
system depending on your use case.

If this was just a matter of slowing down or speeding up performance, I think 
we could change it. But users that are impacted won't just see their system 
slow down. It will crash or hang. Potentially only with production sized 
workloads.

With conserve-sockets=false every thread on the server creates its own sockets 
to other servers. With N servers that's N sockets per thread. With our default 
of a max of 800 threads for client connections and a 20 server cluster you are 
looking at a worst case of 800 * 20 = 16K sending sockets per server, with 
another 16K receiving sockets and 16K receiving threads. That's before 
considering function execution threads, WAN receivers, and various other 
executors we have on the server. Users with too many threads will hit their 
file descriptor or thread limits. Or they will run out of memory for thread 
stacks, socket buffers, etc.

-Dan

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-19 Thread Donal Evans

Just to clarify a comment from Owen, conserve-sockets=true does show improved 
performance over conserve-sockets=false in certain specific scenarios, but this 
isn't a hard and fast rule that applies everywhere, even excluding the cases 
where using conserve-sockets=true can lead to distributed deadlocks. As an 
example, with the new default setting of false, the UpgradeTest job in the CI 
pipeline takes about 10% longer, indicating that, at least for the scenarios 
being tested there, there is some performance impact. That being said, all of 
the geode-benchmarks tests explicitly set conserve-sockets=false, so from the 
point of view of what we're actually testing in terms of performance, no impact 
will be seen due to this change.

From: Jacob Barrett 
Sent: Thursday, November 19, 2020 8:02 AM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false

I would argue that is doesn’t change the outward behavior of the product. It 
does change the internal workings of the product. It does improve the 
performance and reliability. As long as changes to the internals don’t have 
effect on the outward facing behavior of the product I see no problem making 
these internal changes. Yes this may lead to more resource utilization but so 
can and have so other changes along the way between majors. I would also expect 
in the scenarios where this would make the most impact on resources they are 
already configured to use this feature.

+1 make the change.


> On Nov 19, 2020, at 1:16 AM, Ju@N  wrote:
>
> I'm all in for changing the default to *false* but, unfortunately and for
> all the reasons already stated in the thread, I'm hesitant to include this
> change as part of a minor release.
> Best regards.
>
> On Thu, 19 Nov 2020 at 02:48, John Blum  wrote:
>
>> The downside of conserve-sockets = false is that you are (essentially)
>> back to a Thread|Socket / Request model (though Geode limits this system
>> resource consumption to a degree by the use of Thread Pools in p2p
>> distribution layer) and thus, you can run out of file descriptors (per
>> newly opened Socket) pretty quickly if you are not careful.
>>
>> conserve-sockets set to true limits the use of finite system resources why
>> risking deadlocks (i.e. A -> B -> C -> A), which is also contingent on ACKS
>> (and the infamous ReplyProcessor21; at least at 1 time, not sure if it is
>> still in play, but probably!).
>>
>> conserve-sockets set to false uses more system resources but avoids
>> deadlocks.
>>
>> If this change is made, I'd minimally make sure to document that users
>> should adjust their (soft & hard) ulimits accordingly, based on use cases,
>> load, etc.
>>
>> Personally, this has caused enough grief in the past (both ways,
>> actually!) that I 'd say this is a major version change.
>>
>> -j
>>
>>
>> 
>> From: Nabarun Nag 
>> Sent: Wednesday, November 18, 2020 6:09 PM
>> To: dev@geode.apache.org 
>> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to
>> false
>>
>> +1
>>
>>  *   As nearly all of the production use-cases need "conserve-sockets" to
>> be set to false, I think we can aim for changing the default value to false
>> for 1.14.0 release.
>>  *   We can highlight this change in the release notes and emails.
>>
>> Regards,
>> Nabarun
>>
>> 
>> From: Udo Kohlmeyer 
>> Sent: Wednesday, November 18, 2020 6:00 PM
>> To: dev@geode.apache.org 
>> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to
>> false
>>
>> Hi there Donal,
>>
>> Thank you for raising this. It is not an uncommon request to change the
>> default value of this field.
>>
>> This has been discussed many times in the past. I would LOVE to approve
>> this change, but this would mean that users that don’t set this property
>> might suddenly have this property changed. We are not sure that this would
>> mean for these users. BUT
>>
>> That said, there have been very little (to none) complaints about the
>> product stability when `conserve-sockets=false` are set.
>>
>> +1 - if we are allowed to make this change outside of a major version
>> change.
>>
>> --Udo
>>
>> From: Donal Evans 
>> Date: Thursday, November 19, 2020 at 12:04 PM
>> To: dev@geode.apache.org 
>> Subject: [PROPOSAL] Change the default value of conserve-sockets to false
>> Hi Geode dev,
>>
>> First, from the docs[1], a brief explanation of the purpose of the
>> conserve-sockets property:
>>
>> "The conserve-sockets setting indicates whether application threads share
>> sockets with other threads or use their own sockets for member
>> communication. This setting has no effect on communication between a server
>> and its clients, but it does control the server’s communication with its
>> peers or a gateway sender’s communication with a gateway receiver."
>>
>> The current default value for the conserve-sockets property is true, which
>> at

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-19 Thread Jacob Barrett

I would argue that is doesn’t change the outward behavior of the product. It 
does change the internal workings of the product. It does improve the 
performance and reliability. As long as changes to the internals don’t have 
effect on the outward facing behavior of the product I see no problem making 
these internal changes. Yes this may lead to more resource utilization but so 
can and have so other changes along the way between majors. I would also expect 
in the scenarios where this would make the most impact on resources they are 
already configured to use this feature.

+1 make the change.


> On Nov 19, 2020, at 1:16 AM, Ju@N  wrote:
> 
> I'm all in for changing the default to *false* but, unfortunately and for
> all the reasons already stated in the thread, I'm hesitant to include this
> change as part of a minor release.
> Best regards.
> 
> On Thu, 19 Nov 2020 at 02:48, John Blum  wrote:
> 
>> The downside of conserve-sockets = false is that you are (essentially)
>> back to a Thread|Socket / Request model (though Geode limits this system
>> resource consumption to a degree by the use of Thread Pools in p2p
>> distribution layer) and thus, you can run out of file descriptors (per
>> newly opened Socket) pretty quickly if you are not careful.
>> 
>> conserve-sockets set to true limits the use of finite system resources why
>> risking deadlocks (i.e. A -> B -> C -> A), which is also contingent on ACKS
>> (and the infamous ReplyProcessor21; at least at 1 time, not sure if it is
>> still in play, but probably!).
>> 
>> conserve-sockets set to false uses more system resources but avoids
>> deadlocks.
>> 
>> If this change is made, I'd minimally make sure to document that users
>> should adjust their (soft & hard) ulimits accordingly, based on use cases,
>> load, etc.
>> 
>> Personally, this has caused enough grief in the past (both ways,
>> actually!) that I 'd say this is a major version change.
>> 
>> -j
>> 
>> 
>> 
>> From: Nabarun Nag 
>> Sent: Wednesday, November 18, 2020 6:09 PM
>> To: dev@geode.apache.org 
>> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to
>> false
>> 
>> +1
>> 
>>  *   As nearly all of the production use-cases need "conserve-sockets" to
>> be set to false, I think we can aim for changing the default value to false
>> for 1.14.0 release.
>>  *   We can highlight this change in the release notes and emails.
>> 
>> Regards,
>> Nabarun
>> 
>> 
>> From: Udo Kohlmeyer 
>> Sent: Wednesday, November 18, 2020 6:00 PM
>> To: dev@geode.apache.org 
>> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to
>> false
>> 
>> Hi there Donal,
>> 
>> Thank you for raising this. It is not an uncommon request to change the
>> default value of this field.
>> 
>> This has been discussed many times in the past. I would LOVE to approve
>> this change, but this would mean that users that don’t set this property
>> might suddenly have this property changed. We are not sure that this would
>> mean for these users. BUT
>> 
>> That said, there have been very little (to none) complaints about the
>> product stability when `conserve-sockets=false` are set.
>> 
>> +1 - if we are allowed to make this change outside of a major version
>> change.
>> 
>> --Udo
>> 
>> From: Donal Evans 
>> Date: Thursday, November 19, 2020 at 12:04 PM
>> To: dev@geode.apache.org 
>> Subject: [PROPOSAL] Change the default value of conserve-sockets to false
>> Hi Geode dev,
>> 
>> First, from the docs[1], a brief explanation of the purpose of the
>> conserve-sockets property:
>> 
>> "The conserve-sockets setting indicates whether application threads share
>> sockets with other threads or use their own sockets for member
>> communication. This setting has no effect on communication between a server
>> and its clients, but it does control the server’s communication with its
>> peers or a gateway sender’s communication with a gateway receiver."
>> 
>> The current default value for the conserve-sockets property is true, which
>> at first glance makes sense, since in an ideal world, existing sockets
>> could be shared between threads and there would be no need to create and
>> destroy new sockets for each process, which can be somewhat
>> resource-intensive. However, in practice, there are several known issues
>> with using the default setting of true. From the docs[1]:
>> 
>> "For distributed regions, the put operation, and destroy and invalidate
>> for regions and entries, can all be optimized with conserve-sockets set to
>> false. For partitioned regions, setting conserve-sockets to false can
>> improve general throughput.
>> Note: When you have transactions operating on EMPTY, NORMAL or PARTITION
>> regions, make sure that conserve-sockets is set to false to avoid
>> distributed deadlocks."
>> 
>> and[2]:
>> 
>> "WAN deployments increase the messaging demands on a Geode system. To
>> avoid hangs related to WAN messaging, always

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-19 Thread Ju@N

I'm all in for changing the default to *false* but, unfortunately and for
all the reasons already stated in the thread, I'm hesitant to include this
change as part of a minor release.
Best regards.

On Thu, 19 Nov 2020 at 02:48, John Blum  wrote:

> The downside of conserve-sockets = false is that you are (essentially)
> back to a Thread|Socket / Request model (though Geode limits this system
> resource consumption to a degree by the use of Thread Pools in p2p
> distribution layer) and thus, you can run out of file descriptors (per
> newly opened Socket) pretty quickly if you are not careful.
>
> conserve-sockets set to true limits the use of finite system resources why
> risking deadlocks (i.e. A -> B -> C -> A), which is also contingent on ACKS
> (and the infamous ReplyProcessor21; at least at 1 time, not sure if it is
> still in play, but probably!).
>
> conserve-sockets set to false uses more system resources but avoids
> deadlocks.
>
> If this change is made, I'd minimally make sure to document that users
> should adjust their (soft & hard) ulimits accordingly, based on use cases,
> load, etc.
>
> Personally, this has caused enough grief in the past (both ways,
> actually!) that I 'd say this is a major version change.
>
> -j
>
>
> 
> From: Nabarun Nag 
> Sent: Wednesday, November 18, 2020 6:09 PM
> To: dev@geode.apache.org 
> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to
> false
>
> +1
>
>   *   As nearly all of the production use-cases need "conserve-sockets" to
> be set to false, I think we can aim for changing the default value to false
> for 1.14.0 release.
>   *   We can highlight this change in the release notes and emails.
>
> Regards,
> Nabarun
>
> 
> From: Udo Kohlmeyer 
> Sent: Wednesday, November 18, 2020 6:00 PM
> To: dev@geode.apache.org 
> Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to
> false
>
> Hi there Donal,
>
> Thank you for raising this. It is not an uncommon request to change the
> default value of this field.
>
> This has been discussed many times in the past. I would LOVE to approve
> this change, but this would mean that users that don’t set this property
> might suddenly have this property changed. We are not sure that this would
> mean for these users. BUT
>
> That said, there have been very little (to none) complaints about the
> product stability when `conserve-sockets=false` are set.
>
> +1 - if we are allowed to make this change outside of a major version
> change.
>
> --Udo
>
> From: Donal Evans 
> Date: Thursday, November 19, 2020 at 12:04 PM
> To: dev@geode.apache.org 
> Subject: [PROPOSAL] Change the default value of conserve-sockets to false
> Hi Geode dev,
>
> First, from the docs[1], a brief explanation of the purpose of the
> conserve-sockets property:
>
> "The conserve-sockets setting indicates whether application threads share
> sockets with other threads or use their own sockets for member
> communication. This setting has no effect on communication between a server
> and its clients, but it does control the server’s communication with its
> peers or a gateway sender’s communication with a gateway receiver."
>
> The current default value for the conserve-sockets property is true, which
> at first glance makes sense, since in an ideal world, existing sockets
> could be shared between threads and there would be no need to create and
> destroy new sockets for each process, which can be somewhat
> resource-intensive. However, in practice, there are several known issues
> with using the default setting of true. From the docs[1]:
>
> "For distributed regions, the put operation, and destroy and invalidate
> for regions and entries, can all be optimized with conserve-sockets set to
> false. For partitioned regions, setting conserve-sockets to false can
> improve general throughput.
> Note: When you have transactions operating on EMPTY, NORMAL or PARTITION
> regions, make sure that conserve-sockets is set to false to avoid
> distributed deadlocks."
>
> and[2]:
>
> "WAN deployments increase the messaging demands on a Geode system. To
> avoid hangs related to WAN messaging, always set `conserve-sockets=false`
> for Geode members that participate in a WAN deployment."
>
> Given that it is generally accepted as best practice to set
> conserve-sockets to false for almost all use cases of Geode beyond the most
> simple, it would make sense to also change the default value to false, to
> prevent people having to encounter a problem, search for the solution, then
> change the setting to what is almost always the "correct" value.
>
> I have done some experimenting to see what it would take to make this
> proposal a reality, and the changes required are very minimal, with only
> two existing DUnit tests that need to be modified to explicitly set the
> value of conserve-sockets that were previously relying on the default being
> true.
>
> Any feedback on this

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-18 Thread John Blum

The downside of conserve-sockets = false is that you are (essentially) back to 
a Thread|Socket / Request model (though Geode limits this system resource 
consumption to a degree by the use of Thread Pools in p2p distribution layer) 
and thus, you can run out of file descriptors (per newly opened Socket) pretty 
quickly if you are not careful.

conserve-sockets set to true limits the use of finite system resources why 
risking deadlocks (i.e. A -> B -> C -> A), which is also contingent on ACKS 
(and the infamous ReplyProcessor21; at least at 1 time, not sure if it is still 
in play, but probably!).

conserve-sockets set to false uses more system resources but avoids deadlocks.

If this change is made, I'd minimally make sure to document that users should 
adjust their (soft & hard) ulimits accordingly, based on use cases, load, etc.

Personally, this has caused enough grief in the past (both ways, actually!) 
that I 'd say this is a major version change.

-j



From: Nabarun Nag 
Sent: Wednesday, November 18, 2020 6:09 PM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false

+1

  *   As nearly all of the production use-cases need "conserve-sockets" to be 
set to false, I think we can aim for changing the default value to false for 
1.14.0 release.
  *   We can highlight this change in the release notes and emails.

Regards,
Nabarun


From: Udo Kohlmeyer 
Sent: Wednesday, November 18, 2020 6:00 PM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false

Hi there Donal,

Thank you for raising this. It is not an uncommon request to change the default 
value of this field.

This has been discussed many times in the past. I would LOVE to approve this 
change, but this would mean that users that don’t set this property might 
suddenly have this property changed. We are not sure that this would mean for 
these users. BUT

That said, there have been very little (to none) complaints about the product 
stability when `conserve-sockets=false` are set.

+1 - if we are allowed to make this change outside of a major version change.

--Udo

From: Donal Evans 
Date: Thursday, November 19, 2020 at 12:04 PM
To: dev@geode.apache.org 
Subject: [PROPOSAL] Change the default value of conserve-sockets to false
Hi Geode dev,

First, from the docs[1], a brief explanation of the purpose of the 
conserve-sockets property:

"The conserve-sockets setting indicates whether application threads share 
sockets with other threads or use their own sockets for member communication. 
This setting has no effect on communication between a server and its clients, 
but it does control the server’s communication with its peers or a gateway 
sender’s communication with a gateway receiver."

The current default value for the conserve-sockets property is true, which at 
first glance makes sense, since in an ideal world, existing sockets could be 
shared between threads and there would be no need to create and destroy new 
sockets for each process, which can be somewhat resource-intensive. However, in 
practice, there are several known issues with using the default setting of 
true. From the docs[1]:

"For distributed regions, the put operation, and destroy and invalidate for 
regions and entries, can all be optimized with conserve-sockets set to false. 
For partitioned regions, setting conserve-sockets to false can improve general 
throughput.
Note: When you have transactions operating on EMPTY, NORMAL or PARTITION 
regions, make sure that conserve-sockets is set to false to avoid distributed 
deadlocks."

and[2]:

"WAN deployments increase the messaging demands on a Geode system. To avoid 
hangs related to WAN messaging, always set `conserve-sockets=false` for Geode 
members that participate in a WAN deployment."

Given that it is generally accepted as best practice to set conserve-sockets to 
false for almost all use cases of Geode beyond the most simple, it would make 
sense to also change the default value to false, to prevent people having to 
encounter a problem, search for the solution, then change the setting to what 
is almost always the "correct" value.

I have done some experimenting to see what it would take to make this proposal 
a reality, and the changes required are very minimal, with only two existing 
DUnit tests that need to be modified to explicitly set the value of 
conserve-sockets that were previously relying on the default being true.

Any feedback on this proposal would be very welcome, and if the response is 
positive, I can create a PR with the changes as soon as a decision is reached.

Thanks,
Donal

[1]

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-18 Thread Nabarun Nag

+1

  *   As nearly all of the production use-cases need "conserve-sockets" to be 
set to false, I think we can aim for changing the default value to false for 
1.14.0 release.
  *   We can highlight this change in the release notes and emails.

Regards,
Nabarun


From: Udo Kohlmeyer 
Sent: Wednesday, November 18, 2020 6:00 PM
To: dev@geode.apache.org 
Subject: Re: [PROPOSAL] Change the default value of conserve-sockets to false

Hi there Donal,

Thank you for raising this. It is not an uncommon request to change the default 
value of this field.

This has been discussed many times in the past. I would LOVE to approve this 
change, but this would mean that users that don’t set this property might 
suddenly have this property changed. We are not sure that this would mean for 
these users. BUT

That said, there have been very little (to none) complaints about the product 
stability when `conserve-sockets=false` are set.

+1 - if we are allowed to make this change outside of a major version change.

--Udo

From: Donal Evans 
Date: Thursday, November 19, 2020 at 12:04 PM
To: dev@geode.apache.org 
Subject: [PROPOSAL] Change the default value of conserve-sockets to false
Hi Geode dev,

First, from the docs[1], a brief explanation of the purpose of the 
conserve-sockets property:

"The conserve-sockets setting indicates whether application threads share 
sockets with other threads or use their own sockets for member communication. 
This setting has no effect on communication between a server and its clients, 
but it does control the server’s communication with its peers or a gateway 
sender’s communication with a gateway receiver."

The current default value for the conserve-sockets property is true, which at 
first glance makes sense, since in an ideal world, existing sockets could be 
shared between threads and there would be no need to create and destroy new 
sockets for each process, which can be somewhat resource-intensive. However, in 
practice, there are several known issues with using the default setting of 
true. From the docs[1]:

"For distributed regions, the put operation, and destroy and invalidate for 
regions and entries, can all be optimized with conserve-sockets set to false. 
For partitioned regions, setting conserve-sockets to false can improve general 
throughput.
Note: When you have transactions operating on EMPTY, NORMAL or PARTITION 
regions, make sure that conserve-sockets is set to false to avoid distributed 
deadlocks."

and[2]:

"WAN deployments increase the messaging demands on a Geode system. To avoid 
hangs related to WAN messaging, always set `conserve-sockets=false` for Geode 
members that participate in a WAN deployment."

Given that it is generally accepted as best practice to set conserve-sockets to 
false for almost all use cases of Geode beyond the most simple, it would make 
sense to also change the default value to false, to prevent people having to 
encounter a problem, search for the solution, then change the setting to what 
is almost always the "correct" value.

I have done some experimenting to see what it would take to make this proposal 
a reality, and the changes required are very minimal, with only two existing 
DUnit tests that need to be modified to explicitly set the value of 
conserve-sockets that were previously relying on the default being true.

Any feedback on this proposal would be very welcome, and if the response is 
positive, I can create a PR with the changes as soon as a decision is reached.

Thanks,
Donal

[1] 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgeode.apache.org%2Fdocs%2Fguide%2F113%2Fmanaging%2Fmonitor_tune%2Fperformance_controls_controlling_socket_use.htmldata=04%7C01%7Cnnag%40vmware.com%7Cb3a63185f8f948ce5e1708d88c2ee809%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637413480399258567%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=hwVkxp99IM%2FeRWDt39P81GO%2Bq7LD7maQewAKLYqQlj0%3Dreserved=0
[2] 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgeode.apache.org%2Fdocs%2Fguide%2F113%2Fmanaging%2Fmonitor_tune%2Fsockets_and_gateways.htmldata=04%7C01%7Cnnag%40vmware.com%7Cb3a63185f8f948ce5e1708d88c2ee809%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637413480399268565%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=FOBKgXh1tOj3WDsLZRbBXoufPeaUu4YTKkSDKD81P9A%3Dreserved=0

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-18 Thread Owen Nichols

I’m not familiar with the inner workings, but your writeup is excellent and 
makes a compelling case.

It sounds like you are saying that the original motivation for 
conserve-sockets=true was to improve performance, but in fact it makes 
performance worse.  Do you have some numbers to quantify what magnitude of a 
difference this can make?

It also looks bad that we actually document that Geode may deadlock if you use 
the default.

To Udo’s concern about changing existing behavior, it sounds like you are 
saying there is no possible downside to conserve-sockets=false, but has that 
question been explored exhaustively?  Is there ever any circumstance where we 
would still recommend a user to explicitly set conserve-sockets=true, once 
false is the default?

From: Donal Evans 
Date: Wednesday, November 18, 2020 at 5:04 PM
To: dev@geode.apache.org 
Subject: [PROPOSAL] Change the default value of conserve-sockets to false
Hi Geode dev,

First, from the docs[1], a brief explanation of the purpose of the 
conserve-sockets property:

"The conserve-sockets setting indicates whether application threads share 
sockets with other threads or use their own sockets for member communication. 
This setting has no effect on communication between a server and its clients, 
but it does control the server’s communication with its peers or a gateway 
sender’s communication with a gateway receiver."

The current default value for the conserve-sockets property is true, which at 
first glance makes sense, since in an ideal world, existing sockets could be 
shared between threads and there would be no need to create and destroy new 
sockets for each process, which can be somewhat resource-intensive. However, in 
practice, there are several known issues with using the default setting of 
true. From the docs[1]:

"For distributed regions, the put operation, and destroy and invalidate for 
regions and entries, can all be optimized with conserve-sockets set to false. 
For partitioned regions, setting conserve-sockets to false can improve general 
throughput.
Note: When you have transactions operating on EMPTY, NORMAL or PARTITION 
regions, make sure that conserve-sockets is set to false to avoid distributed 
deadlocks."

and[2]:

"WAN deployments increase the messaging demands on a Geode system. To avoid 
hangs related to WAN messaging, always set `conserve-sockets=false` for Geode 
members that participate in a WAN deployment."

Given that it is generally accepted as best practice to set conserve-sockets to 
false for almost all use cases of Geode beyond the most simple, it would make 
sense to also change the default value to false, to prevent people having to 
encounter a problem, search for the solution, then change the setting to what 
is almost always the "correct" value.

I have done some experimenting to see what it would take to make this proposal 
a reality, and the changes required are very minimal, with only two existing 
DUnit tests that need to be modified to explicitly set the value of 
conserve-sockets that were previously relying on the default being true.

Any feedback on this proposal would be very welcome, and if the response is 
positive, I can create a PR with the changes as soon as a decision is reached.

Thanks,
Donal

[1] 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgeode.apache.org%2Fdocs%2Fguide%2F113%2Fmanaging%2Fmonitor_tune%2Fperformance_controls_controlling_socket_use.htmldata=04%7C01%7Conichols%40vmware.com%7C73edfbbf4fe8435de38d08d88c271e40%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637413446927237420%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=qGAh1BfdFovLpF11C0A1jtHUwRPpQDGVO48bksfzhOw%3Dreserved=0
[2] 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgeode.apache.org%2Fdocs%2Fguide%2F113%2Fmanaging%2Fmonitor_tune%2Fsockets_and_gateways.htmldata=04%7C01%7Conichols%40vmware.com%7C73edfbbf4fe8435de38d08d88c271e40%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637413446927237420%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=kRoPhR391aHl1iJLAyrs7JhzyGqG2fJ4dTFLxdb4Zb0%3Dreserved=0

Re: [PROPOSAL] Change the default value of conserve-sockets to false

2020-11-18 Thread Udo Kohlmeyer

Hi there Donal,

Thank you for raising this. It is not an uncommon request to change the default 
value of this field.

This has been discussed many times in the past. I would LOVE to approve this 
change, but this would mean that users that don’t set this property might 
suddenly have this property changed. We are not sure that this would mean for 
these users. BUT

That said, there have been very little (to none) complaints about the product 
stability when `conserve-sockets=false` are set.

+1 - if we are allowed to make this change outside of a major version change.

--Udo

From: Donal Evans 
Date: Thursday, November 19, 2020 at 12:04 PM
To: dev@geode.apache.org 
Subject: [PROPOSAL] Change the default value of conserve-sockets to false
Hi Geode dev,

First, from the docs[1], a brief explanation of the purpose of the 
conserve-sockets property:

"The conserve-sockets setting indicates whether application threads share 
sockets with other threads or use their own sockets for member communication. 
This setting has no effect on communication between a server and its clients, 
but it does control the server’s communication with its peers or a gateway 
sender’s communication with a gateway receiver."

The current default value for the conserve-sockets property is true, which at 
first glance makes sense, since in an ideal world, existing sockets could be 
shared between threads and there would be no need to create and destroy new 
sockets for each process, which can be somewhat resource-intensive. However, in 
practice, there are several known issues with using the default setting of 
true. From the docs[1]:

"For distributed regions, the put operation, and destroy and invalidate for 
regions and entries, can all be optimized with conserve-sockets set to false. 
For partitioned regions, setting conserve-sockets to false can improve general 
throughput.
Note: When you have transactions operating on EMPTY, NORMAL or PARTITION 
regions, make sure that conserve-sockets is set to false to avoid distributed 
deadlocks."

and[2]:

"WAN deployments increase the messaging demands on a Geode system. To avoid 
hangs related to WAN messaging, always set `conserve-sockets=false` for Geode 
members that participate in a WAN deployment."

Given that it is generally accepted as best practice to set conserve-sockets to 
false for almost all use cases of Geode beyond the most simple, it would make 
sense to also change the default value to false, to prevent people having to 
encounter a problem, search for the solution, then change the setting to what 
is almost always the "correct" value.

I have done some experimenting to see what it would take to make this proposal 
a reality, and the changes required are very minimal, with only two existing 
DUnit tests that need to be modified to explicitly set the value of 
conserve-sockets that were previously relying on the default being true.

Any feedback on this proposal would be very welcome, and if the response is 
positive, I can create a PR with the changes as soon as a decision is reached.

Thanks,
Donal

[1] 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgeode.apache.org%2Fdocs%2Fguide%2F113%2Fmanaging%2Fmonitor_tune%2Fperformance_controls_controlling_socket_use.htmldata=04%7C01%7Cudo%40vmware.com%7Ca64b45dfe47446cdb7b608d88c271f1b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637413446938137469%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=yFfjxy0smP0PBWdHp9ui6XwrOGZPIfLQF5L0gkdT9kE%3Dreserved=0
[2] 
https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgeode.apache.org%2Fdocs%2Fguide%2F113%2Fmanaging%2Fmonitor_tune%2Fsockets_and_gateways.htmldata=04%7C01%7Cudo%40vmware.com%7Ca64b45dfe47446cdb7b608d88c271f1b%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C637413446938137469%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000sdata=tNb0tpFLZU%2BGEIShpFpVXy4vPUv1cCyVgCF%2B1WmMjzk%3Dreserved=0

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

Re: [PROPOSAL] Change the default value of conserve-sockets to false

26 matches

Site Navigation

Mail list logo

Footer information