JackieTien97 commented on code in PR #19092:
URL: https://github.com/apache/iotdb/pull/19092#discussion_r4450536312


##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -836,7 +850,8 @@ public synchronized void close() {
         LOGGER.warn(CLOSE_THE_SESSION_FAILED, e);
       }
     }
-    for (ISession session : occupied.keySet()) {
+    // sessions executing a request are closed as well; that request then fails
+    for (ISession session : occupied) {

Review Comment:
   Closing sessions that are still in use from another thread has side effects 
beyond making the running request fail:
   
   1. **`close()` now waits for every in-flight RPC while holding the pool 
monitor.** `Session.close()` → `SessionConnection.close()` sends `closeSession` 
through the client from `RpcUtils.newSynchronizedClient`. `SynchronizedHandler` 
holds `synchronized (client)` for the whole duration of a call, so 
`closeSession` has to queue behind the in-flight call (an insert, `flush`, a 
long query, …). Meanwhile, every `putBack()` blocks on the pool monitor, and so 
does any `getSession()` that has to wait or create a session. With the default 
`connectionTimeoutInMs = 0` there is no read timeout, so a hung DataNode or a 
half-open connection makes `close()` hang indefinitely.
      I reproduced this with a fake Thrift server (a mocked 
`IClientRPCService.Iface` behind `TThreadPoolServer`) and a real `SessionPool`, 
with one `insertTablet` held server-side for ~3 s:
      - on this branch, `close()` returned after ~4 s, and another thread that 
tried to synchronize on the pool was blocked the whole time;
      - on master, `close()` returned immediately, leaking that session (bug 3).
   2. **Sessions borrowed from `TableSessionPool` are closed under the user.** 
A later `ITableSession.close()` (e.g. at the end of a try-with-resources block) 
throws `IoTDBConnectionException: Session is not open, please invoke 
Session.open() first` if the user changed the database. The reason is that 
`TableSessionWrapper.close()` runs `use <pool database>` on the closed session 
and only catches `StatementExecutionException`. Reproduced with the same fake 
server: pool built with `database("db1")`, then `use db2`, then `pool.close()`, 
then `tableSession.close()` throws. On master it returns normally.
   3. **A retrying request can bring the closed session back to life.** If the 
request is between attempts in `SessionConnection.callWithRetry` (by default up 
to 60 × 500 ms), the next `reconnect()` re-inits the transport. It then calls 
`session.setDefaultSessionConnection(this)` on a `Session` that is already 
marked closed. From then on `Session.close()` returns early, so the new 
connection is never closed: the same symptom as bug 3.
   
   Suggestion: instead of closing sessions while they are in use, close them 
when they come back to a closed pool:
   - In `putBack()`, do `occupied.remove`, the `closed` check, and then 
`queue.push` or `session.close()`, all inside the `synchronized (this)` block 
it already has.
   - In `getSession()`, close the polled session when the pool turns out to be 
closed.
   - Make `closed` volatile.
   
   In-flight requests then finish normally, their connections are closed when 
they are returned, and `close()` never has to wait on another thread's RPC 
while holding the monitor. If you'd still like `close()` to wait for in-flight 
work, a bounded wait outside the monitor would be safer. Happy to share the 
probe test if it helps.



##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -100,8 +101,14 @@ public class SessionPool implements ISessionPool {
   private static final int FINAL_RETRY = RETRY - 1;
 
   private final ConcurrentLinkedDeque<ISession> queue = new 
ConcurrentLinkedDeque<>();
-  // for session whose resultSet is not released.
-  private final ConcurrentMap<ISession, ISession> occupied = new 
ConcurrentHashMap<>();
+  // Every session taken by getSession() and not yet returned (put back or 
cleaned), whatever it
+  // is used for.
+  private final Set<ISession> occupied = ConcurrentHashMap.newKeySet();

Review Comment:
   `TableSessionWrapper.close()` (not part of this diff) still bypasses this 
invariant. When restoring `sql_dialect` or the database fails, it either calls 
`session.close()` directly (TableSessionWrapper.java L121, L137) or lets an 
`IoTDBConnectionException` escape (L114, L131). In both cases the session never 
leaves `occupied`, so `currentOccupiedSize()` can still drift. In a unit test 
it stays at 1 after such a failure; master reports 0.
   
   Routing these branches through 
`sessionPool.cleanSessionAndMayThrowConnectionException(session)` would keep 
the count right. It would also stop the pool from losing capacity, because 
`size` is never decremented on those paths today.



##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -802,13 +811,18 @@ public int currentAvailableSize() {
     return queue.size();
   }
 
+  /**
+   * The number of sessions currently taken from the pool: those executing a 
request (insert,
+   * non-query, query, ...) and those held by a query result set that has not 
been closed yet.
+   */
   @Override
   public int currentOccupiedSize() {
     return occupied.size();
   }
 
   @SuppressWarnings({"squid:S2446"})
   protected void putBack(ISession session) {
+    occupied.remove(session);

Review Comment:
   `occupied.remove(session)` and `queue.push(session)` run outside the 
monitor, and so do `queue.poll()` → `occupied.add(session)` in `getSession()`. 
On the create path the gap is between the `closed` check (L782–L788) and L801. 
While a session is between the two calls, it is in neither collection, and a 
`close()` or setter that runs in that window misses it:
   - `close()` doesn't close it, so it is later pushed into the cleared queue 
and its connection leaks.
   - A setter doesn't update it, so the session keeps the old value.
   
   These are the same symptoms as bugs (2) and (3), just in a narrower window. 
Moving the remove/push (and a `closed` check) into the existing `synchronized 
(this)` block would close the window on the `putBack()` side. For the 
`getSession()` side, see the suggestion on `close()`; for the setters, the pool 
could also re-apply its current settings when it hands out a session.



##########
iotdb-client/session/src/test/java/org/apache/iotdb/session/pool/SessionPoolTest.java:
##########
@@ -1634,4 +1634,223 @@ private List<ByteBuffer> FakedFirstFetchTsBlockResult() 
{
 
     return Collections.singletonList(tsBlock);
   }
+
+  @Test
+  public void test_CurrentOccupiedSize_WhenInsertIsRunning_CountsItsSession()
+      throws IoTDBConnectionException, StatementExecutionException {
+    int[] occupiedDuringInsert = {-1};
+    Mockito.doAnswer(
+            invocation -> {
+              occupiedDuringInsert[0] = sessionPool.currentOccupiedSize();
+              return null;
+            })
+        .when(session)
+        .insertTablet(any(Tablet.class), Mockito.anyBoolean());
+
+    sessionPool.insertTablet(createBooleanTablet());
+
+    assertEquals(1, occupiedDuringInsert[0]);
+    assertEquals(0, sessionPool.currentOccupiedSize());
+    assertEquals(1, sessionPool.currentAvailableSize());
+  }
+
+  @Test
+  public void test_CurrentOccupiedSize_WhenNonQueryIsRunning_CountsItsSession()
+      throws IoTDBConnectionException, StatementExecutionException {
+    int[] occupiedDuringNonQuery = {-1};
+    Mockito.doAnswer(
+            invocation -> {
+              occupiedDuringNonQuery[0] = sessionPool.currentOccupiedSize();
+              return null;
+            })
+        .when(session)
+        .executeNonQueryStatement(any(String.class));
+
+    sessionPool.executeNonQueryStatement("flush");
+
+    assertEquals(1, occupiedDuringNonQuery[0]);
+    assertEquals(0, sessionPool.currentOccupiedSize());
+  }
+
+  @Test
+  public void test_CurrentOccupiedSize_WhenInsertWasRejected_IsZero()
+      throws IoTDBConnectionException, StatementExecutionException {
+    doThrow(new StatementExecutionException("rejected"))
+        .when(session)
+        .insertTablet(any(Tablet.class), Mockito.anyBoolean());
+
+    try {
+      sessionPool.insertTablet(createBooleanTablet());
+      Assert.fail("expected StatementExecutionException");
+    } catch (StatementExecutionException e) {
+      // expected
+    }
+
+    assertEquals(0, sessionPool.currentOccupiedSize());
+    assertEquals(1, sessionPool.currentAvailableSize());
+  }
+
+  @Test
+  public void test_CurrentOccupiedSize_WhenBrokenSessionWasCleanedUp_IsZero()
+      throws IoTDBConnectionException, StatementExecutionException {
+    doThrow(new IoTDBConnectionException("broken"))
+        .when(session)
+        .insertTablet(any(Tablet.class), Mockito.anyBoolean());
+
+    try {
+      sessionPool.insertTablet(createBooleanTablet());
+      Assert.fail("expected IoTDBConnectionException");
+    } catch (IoTDBConnectionException e) {
+      // expected
+    }
+
+    assertEquals(0, sessionPool.currentOccupiedSize());
+  }
+
+  @Test
+  public void 
test_CurrentOccupiedSize_WhenResultSetIsOpen_CountsItsSessionUntilClosed()
+      throws IoTDBConnectionException, StatementExecutionException {
+    SessionDataSet dataSet = Mockito.mock(SessionDataSet.class);
+    
Mockito.when(session.executeQueryStatement(any(String.class))).thenReturn(dataSet);
+
+    SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement("select 
* from root.**");
+    assertEquals(1, sessionPool.currentOccupiedSize());
+
+    sessionPool.closeResultSet(wrapper);
+    assertEquals(0, sessionPool.currentOccupiedSize());
+    assertEquals(1, sessionPool.currentAvailableSize());
+  }
+
+  @Test
+  public void 
test_CloseResultSet_WhenCalledTwice_PutsSessionBackIntoPoolOnlyOnce()
+      throws IoTDBConnectionException, StatementExecutionException {
+    SessionDataSet dataSet = Mockito.mock(SessionDataSet.class);
+    
Mockito.when(session.executeQueryStatement(any(String.class))).thenReturn(dataSet);
+
+    SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement("select 
* from root.**");
+    sessionPool.closeResultSet(wrapper);
+    sessionPool.closeResultSet(wrapper);
+
+    assertEquals(0, sessionPool.currentOccupiedSize());
+    assertEquals(1, sessionPool.currentAvailableSize());
+  }
+
+  @Test
+  public void 
test_CloseResultSet_WhenCalledAgainAfterNextQueryTookSession_LeavesItToNextQuery()
+      throws IoTDBConnectionException, StatementExecutionException {
+    SessionDataSet firstDataSet = Mockito.mock(SessionDataSet.class);
+    SessionDataSet secondDataSet = Mockito.mock(SessionDataSet.class);
+    Mockito.when(session.executeQueryStatement(any(String.class)))
+        .thenReturn(firstDataSet, secondDataSet);
+
+    SessionDataSetWrapper first = sessionPool.executeQueryStatement("select * 
from root.a");
+    sessionPool.closeResultSet(first); // as SessionDataSetWrapper.hasNext() 
does when exhausted
+    SessionDataSetWrapper second = sessionPool.executeQueryStatement("select * 
from root.b");
+    Assert.assertSame(first.getSession(), second.getSession());
+
+    sessionPool.closeResultSet(first); // e.g. in the caller's finally block
+
+    assertEquals(0, sessionPool.currentAvailableSize());
+    assertEquals(1, sessionPool.currentOccupiedSize());
+    Mockito.verify(secondDataSet, Mockito.never()).closeOperationHandle();
+
+    sessionPool.closeResultSet(second);
+    assertEquals(0, sessionPool.currentOccupiedSize());
+    assertEquals(1, sessionPool.currentAvailableSize());
+  }
+
+  @Test
+  public void 
test_CloseResultSet_WhenClosingOperationHandleFails_ClosesSessionAndStopsCountingIt()
+      throws IoTDBConnectionException, StatementExecutionException {
+    SessionDataSet dataSet = Mockito.mock(SessionDataSet.class);
+    doThrow(new 
IoTDBConnectionException("broken")).when(dataSet).closeOperationHandle();
+    
Mockito.when(session.executeQueryStatement(any(String.class))).thenReturn(dataSet);
+
+    SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement("select 
* from root.**");
+    sessionPool.closeResultSet(wrapper);
+
+    assertEquals(0, sessionPool.currentOccupiedSize());
+    assertEquals(0, sessionPool.currentAvailableSize());

Review Comment:
   Minor: this test, 
`test_CurrentOccupiedSize_WhenBrokenSessionWasCleanedUp_IsZero` and 
`test_CloseResultSet_WhenCalledAgainAfterFailedClose_DoesNotRetryClose` all 
reach `tryConstructNewSession()`. Each of them therefore opens a real `Session` 
to the fake `host:11`, which took 0.7–1.3 s per test locally, spent waiting for 
DNS resolution to fail.
   
   This assertion effectively checks that the replacement connection failed, 
which depends on the environment. With the default `connectionTimeoutInMs = 0`, 
a resolver that maps `host` to an address that silently drops packets would 
also make these tests hang until the OS TCP timeout. Consider one of these:
   - point the pool at `127.0.0.1` and a port with nothing listening, so the 
replacement fails fast without a DNS lookup;
   - assert on `occupied` instead of `currentAvailableSize()`.



##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -3077,7 +3096,7 @@ public SessionDataSetWrapper executeQueryStatement(String 
sql)
       try {
         SessionDataSet resp = session.executeQueryStatement(sql);
         SessionDataSetWrapper wrapper = new SessionDataSetWrapper(resp, 
session, this);
-        occupy(session);
+        occupy(session, wrapper);

Review Comment:
   Nit: wrapper construction followed by `occupy(session, wrapper)` is now 
repeated in 12 query methods. A small helper such as `wrapResultSet(resp, 
session)` that builds and registers the wrapper would replace `occupy()`. It 
would also keep a future query method from forgetting the registration; if that 
happened, `closeResultSet()` would silently return and the session would never 
go back to the pool.



##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -3522,7 +3541,7 @@ public void setTimeZone(String zoneId)
     for (ISession session : queue) {
       session.setTimeZoneOfSession(zoneId);
     }
-    for (ISession session : occupied.keySet()) {
+    for (ISession session : occupied) {
       session.setTimeZoneOfSession(zoneId);

Review Comment:
   `closeSession()` removes a session from `occupied` and then closes it, but 
this loop uses a weakly consistent iterator, so it can still return a session 
that another thread has just removed and closed. That happens after a request 
fails with a connection error, or on the new `closeResultSet()` failure path. 
`Session.setTimeZoneOfSession()` then throws `IoTDBConnectionException("Session 
is not open")` from `getDefaultSessionConnection()`. By that point the 
server-side time zone and `this.zoneId` have already changed, but the rest of 
`occupied` is skipped, so the pool ends up with mixed time zones again. 
Catching and ignoring the failure per session would make the loop robust; that 
session is going away anyway. This is the only one of the six setters that can 
throw; the others only assign fields.
   
   Unrelated to this PR, but in the same method: the retry loop above never 
`break`s or `return`s after a successful `session.setTimeZone(zoneId); 
putBack(session);`, so it sends the RPC `RETRY` (3) times. If the 3rd attempt 
hits a connection error, `cleanSessionAndMayThrowConnectionException(session, 
FINAL_RETRY, e)` throws, even though the time zone was already set.



##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -855,23 +870,26 @@ public synchronized void close() {
     LOGGER.info(SessionMessages.CLOSING_SESSION_POOL);
     this.closed = true;
     queue.clear();
+    occupiedByResultSet.clear();
     occupied.clear();
   }
 
   @Override
   public void closeResultSet(SessionDataSetWrapper wrapper) {
-    boolean putback = true;
+    ISession session = wrapper.getSession();
+    // SessionDataSetWrapper.hasNext() already releases an exhausted result 
set, so this is often
+    // called twice for the same one. Only the first call may hand its session 
back; a later one
+    // must not release the session again, which may already hold another 
result set by then.
+    if (session == null || !occupiedByResultSet.remove(session, wrapper)) {
+      return;
+    }
     try {
       wrapper.getSessionDataSet().closeOperationHandle();

Review Comment:
   Minor: the old `finally` guaranteed that the session was always put back or 
dropped. Now the session is claimed first (L883), and only 
`IoTDBConnectionException`/`StatementExecutionException` lead to cleanup. Any 
other `RuntimeException`/`Error` thrown here leaves the session neither put 
back nor closed, while it is still counted in `occupied`. One example is an NPE 
when `getSessionDataSet()` is null (`setSessionDataSet` is public). This is 
unlikely, because `SynchronizedHandler` wraps RPC-level failures into 
`TException`. Still, keeping a `try`/`finally` around the release, or also 
catching `RuntimeException`, would be cheap insurance.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to