JackieTien97 commented on code in PR #19092:
URL: https://github.com/apache/iotdb/pull/19092#discussion_r4450536312
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -836,7 +850,8 @@ public synchronized void close() {
LOGGER.warn(CLOSE_THE_SESSION_FAILED, e);
}
}
- for (ISession session : occupied.keySet()) {
+ // sessions executing a request are closed as well; that request then fails
+ for (ISession session : occupied) {
Review Comment:
Closing sessions that are still in use from another thread has side effects
beyond making the running request fail:
1. **`close()` now waits for every in-flight RPC while holding the pool
monitor.** `Session.close()` → `SessionConnection.close()` sends `closeSession`
through the client from `RpcUtils.newSynchronizedClient`. `SynchronizedHandler`
holds `synchronized (client)` for the whole duration of a call, so
`closeSession` has to queue behind the in-flight call (an insert, `flush`, a
long query, …). Meanwhile, every `putBack()` blocks on the pool monitor, and so
does any `getSession()` that has to wait or create a session. With the default
`connectionTimeoutInMs = 0` there is no read timeout, so a hung DataNode or a
half-open connection makes `close()` hang indefinitely.
I reproduced this with a fake Thrift server (a mocked
`IClientRPCService.Iface` behind `TThreadPoolServer`) and a real `SessionPool`,
with one `insertTablet` held server-side for ~3 s:
- on this branch, `close()` returned after ~4 s, and another thread that
tried to synchronize on the pool was blocked the whole time;
- on master, `close()` returned immediately, leaking that session (bug 3).
2. **Sessions borrowed from `TableSessionPool` are closed under the user.**
A later `ITableSession.close()` (e.g. at the end of a try-with-resources block)
throws `IoTDBConnectionException: Session is not open, please invoke
Session.open() first` if the user changed the database. The reason is that
`TableSessionWrapper.close()` runs `use <pool database>` on the closed session
and only catches `StatementExecutionException`. Reproduced with the same fake
server: pool built with `database("db1")`, then `use db2`, then `pool.close()`,
then `tableSession.close()` throws. On master it returns normally.
3. **A retrying request can bring the closed session back to life.** If the
request is between attempts in `SessionConnection.callWithRetry` (by default up
to 60 × 500 ms), the next `reconnect()` re-inits the transport. It then calls
`session.setDefaultSessionConnection(this)` on a `Session` that is already
marked closed. From then on `Session.close()` returns early, so the new
connection is never closed: the same symptom as bug 3.
Suggestion: instead of closing sessions while they are in use, close them
when they come back to a closed pool:
- In `putBack()`, do `occupied.remove`, the `closed` check, and then
`queue.push` or `session.close()`, all inside the `synchronized (this)` block
it already has.
- In `getSession()`, close the polled session when the pool turns out to be
closed.
- Make `closed` volatile.
In-flight requests then finish normally, their connections are closed when
they are returned, and `close()` never has to wait on another thread's RPC
while holding the monitor. If you'd still like `close()` to wait for in-flight
work, a bounded wait outside the monitor would be safer. Happy to share the
probe test if it helps.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -100,8 +101,14 @@ public class SessionPool implements ISessionPool {
private static final int FINAL_RETRY = RETRY - 1;
private final ConcurrentLinkedDeque<ISession> queue = new
ConcurrentLinkedDeque<>();
- // for session whose resultSet is not released.
- private final ConcurrentMap<ISession, ISession> occupied = new
ConcurrentHashMap<>();
+ // Every session taken by getSession() and not yet returned (put back or
cleaned), whatever it
+ // is used for.
+ private final Set<ISession> occupied = ConcurrentHashMap.newKeySet();
Review Comment:
`TableSessionWrapper.close()` (not part of this diff) still bypasses this
invariant. When restoring `sql_dialect` or the database fails, it either calls
`session.close()` directly (TableSessionWrapper.java L121, L137) or lets an
`IoTDBConnectionException` escape (L114, L131). In both cases the session never
leaves `occupied`, so `currentOccupiedSize()` can still drift. In a unit test
it stays at 1 after such a failure; master reports 0.
Routing these branches through
`sessionPool.cleanSessionAndMayThrowConnectionException(session)` would keep
the count right. It would also stop the pool from losing capacity, because
`size` is never decremented on those paths today.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -802,13 +811,18 @@ public int currentAvailableSize() {
return queue.size();
}
+ /**
+ * The number of sessions currently taken from the pool: those executing a
request (insert,
+ * non-query, query, ...) and those held by a query result set that has not
been closed yet.
+ */
@Override
public int currentOccupiedSize() {
return occupied.size();
}
@SuppressWarnings({"squid:S2446"})
protected void putBack(ISession session) {
+ occupied.remove(session);
Review Comment:
`occupied.remove(session)` and `queue.push(session)` run outside the
monitor, and so do `queue.poll()` → `occupied.add(session)` in `getSession()`.
On the create path the gap is between the `closed` check (L782–L788) and L801.
While a session is between the two calls, it is in neither collection, and a
`close()` or setter that runs in that window misses it:
- `close()` doesn't close it, so it is later pushed into the cleared queue
and its connection leaks.
- A setter doesn't update it, so the session keeps the old value.
These are the same symptoms as bugs (2) and (3), just in a narrower window.
Moving the remove/push (and a `closed` check) into the existing `synchronized
(this)` block would close the window on the `putBack()` side. For the
`getSession()` side, see the suggestion on `close()`; for the setters, the pool
could also re-apply its current settings when it hands out a session.
##########
iotdb-client/session/src/test/java/org/apache/iotdb/session/pool/SessionPoolTest.java:
##########
@@ -1634,4 +1634,223 @@ private List<ByteBuffer> FakedFirstFetchTsBlockResult()
{
return Collections.singletonList(tsBlock);
}
+
+ @Test
+ public void test_CurrentOccupiedSize_WhenInsertIsRunning_CountsItsSession()
+ throws IoTDBConnectionException, StatementExecutionException {
+ int[] occupiedDuringInsert = {-1};
+ Mockito.doAnswer(
+ invocation -> {
+ occupiedDuringInsert[0] = sessionPool.currentOccupiedSize();
+ return null;
+ })
+ .when(session)
+ .insertTablet(any(Tablet.class), Mockito.anyBoolean());
+
+ sessionPool.insertTablet(createBooleanTablet());
+
+ assertEquals(1, occupiedDuringInsert[0]);
+ assertEquals(0, sessionPool.currentOccupiedSize());
+ assertEquals(1, sessionPool.currentAvailableSize());
+ }
+
+ @Test
+ public void test_CurrentOccupiedSize_WhenNonQueryIsRunning_CountsItsSession()
+ throws IoTDBConnectionException, StatementExecutionException {
+ int[] occupiedDuringNonQuery = {-1};
+ Mockito.doAnswer(
+ invocation -> {
+ occupiedDuringNonQuery[0] = sessionPool.currentOccupiedSize();
+ return null;
+ })
+ .when(session)
+ .executeNonQueryStatement(any(String.class));
+
+ sessionPool.executeNonQueryStatement("flush");
+
+ assertEquals(1, occupiedDuringNonQuery[0]);
+ assertEquals(0, sessionPool.currentOccupiedSize());
+ }
+
+ @Test
+ public void test_CurrentOccupiedSize_WhenInsertWasRejected_IsZero()
+ throws IoTDBConnectionException, StatementExecutionException {
+ doThrow(new StatementExecutionException("rejected"))
+ .when(session)
+ .insertTablet(any(Tablet.class), Mockito.anyBoolean());
+
+ try {
+ sessionPool.insertTablet(createBooleanTablet());
+ Assert.fail("expected StatementExecutionException");
+ } catch (StatementExecutionException e) {
+ // expected
+ }
+
+ assertEquals(0, sessionPool.currentOccupiedSize());
+ assertEquals(1, sessionPool.currentAvailableSize());
+ }
+
+ @Test
+ public void test_CurrentOccupiedSize_WhenBrokenSessionWasCleanedUp_IsZero()
+ throws IoTDBConnectionException, StatementExecutionException {
+ doThrow(new IoTDBConnectionException("broken"))
+ .when(session)
+ .insertTablet(any(Tablet.class), Mockito.anyBoolean());
+
+ try {
+ sessionPool.insertTablet(createBooleanTablet());
+ Assert.fail("expected IoTDBConnectionException");
+ } catch (IoTDBConnectionException e) {
+ // expected
+ }
+
+ assertEquals(0, sessionPool.currentOccupiedSize());
+ }
+
+ @Test
+ public void
test_CurrentOccupiedSize_WhenResultSetIsOpen_CountsItsSessionUntilClosed()
+ throws IoTDBConnectionException, StatementExecutionException {
+ SessionDataSet dataSet = Mockito.mock(SessionDataSet.class);
+
Mockito.when(session.executeQueryStatement(any(String.class))).thenReturn(dataSet);
+
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement("select
* from root.**");
+ assertEquals(1, sessionPool.currentOccupiedSize());
+
+ sessionPool.closeResultSet(wrapper);
+ assertEquals(0, sessionPool.currentOccupiedSize());
+ assertEquals(1, sessionPool.currentAvailableSize());
+ }
+
+ @Test
+ public void
test_CloseResultSet_WhenCalledTwice_PutsSessionBackIntoPoolOnlyOnce()
+ throws IoTDBConnectionException, StatementExecutionException {
+ SessionDataSet dataSet = Mockito.mock(SessionDataSet.class);
+
Mockito.when(session.executeQueryStatement(any(String.class))).thenReturn(dataSet);
+
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement("select
* from root.**");
+ sessionPool.closeResultSet(wrapper);
+ sessionPool.closeResultSet(wrapper);
+
+ assertEquals(0, sessionPool.currentOccupiedSize());
+ assertEquals(1, sessionPool.currentAvailableSize());
+ }
+
+ @Test
+ public void
test_CloseResultSet_WhenCalledAgainAfterNextQueryTookSession_LeavesItToNextQuery()
+ throws IoTDBConnectionException, StatementExecutionException {
+ SessionDataSet firstDataSet = Mockito.mock(SessionDataSet.class);
+ SessionDataSet secondDataSet = Mockito.mock(SessionDataSet.class);
+ Mockito.when(session.executeQueryStatement(any(String.class)))
+ .thenReturn(firstDataSet, secondDataSet);
+
+ SessionDataSetWrapper first = sessionPool.executeQueryStatement("select *
from root.a");
+ sessionPool.closeResultSet(first); // as SessionDataSetWrapper.hasNext()
does when exhausted
+ SessionDataSetWrapper second = sessionPool.executeQueryStatement("select *
from root.b");
+ Assert.assertSame(first.getSession(), second.getSession());
+
+ sessionPool.closeResultSet(first); // e.g. in the caller's finally block
+
+ assertEquals(0, sessionPool.currentAvailableSize());
+ assertEquals(1, sessionPool.currentOccupiedSize());
+ Mockito.verify(secondDataSet, Mockito.never()).closeOperationHandle();
+
+ sessionPool.closeResultSet(second);
+ assertEquals(0, sessionPool.currentOccupiedSize());
+ assertEquals(1, sessionPool.currentAvailableSize());
+ }
+
+ @Test
+ public void
test_CloseResultSet_WhenClosingOperationHandleFails_ClosesSessionAndStopsCountingIt()
+ throws IoTDBConnectionException, StatementExecutionException {
+ SessionDataSet dataSet = Mockito.mock(SessionDataSet.class);
+ doThrow(new
IoTDBConnectionException("broken")).when(dataSet).closeOperationHandle();
+
Mockito.when(session.executeQueryStatement(any(String.class))).thenReturn(dataSet);
+
+ SessionDataSetWrapper wrapper = sessionPool.executeQueryStatement("select
* from root.**");
+ sessionPool.closeResultSet(wrapper);
+
+ assertEquals(0, sessionPool.currentOccupiedSize());
+ assertEquals(0, sessionPool.currentAvailableSize());
Review Comment:
Minor: this test,
`test_CurrentOccupiedSize_WhenBrokenSessionWasCleanedUp_IsZero` and
`test_CloseResultSet_WhenCalledAgainAfterFailedClose_DoesNotRetryClose` all
reach `tryConstructNewSession()`. Each of them therefore opens a real `Session`
to the fake `host:11`, which took 0.7–1.3 s per test locally, spent waiting for
DNS resolution to fail.
This assertion effectively checks that the replacement connection failed,
which depends on the environment. With the default `connectionTimeoutInMs = 0`,
a resolver that maps `host` to an address that silently drops packets would
also make these tests hang until the OS TCP timeout. Consider one of these:
- point the pool at `127.0.0.1` and a port with nothing listening, so the
replacement fails fast without a DNS lookup;
- assert on `occupied` instead of `currentAvailableSize()`.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -3077,7 +3096,7 @@ public SessionDataSetWrapper executeQueryStatement(String
sql)
try {
SessionDataSet resp = session.executeQueryStatement(sql);
SessionDataSetWrapper wrapper = new SessionDataSetWrapper(resp,
session, this);
- occupy(session);
+ occupy(session, wrapper);
Review Comment:
Nit: wrapper construction followed by `occupy(session, wrapper)` is now
repeated in 12 query methods. A small helper such as `wrapResultSet(resp,
session)` that builds and registers the wrapper would replace `occupy()`. It
would also keep a future query method from forgetting the registration; if that
happened, `closeResultSet()` would silently return and the session would never
go back to the pool.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -3522,7 +3541,7 @@ public void setTimeZone(String zoneId)
for (ISession session : queue) {
session.setTimeZoneOfSession(zoneId);
}
- for (ISession session : occupied.keySet()) {
+ for (ISession session : occupied) {
session.setTimeZoneOfSession(zoneId);
Review Comment:
`closeSession()` removes a session from `occupied` and then closes it, but
this loop uses a weakly consistent iterator, so it can still return a session
that another thread has just removed and closed. That happens after a request
fails with a connection error, or on the new `closeResultSet()` failure path.
`Session.setTimeZoneOfSession()` then throws `IoTDBConnectionException("Session
is not open")` from `getDefaultSessionConnection()`. By that point the
server-side time zone and `this.zoneId` have already changed, but the rest of
`occupied` is skipped, so the pool ends up with mixed time zones again.
Catching and ignoring the failure per session would make the loop robust; that
session is going away anyway. This is the only one of the six setters that can
throw; the others only assign fields.
Unrelated to this PR, but in the same method: the retry loop above never
`break`s or `return`s after a successful `session.setTimeZone(zoneId);
putBack(session);`, so it sends the RPC `RETRY` (3) times. If the 3rd attempt
hits a connection error, `cleanSessionAndMayThrowConnectionException(session,
FINAL_RETRY, e)` throws, even though the time zone was already set.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/pool/SessionPool.java:
##########
@@ -855,23 +870,26 @@ public synchronized void close() {
LOGGER.info(SessionMessages.CLOSING_SESSION_POOL);
this.closed = true;
queue.clear();
+ occupiedByResultSet.clear();
occupied.clear();
}
@Override
public void closeResultSet(SessionDataSetWrapper wrapper) {
- boolean putback = true;
+ ISession session = wrapper.getSession();
+ // SessionDataSetWrapper.hasNext() already releases an exhausted result
set, so this is often
+ // called twice for the same one. Only the first call may hand its session
back; a later one
+ // must not release the session again, which may already hold another
result set by then.
+ if (session == null || !occupiedByResultSet.remove(session, wrapper)) {
+ return;
+ }
try {
wrapper.getSessionDataSet().closeOperationHandle();
Review Comment:
Minor: the old `finally` guaranteed that the session was always put back or
dropped. Now the session is claimed first (L883), and only
`IoTDBConnectionException`/`StatementExecutionException` lead to cleanup. Any
other `RuntimeException`/`Error` thrown here leaves the session neither put
back nor closed, while it is still counted in `occupied`. One example is an NPE
when `getSessionDataSet()` is null (`setSessionDataSet` is public). This is
unlikely, because `SynchronizedHandler` wraps RPC-level failures into
`TException`. Still, keeping a `try`/`finally` around the release, or also
catching `RuntimeException`, would be cheap insurance.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]