yamass opened a new pull request, #19092:
URL: https://github.com/apache/iotdb/pull/19092

   ## Description
   
   ### TL;DR
   SessionPool only tracks sessions held by open result sets (the `occupied` 
map), but treats that map as "all sessions in use".
   As a result, `currentOccupiedSize()`, six setters and `close()` ignore 
sessions executing a request.
   `closeResultSet()` can also release a session that a later query already has 
taken, leaks the session when closing fails,
   and throws an NPE for a wrapper without a session.
   
   *All these bugs stem from the same root cause.* Each bug has one or more 
tests in `SessionPoolTest` that fails on master. See table below.
   
   <details>
   <summary>Longer Description</summary>
   
   ### Problem
   
   `SessionPool` keeps idle sessions in `queue`. Apart from that, it has an 
additional collection: the `occupied` map.
   It contains only sessions whose query result set is still open, until the 
result set is closed.
   A session *executing a request* – an insert, a non-query statement, or a 
query whose result set has not been returned yet – is in neither `queue` nor 
`occupied`.
   
   Note that in the following, I will just refer sessions *executing requests* 
instead of spelling out that these requests are inserts, non-query statements, 
or queries whose result set has not been returned yet.   
   
   However, several parts of the class treat `occupied` as if it held *every* 
session taken from the pool,
   which it doesn't:
   
   1. **`currentOccupiedSize()`** ignores sessions executing a request. On a 
write-heavy pool whose sessions are all
      writing, it reports 0 occupied and 0 available. We saw exactly this in 
production: "occupied" stayed at 0–1 and "available" was 0,
      all while writes queued up.
   2. **`setFetchSize`, `setTimeZone`, `setEnableRedirection`, 
`setEnableQueryRedirection`,
      `setVersion` and `setQueryTimeout`** apply the new value to `queue` and 
`occupied` only. A session executing a request
      misses the change and keeps the old value for its whole life after being 
put back, so the pool ends up with mixed settings.
   3. **`close()`** closes `queue` and `occupied` only. A session executing a 
request at that moment is later put
      back into the already cleared queue, and its connection is never closed.
   
   `closeResultSet()` has problems of its own:
   
   4. **A late second release can steal another thread's session.** A result 
set is usually released
      twice: by `SessionDataSetWrapper.hasNext()` once it is exhausted, and 
again by the caller's
      `closeResultSet()`/`close()` in a `finally` block. The second call only 
checks whether the
      *session* is still in `occupied`. If another thread took that session in 
between and opened a
      new result set with it, the late call puts the session back while the 
other thread is still
      reading, and two threads end up sharing one connection.
   5. **A failed close leaks the session.** When closing the operation handle 
fails, a replacement
      session is opened, but the failed session is neither put back nor closed, 
so its connection
      leaks. The failed close also leaves the data set open, so the second 
release (see 4) sends the
      close request again over the broken session and opens yet another 
replacement for it.
      `tryConstructNewSession()` does not count replacements in `size`, so the 
pool can end up with
      more sessions than `maxSize`.
   6. A wrapper without a session throws a `NullPointerException` 
(`ConcurrentHashMap.remove(null)`).
   
   ### Fix
   
   Only one implementation file was changed: `SessionPool`.
   
   - `occupied` is now a concurrent set of all sessions handed out by 
`getSession()`. A session leaves
     it when it is put back (`putBack`), closed after a connection error 
(`closeSession`, used by both
     `cleanSessionAndMayThrowConnectionException` overloads), or released by 
`closeResultSet()`.
   - `currentOccupiedSize()` and the timeout warning report `occupied.size()` 
(1).
   - The six setters and `close()` iterate `occupied` instead of the result-set 
sessions (2, 3).
     The setters only change client-side fields of the session, as they already 
did for sessions
     held by result sets. A session closed by `close()` makes its running 
request fail with a
     connection error, the same as reading a result set of a closed pool 
already did. The retry then
     fails with "session pool is closed". `Session.close()` returns early when 
already closed, so a
     second close by the request's error handling is harmless.
   - The former `occupied` map is renamed to `occupiedByResultSet` and maps 
each session to the
     `SessionDataSetWrapper` holding it; `occupy(session, wrapper)` stores it.
   - `closeResultSet()` first claims the session with 
`occupiedByResultSet.remove(session, wrapper)`
     and returns if that fail. So only the first release of *this* result set 
closes its operation
     handle and hands the session back (4), even if that close failed (5). A 
failed close goes through
     `cleanSessionAndMayThrowConnectionException(session)`, which closes the 
session and opens a
     replacement (5). A wrapper without a session is ignored (6).
   
   No public API changes.
   </details>
   
   ### Tests
   
   New tests in `SessionPoolTest`. Eight of them fail on master, at least one 
per bug above; the other
   four cover every way a session goes back, so the count cannot drift.
   
   | Test                                                                       
           | Covers                      | On master |
   
|---------------------------------------------------------------------------------------|-----------------------------|-----------|
   | `test_CurrentOccupiedSize_WhenInsertIsRunning_CountsItsSession`            
           | (1)                         | fails     |
   | `test_CurrentOccupiedSize_WhenNonQueryIsRunning_CountsItsSession`          
           | (1)                         | fails     |
   | `test_Setters_WhenCalledWhileInsertIsRunning_ReachSessionExecutingInsert`  
           | (2)                         | fails     |
   | `test_Close_WhenInsertIsRunning_ClosesSessionExecutingInsert`              
           | (3)                         | fails     |
   | 
`test_CloseResultSet_WhenCalledAgainAfterNextQueryTookSession_LeavesItToNextQuery`
    | (4)                         | fails     |
   | 
`test_CloseResultSet_WhenClosingOperationHandleFails_ClosesSessionAndStopsCountingIt`
 | (5)                         | fails     |
   | `test_CloseResultSet_WhenCalledAgainAfterFailedClose_DoesNotRetryClose`    
           | (5)                         | fails     |
   | `test_CloseResultSet_WhenWrapperHasNoSession_DoesNothing`                  
           | (6)                         | fails     |
   | `test_CurrentOccupiedSize_WhenInsertWasRejected_IsZero`                    
           | return via `putBack`        | passes    |
   | `test_CurrentOccupiedSize_WhenBrokenSessionWasCleanedUp_IsZero`            
           | return via cleanup          | passes    |
   | `test_CurrentOccupiedSize_WhenResultSetIsOpen_CountsItsSessionUntilClosed` 
           | return via `closeResultSet` | passes    |
   | `test_CloseResultSet_WhenCalledTwice_PutsSessionBackIntoPoolOnlyOnce`      
           | double release              | passes    |
   
   <hr>
   
   This PR has:
   - [x] been self-reviewed.
   - [x] added comments explaining the "why" and the intent of the code 
wherever would not be obvious
     for an unfamiliar reader.
   - [x] added unit tests or modified existing tests to cover new code paths, 
ensuring the threshold
     for code coverage.
   - [ ] been tested in a test IoTDB cluster.
   
   <!-- Check the items by putting "x" in the brackets for the done things. Not 
all of these items 
   apply to every PR. Remove the items which are not done or not relevant to 
the PR. None of the items 
   from the checklist above are strictly necessary, but it would be very 
helpful if you at least 
   self-review the PR. -->
   
   <hr>
   
   ##### Key changed/added classes (or packages if there are too many classes) 
in this PR
   
   * `SessionPool.java` + tests
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to