equanz commented on PR #20179:
URL: https://github.com/apache/pulsar/pull/20179#issuecomment-1539314184
@poorbarcode
* [Q-1] Yes.
* [Q-2] When I created this PR, I introduced Map structure because it was
necessary. For more details, please see the description("2. Keep the last sent
position per key.").
I just noticed we could avoid introducing a Map structure if we
implemented the last sent position feature, like the mark delete position and
the individually deleted messages feature. In other words,
- The position is already scheduled to be sent.
- All positions less than or equal to it are already scheduled to be sent.
- Manage individually sent positions to update the position as expected.
Consider the following flow.
1. Assume that the
[entries](https://github.com/apache/pulsar/blob/35e9897742b7db4bd29349940075a819b2ad6999/pulsar-broker/src/main/java/org/apache/pulsar/broker/service/persistent/PersistentStickyKeyDispatcherMultipleConsumers.java#L174)
has the following messages,
- `msg-1`, key: `key-a`, position: `1:1`
- `msg-2`, key: `key-a`, position: `1:2`
- `msg-3`, key: `key-a`, position: `1:3`
- `msg-4`, key: `key-b`, position: `1:4`
- `msg-5`, key: `key-b`, position: `1:5`
- `msg-6`, key: `key-b`, position: `1:6`
the dispatcher has two consumers (`c1` `messagesForC` is 1, `c2`
`messageForC` is 1000), and the selector will return `c1` if `key-a` and `c2`
if `key-b`.
2. Send `msg-1` to `c1` and `msg-4` - `msg-6` to `c2`.
- So, the current last sent position is `1:1` and the individually
sent positions is `[[1:3, 1:6]]` (list of closed intervals. not list of list).
- `c1` never acknowledge `msg-1`.
**scenario A**
<ol start="3">
<li>Add new consumer <code class="notranslate">c3</code>, and the
selector will return <code class="notranslate">c3</code> if <code
class="notranslate">key-a</code>.</li>
<li>Can't send <code class="notranslate">msg-2</code> - <code
class="notranslate">msg-3</code> to <code class="notranslate">c3</code> because
<code class="notranslate">1:2</code>, and <code class="notranslate">1:3</code>
are greater than the last sent position, <code
class="notranslate">1:1</code>.</li>
<li>Disconnect <code class="notranslate">c1</code>.</li>
<li>Send <code class="notranslate">msg-1</code> - <code
class="notranslate">msg-3</code> to <code
class="notranslate">c3</code>.<br>Now, <code class="notranslate">c3</code>
receives messages with expected order about <code
class="notranslate">key-a</code>.</li>
</ol>
**scenario B**
<ol start="3">
<li><code class="notranslate">c1</code> <code
class="notranslate">messagesForC</code> is back to 999.</li>
<li>Send <code class="notranslate">msg-2</code> - <code
class="notranslate">msg-3</code> to <code class="notranslate">c1</code></li>
<ul dir="auto">
<li>So, the current last sent position is <code
class="notranslate">1:6</code>, and the individually sent positions is <code
class="notranslate">[]</code>.</li>
</ul>
</ol>
My motivation is not to improve Key_Shared but fix Key_Shared.
Therefore, if the current approach is not accepted as a fix, I introduce
it and remove the feature by Map per key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
