Also discussing with @merlimat @srkukarni I don't think (#2438) is the best design for solving the issue. Because even with #2438, we will still have the inefficiency of submitting/updating one function requiring the publishing of all assignments which can be large.
After discussion we think, the best way to solve size and scalability issues is to use topic compaction. In the assignment topic each message will have a key that is the function id and the value with be the assignments of its instances. And we can use topic compaction to trim old entries. This solves the message size issue as well as preventing writing/sending every assignment on the update of a single assignment [ Full content available at: https://github.com/apache/incubator-pulsar/pull/2474 ] This message was relayed via gitbox.apache.org for [email protected]
