codelipenghui commented on a change in pull request #5824: Publish a blog post to apache pulsar blog URL: https://github.com/apache/pulsar/pull/5824#discussion_r355368565
########## File path: site2/website/blog/2019-12-04-Apache-Pulsar-2-4-2.md ########## @@ -0,0 +1,103 @@ +--- +author: Xiaolong Ran +authorURL: https://twitter.com/wolf4j1 +title: Apache Pulsar 2.4.2 +--- + +We are glad to publish Apache Pulsar 2.4.2. This is the result of a huge +effort from the community, with over 110 commits, general improvements and bug fixes. + +Check out the <b>[release notes](../release-notes.md/)</b> for a detailed list of +the changes, with links to the relevant pull requests, discussions and documentation. + +Regarding improvements and bug fixes introduced, I just want to highlight here a tiny subset of them: + +<!--truncate--> + +- In Pulsar 2.4.1, we instead of using a shaded JAR to start a java function instance, use different classLoaders to +load the internal pulsar code, user code, and the interfaces that the two interacts with each other. This is a good +change, but this change will cause the following two problems: + - The windowed functions were broken when we changed java function instance to use classLoaders. + - When using the `--output-serde-classname` option, functionClassLoader is not set correctly. + + In Pulsar 2.4.2, we fixed this issue to make sure them works. + +- In Pulsar 2.4.1, Broker fails to start with function worker enabled and broker client using TLS. Looking at the +startup code when running the function worker with the broker, it is checking for TLS enabled in the `function_worker.yml` +file to determine whether or not to use the TLS port, but when setting TLS enabled on the function worker, +it is checking the `broker.conf`. Since the function worker is running with the broker, it makes sense to look to +the `broker.conf` as the single source of truth about whether or not to use TLS. In Pulsar 2.4.2 changed the code to +check the broker client is configured to use TLS. If it is, then use TLS for the function worker, otherwise use plain text. + +- In Pulsar Functions, we support the use of Bookkeeper to store the state of functions. But when user attempts to +fetch from function state a key that doesn't exist, an NPE will happen. In Pulsar 2.4.2, we add the correct error +handling for keys that don't exist. + +- Since deduplication drops messages based on the the largest sequence id recorded pre-persist, if there’s an error persisting +to BK, a retry attempt will just be “deduplicated” with no message ever getting persisted. The fix is two-fold: + - Double check the pending messages and return error to producer when the dup status is uncertain (eg. message is still pending) + - Sync back the lastPushed map with the lastStored map after the failures + +- In Pulsar Sinks, the data in topics is consumed from the latest location by default. But in some scenarios, users want to +consume earliest data in sink topic. In Pulsar 2.4.2, we add `--subs-position` for Pulsar Sinks, allow users to consume +data from specified locations. + +- When the cursor is recovered from a ledger, the ledgerHandle is kept open so that we can delete that ledger after +we update the cursor status. If we attempt to close the cursor before having had any updates on the cursor itself, we +would end up having a harmless error, saying that we're trying to append to a read-only ledger. Also, when we append on +the ledger during the close operation, we need to explicitly close the ledger to avoid the expensive recovery when loading the topic. + +- If the subscription type on a topic changes, a new dispatcher is created and the old one is discarded. However, this +old dispatcher is not closed. This will cause a memory leak. If cursor is not durable, the subscription is closed and removed +from the topic when all consumers are removed. The dispatcher also needs to be closed at this time. Otherwise, +RateLimiter instances will not be garbage collected, causing a memory leak. In pulsar 2.4.2, When the type of a subscription changes +and a new dispatcher is created, close the previous one, avoiding memory leaks. + +- Instead of sorting the consumers based on priority level and consumer name then pick a active consumer, which could +cause subscription getting into a flaky state, where the "active" consumer joins and leaves, no consumer is actually +elected as "active" and consuming the messages. In Pulsar 2.4.2, fixed logic to always pick the first consumer in +the consumer list without sorting consumers. So consumers will be picked as acive consumer based on the order of +their subscription. + +- In Pulsar 2.4.1, broker tries to clean up stale failed-producer from the connection however, while cleaning up Review comment: ```suggestion - Before Pulsar 2.4.2, broker tries to clean up stale failed-producer from the connection however, while cleaning up ``` ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
