Hi Brian, Comments inline.
On Sun, Nov 10, 2019 at 5:53 PM Brian Candler <[email protected]> wrote: > I would like to know, what is the maximum size of: > > (1) a pulsar function? (which in the case of python at least, can be a > zip file) > I don't think there is any *hardcoded* limit for a pulsar function. A pulsar function user code package is uploaded to Pulsar first. It is downloaded to function workers when it is scheduled to run there. So the size limit of a function will be: 1) the function worker has enough local disk space to download the user code package. 2) the function worker has enough memory to run the function. In python case, Pulsar does support zip file. I remembered that it was documented. If you didn't find one, please create a Github issue. We will follow up and update the documentation. > > (2) a pulsar function's user-config? > The user config is saved as part of pulsar function metadata during submission. So it is subject to the pulsar message size limit. > > I know that pulsar messages are subject to a MaxMessageSize: > > https://medium.com/streamnative/whats-new-in-apache-pulsar-2-4-0-d646f6727642 > (aside: this doesn't seem to be mentioned in site2 documentation anywhere) > > But are pulsar functions and user-config distributed through the cluster > via messages, and if so, are they subject to the same limit? > > Reason: I want to install a python function with a large configuration > object (a liblognorm rulebase). If I can bundle the rulebase with the > function itself, or I can push it our via user-config, that would be > more convenient than building a separate infrastructure, e.g. the > function has to fetch it from a URL at startup. > Bundle it with function itself is probably a better solution given the size limit on user-config. Also you might consider adding the rules to the state and let your function access the rules from the state. > > Finally: it would be useful to know what happens when user-config is > updated. Does the running function pick up such changes in its context > object immediately? Is the function redeployed/restarted? > The only way to update user config is to update a function via `pulsar-admin functions update`. Once it happens, all the function instances will be restarted to load the new user-config. > > Thanks, > > Brian. > >
