This is an automated email from the ASF dual-hosted git repository. sijie pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/incubator-pulsar.git
The following commit(s) were added to refs/heads/master by this push: new cb4f6ad Pulsar Functions resource docs (#1760) cb4f6ad is described below commit cb4f6ad71f6365b5369ca9705c3bf6985335995a Author: Luc Perkins <lucperk...@gmail.com> AuthorDate: Sat Jun 9 14:46:40 2018 -0500 Pulsar Functions resource docs (#1760) * add section on function resources to overview * add resource flags to YAML * add section to deployment doc * add runtime info --- site/_data/cli/pulsar-admin.yaml | 12 ++++++++++ site/docs/latest/admin-api/persistent-topics.md | 2 -- site/docs/latest/functions/deployment.md | 26 ++++++++++++++++++++- site/docs/latest/functions/overview.md | 30 ++++++++++++++++++++++--- 4 files changed, 64 insertions(+), 6 deletions(-) diff --git a/site/_data/cli/pulsar-admin.yaml b/site/_data/cli/pulsar-admin.yaml index 110be5c..9470d7c 100644 --- a/site/_data/cli/pulsar-admin.yaml +++ b/site/_data/cli/pulsar-admin.yaml @@ -120,6 +120,12 @@ commands: - name: localrun description: Run a Pulsar Function locally options: + - flags: --cpu + description: The CPU to allocate to each function instance (in number of cores) + - flags: --ram + description: The RAM to allocate to each function instance (in bytes) + - flags: --disk + description: The disk space to allocate to each function instance (in bytes) - flags: --brokerServiceUrl description: The URL of the Pulsar broker - flags: --className @@ -163,6 +169,12 @@ commands: - name: create description: Creates a new Pulsar Function on the target infrastructure options: + - flags: --cpu + description: The CPU to allocate to each function instance (in number of cores) + - flags: --ram + description: The RAM to allocate to each function instance (in bytes) + - flags: --disk + description: The disk space to allocate to each function instance (in bytes) - flags: --className description: The name of the function's class - flags: --customSerdeInputs diff --git a/site/docs/latest/admin-api/persistent-topics.md b/site/docs/latest/admin-api/persistent-topics.md index 452ece9..4dbd73b 100644 --- a/site/docs/latest/admin-api/persistent-topics.md +++ b/site/docs/latest/admin-api/persistent-topics.md @@ -93,8 +93,6 @@ Permission can be fetched using [`permissions`](../../reference/CliTools#permiss #### pulsar-admin -TODO: admin - ```shell $ pulsar-admin persistent permissions \ persistent://test-tenant/ns1/tp1 \ diff --git a/site/docs/latest/functions/deployment.md b/site/docs/latest/functions/deployment.md index c0871bf..aceb7b0 100644 --- a/site/docs/latest/functions/deployment.md +++ b/site/docs/latest/functions/deployment.md @@ -40,7 +40,7 @@ When managing Pulsar Functions, you'll need to specify a variety of information Parameter | Default :---------|:------- -Function name | Whichever value is specified for the class name. For example, `--className org.example.MyFunction` would give the function a name of `MyFunction` +Function name | Whichever value is specified for the class name (minus org, library, etc.). The flag `--className org.example.MyFunction`, for example, would give the function a name of `MyFunction`. Tenant | Derived from the input topics' names. If the input topics are under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/{namespace}/{topicName}`---then the tenant will be `marketing`. Namespace | Derived from the input topics' names. If the input topics are under the `asia` namespace under the `marketing` tenant---i.e. the topic names have the form `persistent://marketing/asia/{topicName}`, then the namespace will be `asia`. Output topic | `{input topic}-{function name}-output`. A function with an input topic name of `incoming` and a function name of `exclamation`, for example, would have an output topic of `incoming-exclamation-output`. @@ -143,6 +143,30 @@ $ bin/pulsar-admin functions update \ --functionConfigFile function-config.yaml ``` +### Function instance resources {#resources} + +When you run Pulsar Functions in [cluster run](#cluster-run) mode, you can specify the resources that are assigned to each function [instance](#parallelism): + +Resource | Specified as... | Runtimes +:--------|:----------------|:-------- +CPU | The number of cores | Docker (coming soon) +RAM | The number of bytes | Process, Docker +Disk space | The number of bytes | Docker + +Here's an example function creation command that allocates 8 cores, 8 GB of RAM, and 10 GB of disk space to a function: + +```bash +$ bin/pulsar-admin functions create \ + --jar target/my-functions.jar \ + --className org.example.functions.MyFunction \ + --cpu 8 \ + --ram 8589934592 \ + --disk 10737418240 +``` + +{% include admonition.html type="warning" title="Resources are *per instance*" + content="The resources that you apply to a given Pulsar Function are applied to each [instance](#parallelism) of the function. If you apply 8 GB of RAM to a function with a paralellism of 5, for example, then you are applying 40 GB of RAM total for the function. You should always make sure to factor paralellism---i.e. the number of instances---into your resource calculations." %} + ## Triggering Pulsar Functions {#triggering} If a Pulsar Function is running in [cluster mode](#cluster-mode), you can **trigger** it at any time using the command line. Triggering a function means that you send a message with a specific value to the function and get the function's output (if any) via the command line. diff --git a/site/docs/latest/functions/overview.md b/site/docs/latest/functions/overview.md index d9780ee..8654b95 100644 --- a/site/docs/latest/functions/overview.md +++ b/site/docs/latest/functions/overview.md @@ -194,9 +194,10 @@ Pulsar Functions can currently be written in [Java](../../functions/api#java) an ## The Pulsar Functions API {#api} -* Type safe (bytes versus specific types) -* SerDe (built-in vs. custom) -* Pulsar messages are always just bytes, but Pulsar Functions handles data types for you *unless* you need custom types +The Pulsar Functions API enables you to create processing logic that is: + +* Type safe. Pulsar Functions can process raw bytes or more complex, application-specific types. +* Based on SerDe (**Ser**ialization/**De**serialization). A variety of types are supported "out of the box" but you can also create your own custom SerDe logic. ### Function context {#context} @@ -317,6 +318,29 @@ $ bin/pulsar-admin functions create \ --parallelism 5 ``` +### Function instance resources {#resources} + +When you run Pulsar Functions in [cluster run](#cluster-run) mode, you can specify the resources that are assigned to each function [instance](#parallelism): + +Resource | Specified as... | Runtimes +:--------|:----------------|:-------- +CPU | The number of cores | Docker (coming soon) +RAM | The number of bytes | Process, Docker +Disk space | The number of bytes | Docker + +Here's an example function creation command that allocates 8 cores, 8 GB of RAM, and 10 GB of disk space to a function: + +```bash +$ bin/pulsar-admin functions create \ + --jar target/my-functions.jar \ + --className org.example.functions.MyFunction \ + --cpu 8 \ + --ram 8589934592 \ + --disk 10737418240 +``` + +For more information on resources, see the [Deploying and Managing Pulsar Functions](../deployment#resources) documentation. + ### Logging Pulsar Functions created using the [Pulsar Functions SDK(#sdk) can send logs to a log topic that you specify as part of the function's configuration. The function created using the command below, for example, would produce all logs on the `persistent://public/default/my-func-1-log` topic: -- To stop receiving notification emails like this one, please contact si...@apache.org.