GitHub user nickwallen opened a pull request:
https://github.com/apache/incubator-metron/pull/354
METRON-557 Create Stellar Functions for Kafka
## [METRON-557](https://issues.apache.org/jira/browse/METRON-557)
Add functions to read and write messages to Kafka topics. This should
enable a user to debug a Grok expression or Enrichment configuration inside the
REPL.
For example, I can pull a message off of the error topic to see why it
failed to parse. Then fix the configuration and resubmit the message to try
again. I also wonât have to go outside of the Stellar REPL to monitor topic
activity.
### `KAFKA_GET`
Retrieves messages from a Kafka topic. Subsequent calls will continue
retrieving messages sequentially from the original offset.
Example: Retrieve one message from a topic.
```
KAFKA_GET('topic')
```
Example: Retrieve 10 messages from a topic.
```
KAFKA_GET('topic', 10)
```
Example: Retrieve the first message in a topic. This must be the first
retrieval from the topic, otherwise the messages will be retrieved starting
from the previously stored consumer offset.
```
KAFKA_GET('topic', 1, { "auto.offset.reset": "earliest" })
```
### `KAFKA_TAIL`
Retrieves messages from a Kafka topic always starting with the most recent
message first.
Example: Retrieve the latest message from a topic.
```
KAFKA_TAIL('topic')
```
Example: Retrieve 10 messages from a topic starting with the latest.
```
KAFKA_TAIL('topic', 10)
```
### `KAFKA_PUT`
Sends messages to a Kafka topic.
Example: Put two messages on the topic 'topic'.
```
KAFKA_PUT('topic', ["message1", "message2"])
```
Example: Put a message on a topic and also define an alternative Kafka
broker.
```
KAFKA_PUT('topic', ["message1"], { "bootstrap.servers":
"kafka-broker-1:6667" })
```
### `KAFKA_PROPS`
Retrieves the Kafka properties that are used by other KAFKA_* functions
like KAFKA_GET and KAFKA_PUT. The Kafka properties are compiled from a set of
default properties, the global properties, and any overrides.
Example: Retrieve the current Kafka properties.
```
KAFKA_PROPS()
```
Example: Retrieve the current Kafka properties taking into account a set of
overrides.
```
KAFKA_PROPS({ "max.poll.records": 1 })
```
## Testing
These were tested on a local Stellar Shell in addition to running within
the "Quick Dev" environment. Note that in the "Quick Dev" environment, you
must manually copy the jar built within the Management project
(`metron-platform/metron-management/target/...`) to `/usr/metron/0.2.1BETA/lib`.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/nickwallen/incubator-metron METRON-557
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-metron/pull/354.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #354
----
commit 4c2d5daaa603405a0ba8b21b75cd55d8e59efd02
Author: Nick Allen <[email protected]>
Date: 2016-11-10T16:32:14Z
METRON-557 Create Stellar Functions for Kafka
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
