Repository: incubator-metron Updated Branches: refs/heads/master 001b5cc5b -> 4a4cb8b11
METRON-435: Create Stellar REPL (nickwallen via cestella) closes apache/incubator-metron#262 Project: http://git-wip-us.apache.org/repos/asf/incubator-metron/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-metron/commit/4a4cb8b1 Tree: http://git-wip-us.apache.org/repos/asf/incubator-metron/tree/4a4cb8b1 Diff: http://git-wip-us.apache.org/repos/asf/incubator-metron/diff/4a4cb8b1 Branch: refs/heads/master Commit: 4a4cb8b117dbb66bbfb4915bca9d871a06682c28 Parents: 001b5cc Author: nickwallen <n...@nickallen.org> Authored: Tue Sep 20 16:56:45 2016 -0400 Committer: cstella <ceste...@gmail.com> Committed: Tue Sep 20 16:56:45 2016 -0400 ---------------------------------------------------------------------- .../metron/common/stellar/shell/README.md | 126 ++++++++++ .../common/stellar/shell/StellarExecutor.java | 152 +++++++++++ .../common/stellar/shell/StellarShell.java | 249 +++++++++++++++++++ .../metron-common/src/main/scripts/stellar | 34 +++ 4 files changed, 561 insertions(+) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/4a4cb8b1/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/README.md ---------------------------------------------------------------------- diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/README.md b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/README.md new file mode 100644 index 0000000..32bad31 --- /dev/null +++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/README.md @@ -0,0 +1,126 @@ +# Stellar Shell + +A REPL (Read Eval Print Loop) for the Stellar language that helps in debugging, troubleshooting and learning Stellar. The Stellar DSL (domain specific language) is used to act upon streaming data within Apache Storm. It is difficult to troubleshoot Stellar when it can only be executed within a Storm topology. This REPL is intended to help mitigate that problem by allowing a user to replicate data encountered in production, isolate initialization errors, or understand function resolution problems. + +### Getting Started + +``` +$ /usr/metron/<version>/bin/stellar + +Stellar, Go! +{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH} + +>>> %functions +BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, ... + +>>> ?PROTOCOL_TO_NAME +PROTOCOL_TO_NAME + desc: Convert the IANA protocol number to the protocol name + args: IANA Number + ret: The protocol name associated with the IANA number. + +>>> 6 +[=] 6 +[?] save as: ip.protocol + +>>> %vars +ip.protocol = 6 + +>>> PROTOCOL_TO_NAME(ip.protocol) +[=] TCP +``` + +### Command Line Options + +``` +$ /usr/metron/<version>/bin/stellar -h + +usage: stellar + -h,--help Print help + -z,--zookeeper <arg> Zookeeper URL +``` + +#### -z, --zookeeper + +*Optional* + +Attempts to connect to Zookeeper and read the Metron global configuration. Stellar functions may require the global configuration to work properly. If found, the global configuration values are printed to the console. + +``` +$ /usr/metron/<version>/bin/stellar -z node1:2181 +Stellar, Go! +{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH} +>>> +``` + +### Variable Assignment + +Stellar has no concept of variable assignment. For testing and debugging purposes, it is important to be able to create variables that simulate data contained within incoming messages. The REPL has created a means for a user to perform variable assignment outside of the core Stellar language. + +After execution of a Stellar expression, the REPL will prompt for the name of a variable to save the expression result to. If no name is provided, the result is not saved. + +``` +>>> 2+2 +[=] 4.0 +[?] save as: foo +>>> %vars +foo = 4.0 +>>> +``` + +### Magic Commands + +The REPL has a set of magic commands that provide the REPL user with information about the Stellar execution environment. The following magic commands are supported. + +#### `%functions` + +This command lists all functions resolvable in the Stellar environment. Stellar searches the classpath for Stellar functions. This can make it difficult in some cases to understand which functions are resolvable. + +``` +>>> %functions +BLOOM_ADD, BLOOM_EXISTS, BLOOM_INIT, BLOOM_MERGE, DAY_OF_MONTH, DAY_OF_WEEK, DAY_OF_YEAR, +DOMAIN_REMOVE_SUBDOMAINS, DOMAIN_REMOVE_TLD, DOMAIN_TO_TLD, ENDS_WITH, GET, GET_FIRST, +GET_LAST, IN_SUBNET, IS_DATE, IS_DOMAIN, IS_EMAIL, IS_EMPTY, IS_INTEGER, IS_IP, IS_URL, +JOIN, MAAS_GET_ENDPOINT, MAAS_MODEL_APPLY, MAP_EXISTS, MAP_GET, MONTH, PROTOCOL_TO_NAME, +REGEXP_MATCH, SPLIT, STARTS_WITH, STATS_ADD, STATS_COUNT, STATS_GEOMETRIC_MEAN, STATS_INIT, +STATS_KURTOSIS, STATS_MAX, STATS_MEAN, STATS_MERGE, STATS_MIN, STATS_PERCENTILE, +STATS_POPULATION_VARIANCE, STATS_QUADRATIC_MEAN, STATS_SD, STATS_SKEWNESS, STATS_SUM, +STATS_SUM_LOGS, STATS_SUM_SQUARES, STATS_VARIANCE, TO_DOUBLE, TO_EPOCH_TIMESTAMP, +TO_INTEGER, TO_LOWER, TO_STRING, TO_UPPER, TRIM, URL_TO_HOST, URL_TO_PATH, URL_TO_PORT, +URL_TO_PROTOCOL, WEEK_OF_MONTH, WEEK_OF_YEAR, YEAR +>>> +``` + +#### `%vars` + +Lists all variables in the Stellar environment. + +``` +Stellar, Go! +{es.clustername=metron, es.ip=node1, es.port=9300, es.date.format=yyyy.MM.dd.HH} +>>> %vars +>>> 2+2 +[=] 4.0 +[?] save as: foo +>>> %vars +foo = 4.0 +>>> +``` + +#### `?<function>` + +Returns formatted documentation of the Stellar function. Provides the description of the function along with the expected arguments. + +``` +>>> ?BLOOM_ADD +BLOOM_ADD + desc: Adds an element to the bloom filter passed in + args: bloom - The bloom filter, value* - The values to add + ret: Bloom Filter +>>> ?IS_EMAIL +IS_EMAIL + desc: Tests if a string is a valid email address + args: address - The String to test + ret: True if the string is a valid email address and false otherwise. +>>> +``` \ No newline at end of file http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/4a4cb8b1/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarExecutor.java ---------------------------------------------------------------------- diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarExecutor.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarExecutor.java new file mode 100644 index 0000000..8268dfc --- /dev/null +++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarExecutor.java @@ -0,0 +1,152 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.metron.common.stellar.shell; + +import com.fasterxml.jackson.core.type.TypeReference; +import org.apache.commons.lang.StringUtils; +import org.apache.curator.framework.CuratorFramework; +import org.apache.metron.common.configuration.ConfigurationsUtils; +import org.apache.metron.common.dsl.Context; +import org.apache.metron.common.dsl.FunctionResolver; +import org.apache.metron.common.dsl.MapVariableResolver; +import org.apache.metron.common.dsl.StellarFunctions; +import org.apache.metron.common.dsl.VariableResolver; +import org.apache.metron.common.stellar.StellarProcessor; +import org.apache.metron.common.utils.JSONUtils; + +import java.io.ByteArrayInputStream; +import java.util.Collections; +import java.util.HashMap; +import java.util.Map; +import java.util.Optional; + +import static org.apache.metron.common.configuration.ConfigurationsUtils.readGlobalConfigBytesFromZookeeper; + +/** + * Executes Stellar expressions and maintains state across multiple invocations. + */ +public class StellarExecutor { + + /** + * The variables known by Stellar. + */ + private Map<String, Object> variables; + + /** + * The function resolver. + */ + private FunctionResolver functionResolver ; + + /** + * A Zookeeper client. Only defined if given a valid Zookeeper URL. + */ + private Optional<CuratorFramework> client; + + /** + * The Stellar execution context. + */ + private Context context; + + public StellarExecutor() throws Exception { + this(null); + } + + public StellarExecutor(String zookeeperUrl) throws Exception { + this.variables = new HashMap<>(); + this.functionResolver = new StellarFunctions().FUNCTION_RESOLVER(); + this.client = createClient(zookeeperUrl); + this.context = createContext(); + } + + /** + * Creates a Zookeeper client. + * @param zookeeperUrl The Zookeeper URL. + */ + private Optional<CuratorFramework> createClient(String zookeeperUrl) { + + // can only create client, if have valid zookeeper URL + if(StringUtils.isNotBlank(zookeeperUrl)) { + CuratorFramework client = ConfigurationsUtils.getClient(zookeeperUrl); + client.start(); + return Optional.of(client); + + } else { + return Optional.empty(); + } + } + + /** + * Creates a Context initialized with configuration stored in Zookeeper. + */ + private Context createContext() throws Exception { + Context context = Context.EMPTY_CONTEXT(); + + // load global configuration from zookeeper + if (client.isPresent()) { + + // fetch the global configuration + Map<String, Object> global = JSONUtils.INSTANCE.load( + new ByteArrayInputStream(readGlobalConfigBytesFromZookeeper(client.get())), + new TypeReference<Map<String, Object>>() { + }); + + context = new Context.Builder() + .with(Context.Capabilities.GLOBAL_CONFIG, () -> global) + .with(Context.Capabilities.ZOOKEEPER_CLIENT, () -> client.get()) + .build(); + } + + return context; + } + + /** + * Executes the Stellar expression and returns the result. + * @param expression The Stellar expression to execute. + * @return The result of the expression. + */ + public Object execute(String expression) { + VariableResolver variableResolver = new MapVariableResolver(variables, Collections.emptyMap()); + StellarProcessor processor = new StellarProcessor(); + return processor.parse(expression, variableResolver, functionResolver, context); + } + + /** + * Assigns a value to a variable. + * @param variable The name of the variable. + * @param value The value of the variable + */ + public void assign(String variable, Object value) { + this.variables.put(variable, value); + } + + public Map<String, Object> getVariables() { + return this.variables; + } + + public FunctionResolver getFunctionResolver() { + return functionResolver; + } + + public Context getContext() { + return context; + } +} + http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/4a4cb8b1/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java ---------------------------------------------------------------------- diff --git a/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java new file mode 100644 index 0000000..cdf7eaf --- /dev/null +++ b/metron-platform/metron-common/src/main/java/org/apache/metron/common/stellar/shell/StellarShell.java @@ -0,0 +1,249 @@ +/* + * + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + * + */ + +package org.apache.metron.common.stellar.shell; + +import org.apache.commons.cli.CommandLine; +import org.apache.commons.cli.CommandLineParser; +import org.apache.commons.cli.HelpFormatter; +import org.apache.commons.cli.Options; +import org.apache.commons.cli.PosixParser; +import org.apache.commons.lang3.StringUtils; +import org.apache.metron.common.dsl.Context; +import org.apache.metron.common.dsl.StellarFunctionInfo; + +import java.util.Scanner; +import java.util.stream.Collectors; +import java.util.stream.StreamSupport; + +/** + * A REPL environment for Stellar. + * + * Useful for debugging Stellar expressions. + */ +public class StellarShell { + + private static final String WELCOME = "Stellar, Go!"; + private static final String EXPRESSION_PROMPT = ">>> "; + private static final String RESULT_PROMPT = "[=] "; + private static final String ERROR_PROMPT = "[!] "; + private static final String ASSIGN_PROMPT = "[?] save as: "; + private static final String MAGIC_PREFIX = "%"; + private static final String MAGIC_FUNCTIONS = "%functions"; + private static final String MAGIC_VARS = "%vars"; + private static final String DOC_PREFIX = "?"; + + private StellarExecutor executor; + + /** + * Execute the Stellar REPL. + */ + public static void main(String[] args) throws Exception { + StellarShell shell = new StellarShell(args); + shell.execute(); + } + + /** + * Create a Stellar REPL. + * @param args The commmand-line arguments. + */ + public StellarShell(String[] args) throws Exception { + + // define valid command-line options + Options options = new Options(); + options.addOption("z", "zookeeper", true, "Zookeeper URL"); + options.addOption("h", "help", false, "Print help"); + + CommandLineParser parser = new PosixParser(); + CommandLine commandLine = parser.parse(options, args); + + // print help + if(commandLine.hasOption("h")) { + HelpFormatter formatter = new HelpFormatter(); + formatter.printHelp("stellar", options); + System.exit(0); + } + + // create the executor + if(commandLine.hasOption("z")) { + String zookeeperUrl = commandLine.getOptionValue("z"); + executor = new StellarExecutor(zookeeperUrl); + + } else { + executor = new StellarExecutor(); + } + } + + /** + * Handles the main loop for the REPL. + */ + public void execute() { + + // welcome message and print globals + writeLine(WELCOME); + executor.getContext() + .getCapability(Context.Capabilities.GLOBAL_CONFIG) + .ifPresent(conf -> writeLine(conf.toString())); + + Scanner scanner = new Scanner(System.in); + boolean done = false; + while(!done) { + + // prompt the user for an expression + write(EXPRESSION_PROMPT); + String expression = scanner.nextLine(); + if(StringUtils.isNotBlank(expression)) { + + if(isMagic(expression)) { + handleMagic(scanner, expression); + + } else if(isDoc(expression)) { + handleDoc(scanner, expression); + + } else { + handleStellar(scanner, expression); + } + } + } + } + + /** + * Handles user interaction when executing a Stellar expression. + * @param scanner The scanner used to read user input. + * @param expression The expression to execute. + */ + private void handleStellar(Scanner scanner, String expression) { + + Object result = executeStellar(expression); + if(result != null) { + + // echo the result + write(RESULT_PROMPT); + writeLine(result.toString()); + + // assign the result to a variable? + write(ASSIGN_PROMPT); + String variable = scanner.nextLine(); + if(StringUtils.isNotBlank(variable)) { + executor.assign(variable, result); + } + } + } + + /** + * Handles user interaction when executing a Magic command. + * @param scanner The scanner used to read user input. + * @param expression The expression to execute. + */ + private void handleMagic(Scanner scanner, String expression) { + + if(MAGIC_FUNCTIONS.equals(expression)) { + + // list all functions + String functions = StreamSupport + .stream(executor.getFunctionResolver().getFunctionInfo().spliterator(), false) + .map(info -> String.format("%s", info.getName())) + .sorted() + .collect(Collectors.joining(", ")); + writeLine(functions); + + } else if(MAGIC_VARS.equals(expression)) { + + // list all variables + executor.getVariables() + .forEach((k,v) -> writeLine(String.format("%s = %s", k, v))); + + } else { + writeLine(ERROR_PROMPT + "undefined magic command: " + expression); + } + } + + /** + * Handles user interaction when executing a doc command. + * @param scanner The scanner used to read user input. + * @param expression The expression to execute. + */ + private void handleDoc(Scanner scanner, String expression) { + + String functionName = StringUtils.substring(expression, 1); + StreamSupport + .stream(executor.getFunctionResolver().getFunctionInfo().spliterator(), false) + .filter(info -> StringUtils.equals(functionName, info.getName())) + .map(info -> format(info)) + .forEach(doc -> write(doc)); + } + + /** + * Formats the Stellar function info object into a readable string. + * @param info The stellar function info object. + * @return A readable string. + */ + private String format(StellarFunctionInfo info) { + return String.format( + "%s\n desc: %-60s\n args: %-60s\n ret: %-60s\n", + info.getName(), + info.getDescription(), + StringUtils.join(info.getParams(), ", "), + info.getReturns()); + } + + /** + * Is a given expression a built-in magic? + * @param expression The expression. + */ + private boolean isMagic(String expression) { + return StringUtils.startsWith(expression, MAGIC_PREFIX); + } + + /** + * Is a given expression asking for function documentation? + * @param expression The expression. + */ + private boolean isDoc(String expression) { + return StringUtils.startsWith(expression, DOC_PREFIX); + } + + /** + * Executes a Stellar expression. + * @param expression The expression to execute. + * @return The result of the expression. + */ + private Object executeStellar(String expression) { + Object result = null; + + try { + result = executor.execute(expression); + + } catch(Throwable t) { + writeLine(ERROR_PROMPT + t.getMessage()); + t.printStackTrace(); + } + + return result; + } + + private void write(String out) { + System.out.print(out); + } + + private void writeLine(String out) { + System.out.println(out); + } +} http://git-wip-us.apache.org/repos/asf/incubator-metron/blob/4a4cb8b1/metron-platform/metron-common/src/main/scripts/stellar ---------------------------------------------------------------------- diff --git a/metron-platform/metron-common/src/main/scripts/stellar b/metron-platform/metron-common/src/main/scripts/stellar new file mode 100644 index 0000000..8cc48c9 --- /dev/null +++ b/metron-platform/metron-common/src/main/scripts/stellar @@ -0,0 +1,34 @@ +#!/bin/bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + + +BIGTOP_DEFAULTS_DIR=${BIGTOP_DEFAULTS_DIR-/etc/default} +[ -n "${BIGTOP_DEFAULTS_DIR}" -a -r ${BIGTOP_DEFAULTS_DIR}/hbase ] && . ${BIGTOP_DEFAULTS_DIR}/hbase + +# Autodetect JAVA_HOME if not defined +if [ -e /usr/libexec/bigtop-detect-javahome ]; then + . /usr/libexec/bigtop-detect-javahome +elif [ -e /usr/lib/bigtop-utils/bigtop-detect-javahome ]; then + . /usr/lib/bigtop-utils/bigtop-detect-javahome +fi + +export HBASE_CONFIGS=/etc/hbase/conf +export METRON_VERSION=${project.version} +export METRON_HOME=/usr/metron/$METRON_VERSION +java -cp "$HBASE_CONFIGS:$METRON_HOME/lib/*" org.apache.metron.common.stellar.shell.StellarShell "$@"
