[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105328056
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extension
+case "bash":
+if (name.contains(".sh")) {
+isOnTheList = true;
+ 

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105327868
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
--- End diff --

The ShellFunctions btw, are limited to the management package.
Is that were this needs to live?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

SHELL_EDIT Stellar command and execution context

[Otto Fowler]

I was looking at this command that the capabilities functionality in
Context and I saw that the call for capabilities is:

Optional console =  context.getCapability(CONSOLE, false);


This means that if we are NOT in CONSOLE, there will not be an error.
I don’t think this is correct is it?

[GitHub] incubator-metron issue #439: METRON-571 add stellar external functions featu...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/439
  
This is missing documentation.  There are going to be requirements to the 
types of scripts that can be called, and they all need to be listed out.  A 
sample script or even a template should be provided.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325446
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extension
+case "bash":
+if (name.contains(".sh")) {
+isOnTheList = true;
+ 

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325529
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extension
+case "bash":
+if (name.contains(".sh")) {
+isOnTheList = true;
+ 

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325302
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extension
+case "bash":
+if (name.contains(".sh")) {
+isOnTheList = true;
+ 

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325249
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extension
+case "bash":
+if (name.contains(".sh")) {
+isOnTheList = true;
+ 

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105325103
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
+try {
+File script = new File(path);
+if (!script.exists() || script.isDirectory()) {
+return null;
+}
+}
+catch (NullPointerException e)  {
+System.err.println("Error: " + e.toString());
+return null;
+}
+
+switch (exec) { //check if matches extension
+case "bash":
+if (name.contains(".sh")) {
+isOnTheList = true;
+ 

[GitHub] incubator-metron pull request #439: METRON-571 add stellar external function...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/439#discussion_r105324915
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/ExternalFunctions.java
 ---
@@ -0,0 +1,292 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.metron.common.dsl.functions;
+
+import java.io.BufferedReader;
+import java.io.File;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.InputStreamReader;
+import java.io.OutputStream;
+import java.io.PrintWriter;
+import java.util.List;
+import java.lang.ProcessBuilder;
+import java.lang.ClassLoader;
+import java.lang.reflect.Method;
+import java.util.Map;
+import java.util.regex.Pattern;
+import com.google.common.base.Joiner;
+import com.google.common.base.Splitter;
+import com.google.common.collect.Iterables;
+import org.apache.metron.common.dsl.Context;
+import org.apache.metron.common.dsl.StellarFunction;
+import org.apache.metron.common.dsl.ParseException;
+import org.apache.metron.common.dsl.Stellar;
+
+/**
+ * Executes external script on server via stellar process
+ */
+public class ExternalFunctions {
+
+   public static class ExecuteScript implements StellarFunction {
+
+private ThreadedStreamHandler inStream;
+private ThreadedStreamHandler errStream;
+private boolean isOnTheList = false;
+
+@Stellar(name="EXEC_SCRIPT",
+description = "Executes an external shell function via 
stellar.",
+params = {
+"exec - the executing cmd (ie. bash, sh, python)",
+"name - name of the script, located in /scripts " +
+"Do NOT include any special chars 
except(_), Do include file extension"
+},
+returns = "the return value of the function"
+)
+
+   @Override
+public Object apply(List args, Context context) throws 
ParseException {
+String exec = "";
+String name = "";
+String path = "";
+
+// if args are provided, get args, only if in whitelist
+if (args.size() >= 1) {
+Object execObj = args.get(0);
+if (!(execObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) execObj).length() > 0) {
+exec = (String) execObj;
+}
+else {
+return null;
+}
+
+Object nameObj = args.get(1);
+if (!(nameObj instanceof String)) { //check if string
+return null;
+}
+else if (((String) nameObj).length() > 0) {
+name = (String) nameObj;
+}
+else {
+return null;
+}
+
+if (!Pattern.matches("[0-9A-Za-z.]+", name)) {
+return null; //if not on whitelist
+}
+
+path = "/scripts" + name;
--- End diff --

what file chooser dialog is that?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #405: METRON-641: Fixed kibana_master.py Pytho...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/405


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss

[Otto Fowler]

Actually, I’m going to run it again. disregard


On March 9, 2017 at 21:31:30, Otto Fowler (ottobackwa...@gmail.com) wrote:


Failed tests:

  BasicAsaParserTest.testIp6Addr:151 null

  GrokWebSphereParserTest.testParseLoginLine:60 expected:<14[92278]448000>
but was:<14[60742]448000>

  GrokWebSphereParserTest.testParseMalformedLoginLine:151
expected:<14[92278]448000> but was:<14[60742]448000>

  GrokWebSphereParserTest.tetsParseLogoutLine:84 expected:<14[92279]347000>
but was:<14[60743]347000>

  GrokWebSphereParserTest.tetsParseMalformedLogoutLine:175
expected:<14[92279]347000> but was:<14[60743]347000>

  GrokWebSphereParserTest.tetsParseMalformedOtherLine:220
expected:<14[92276]654000> but was:<14[60740]654000>

  GrokWebSphereParserTest.tetsParseMalformedRBMLine:198
expected:<14[92277]795000> but was:<14[60741]795000>

  GrokWebSphereParserTest.tetsParseOtherLine:129 expected:<14[92276]654000>
but was:<14[60740]654000>

  GrokWebSphereParserTest.tetsParseRBMLine:107 expected:<14[92277]795000>
but was:<14[60741]795000>




On March 9, 2017 at 19:16:24, ottobackwards (g...@git.apache.org) wrote:

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/405

I will later tonight or first thing tomorrow am


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss

[Otto Fowler]

Failed tests:

  BasicAsaParserTest.testIp6Addr:151 null

  GrokWebSphereParserTest.testParseLoginLine:60 expected:<14[92278]448000>
but was:<14[60742]448000>

  GrokWebSphereParserTest.testParseMalformedLoginLine:151
expected:<14[92278]448000> but was:<14[60742]448000>

  GrokWebSphereParserTest.tetsParseLogoutLine:84 expected:<14[92279]347000>
but was:<14[60743]347000>

  GrokWebSphereParserTest.tetsParseMalformedLogoutLine:175
expected:<14[92279]347000> but was:<14[60743]347000>

  GrokWebSphereParserTest.tetsParseMalformedOtherLine:220
expected:<14[92276]654000> but was:<14[60740]654000>

  GrokWebSphereParserTest.tetsParseMalformedRBMLine:198
expected:<14[92277]795000> but was:<14[60741]795000>

  GrokWebSphereParserTest.tetsParseOtherLine:129 expected:<14[92276]654000>
but was:<14[60740]654000>

  GrokWebSphereParserTest.tetsParseRBMLine:107 expected:<14[92277]795000>
but was:<14[60741]795000>




On March 9, 2017 at 19:16:24, ottobackwards (g...@git.apache.org) wrote:

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/405

I will later tonight or first thing tomorrow am


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss

[Otto Fowler]

Dima, what email do you use for github?

On March 9, 2017 at 19:16:21, ottobackwards (g...@git.apache.org) wrote:

> Github user ottobackwards commented on the issue:
>
> https://github.com/apache/incubator-metron/pull/405
>
> I will later tonight or first thing tomorrow am
>
>
> ---
> If your project is set up for it, you can reply to this email and have
> your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working,
> please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

[GitHub] incubator-metron issue #405: METRON-641: Fixed kibana_master.py Python3 miss

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/405
  
I will later tonight or first thing tomorrow am


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
I am +1.  I have been working downstream and building off of this for a 
bit, and been able to get quick and full up with everything started.  I have 
some things that I would like to improve on, but like @dlyle65535 says, this is 
step 2 of many


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
That's great, @mmiklavc.  I am also a +1.  I was able to test this 
successfully on Quick Dev and Full Dev.  


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
Thanks for all the help! I intend to merge this in tomorrow afternoon. 
@ottobackwards, @justinleet, I wanted to make sure you two were good. I also 
want to make sure there's no other feedback I've missed.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
Thanks @mmiklavc!


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
Looks like everything came up correctly for me in AWS. So +1 to that part 
of it.


![image](https://cloud.githubusercontent.com/assets/658443/23773173/4a6a09e2-04db-11e7-9f0e-f39e27f345df.png)



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@mmiklavc - Exactly. We were wanting to make sure that @nickwallen was 
having environmental issues.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@dlyle65535 is the test for EC2 to just verify everything spins up as 
normal? Any additional specific items to test or smoke test?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [MENTORS][REQUEST] Please review the release currently going on in incubator-general

[Casey Stella]

Sorry, I missed an email in my search.  We have 1 review (thanks Billie!
:), need 2 more.

On Thu, Mar 9, 2017 at 1:25 PM, Casey Stella  wrote:

> Hi Mentors,
>
> We have a release going on in incubator-general for around 10 days now
> with no reviews.  Would a couple of y'all mind reviewing for us?
>
> Thanks,
>
> Casey
>

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@dlyle65535 fyi I'm deploying to ec2 right now. I'll update shortly.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@justinleet - Accepted. Thanks.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #473: METRON-712: Separate evaluation from par...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/473


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@dlyle65535 METRON-745 is in (as I'm sure you can tell from the conflict 
list).  I already incorporated the Kibana map changes, so you should just be 
able to accept master's version.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #471: METRON-755 Update GitHub PR Template

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/471


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #470: METRON-752 Fix documentation typos and M...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/470


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #475: METRON-745: Create Error Dashboards

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/475


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #475: METRON-745: Create Error Dashboards

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/475
  
+1 by inspection, great job @justinleet 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[MENTORS][REQUEST] Please review the release currently going on in incubator-general

[Casey Stella]

Hi Mentors,

We have a release going on in incubator-general for around 10 days now with
no reviews.  Would a couple of y'all mind reviewing for us?

Thanks,

Casey

[GitHub] incubator-metron issue #436: METRON-671: Refactor existing Ansible deploymen...

[dlyle65535]

Github user dlyle65535 commented on the issue:

https://github.com/apache/incubator-metron/pull/436
  
@justinleet - that last commit adds the quotation requiremnt to the tool 
tip.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---