[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

2017-03-22 Thread cestella 
Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
I was planning to make a JIRA around the multiple topic bit.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread justinleet 
Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107438798
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/components/ParserTopologyComponent.java
 ---
@@ -97,6 +99,19 @@ public void start() throws UnableToStartException {
   public void stop() {
 if(stormCluster != null) {
   stormCluster.shutdown();
+  if(new File("logs/workers-artifacts").exists()) {
+Path rootPath = Paths.get("logs");
+Path destPath = Paths.get("target/logs");
+try {
+  Files.move(rootPath, destPath);
+  Files.walk(destPath)
--- End diff --

Same deal with FileUtils.deleteDirectory() here


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread justinleet 
Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107522830
  
--- Diff: 
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
 ---
@@ -96,16 +109,18 @@ public HDFSWriterCallback withConfig(HDFSWriterConfig 
config) {
 this.config = config;
 return this;
 }
+
 @Override
 public List apply(List tuple, EmitContext context) {
-
-List keyValue = (List) tuple.get(0);
-LongWritable ts = (LongWritable) keyValue.get(0);
-BytesWritable rawPacket = (BytesWritable)keyValue.get(1);
+byte[] key = (byte[]) tuple.get(0);
+byte[] value = (byte[]) tuple.get(1);
+if(!config.getDeserializer().deserializeKeyValue(key, value, 
KeyValue.key.get(), KeyValue.value.get())) {
+LOG.debug("Dropping malformed packet...");
--- End diff --

Is it reasonable to include the key and value we're having issues with in 
the debug statement?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread justinleet 
Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107524059
  
--- Diff: 
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/Deserializers.java
 ---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.spout.pcap.deserializer;
+
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.common.utils.timestamp.TimestampConverter;
+
+import java.util.function.Function;
+
+/**
+ * Deserializers take the raw bytes from kafka key and value and construct 
the timestamp and raw bytes for PCAP.
+ */
+public enum Deserializers {
+  /**
+   * Extract the timestamp from the key and the raw packet 
(global-headerless) from the value
+   */
+   FROM_KEY( converter -> new FromKeyDeserializer(converter))
+  /**
+   * Ignore the key and pull the timestamp directly from the packet 
itself.  Also, assume that the packet isn't global-headerless.
+   */
+  ,FROM_PACKET(converter -> new FromPacketDeserializer());
+  ;
+  Function creator;
+  Deserializers(Function creator)
+  {
+this.creator = creator;
+  }
+
+  public static KeyValueDeserializer create(String scheme, 
TimestampConverter converter) {
+try {
+  Deserializers ts = Deserializers.valueOf(scheme.toUpperCase());
+  return ts.creator.apply(converter);
+}
+catch(IllegalArgumentException iae) {
+  return Deserializers.FROM_KEY.creator.apply(converter);
+}
+  }
+
+  public static KeyValueDeserializer create(String scheme, String 
converter) {
+return create(scheme, 
TimestampConverters.valueOf(converter.toUpperCase()));
--- End diff --

Shouldn't this be a call to TimestampConverters.getConverter()?

And if we're uppercasing things, shouldn't it be in the TimestampConverter 
(given that it's meant to match an Enum value)


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread justinleet 
Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107521465
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
 ---
@@ -106,19 +105,26 @@ public static TopologyBuilder build(String 
zookeeperUrl,
   /**
* Create a spout that consumes tuples from a Kafka topic.
*
-   * @param zookeeperUrlZookeeper URL
+   * @param zkQuorum Zookeeper URL
* @param sensorType  Type of sensor
-   * @param offset  Kafka topic offset where the topology 
will start; BEGINNING, END, WHERE_I_LEFT_OFF
-   * @param kafkaSpoutConfigOptions Configuration options for the kafka 
spout
+   * @param kafkaConfigOptional Configuration options for the kafka 
spout
* @param parserConfigConfiguration for the parser
* @return
*/
-  private static KafkaSpout createKafkaSpout(String zookeeperUrl, String 
sensorType, SpoutConfig.Offset offset, EnumMap 
kafkaSpoutConfigOptions, SensorParserConfig parserConfig) {
-
+  private static StormKafkaSpout createKafkaSpout(String zkQuorum, String 
sensorType, Optional> kafkaConfigOptional, 
SensorParserConfig parserConfig) {
+Map kafkaSpoutConfigOptions = 
kafkaConfigOptional.orElse(new HashMap<>());
 String inputTopic = parserConfig.getSensorTopic() != null ? 
parserConfig.getSensorTopic() : sensorType;
-SpoutConfig spoutConfig = new SpoutConfig(new ZkHosts(zookeeperUrl), 
inputTopic, "", inputTopic).from(offset);
-SpoutConfigOptions.configure(spoutConfig, kafkaSpoutConfigOptions);
-return new KafkaSpout(spoutConfig);
+
if(!kafkaSpoutConfigOptions.containsKey(SpoutConfiguration.FIRST_POLL_OFFSET_STRATEGY.key))
 {
--- End diff --

Can you make these putIfAbsent calls()?

e.g.
```

kafkaSpoutConfigOptions.putIfAbsent(SpoutConfiguration.FIRST_POLL_OFFSET_STRATEGY.key,

KafkaSpoutConfig.FirstPollOffsetStrategy.UNCOMMITTED_EARLIEST.toString());
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread justinleet 
Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107534534
  
--- Diff: pom.xml ---
@@ -67,20 +67,44 @@
 
 1.0.1
 1.0.1
-0.10.0.1
+0.10.0
 2.7.1
 1.1.1
-1.8.0
 1.5.2
 
+1.8.0
 4.5
 3.7
 2.7.1
 3.3
-${base_storm_version}
+1.0.3
+
+
1.0.1.2.5.0.0-1245
--- End diff --

I'm sure you've already thought of this, but assuming we do go with this, 
please make sure this gets a JIRA associated with it.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread justinleet 
Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107531858
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.storm.kafka.flux;
+
+import com.google.common.base.Joiner;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.storm.kafka.spout.*;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.OutputFieldsGetter;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/**
+ * This is a convenience layer on top of the KafkaSpoutConfig.Builder 
available in storm-kafka-client.
+ * The justification for this class is two-fold.  First, there are a lot 
of moving parts and a simplified
+ * approach to constructing spouts is useful.  Secondly, and perhaps more 
importantly, the Builder pattern
+ * is decidedly unfriendly to use inside of Flux.  Finally, we can make 
things a bit more friendly by only requiring
+ * zookeeper and automatically figuring out the brokers for the bootstrap 
server.
+ *
+ * @param  The kafka key type
+ * @param  The kafka value type
+ */
+public class SimpleStormKafkaBuilder extends 
KafkaSpoutConfig.Builder {
+  final static String STREAM = "default";
+
+  /**
+   * The fields exposed by the kafka consumer.  These will show up in the 
Storm tuple.
+   */
+  public enum FieldsConfiguration {
+KEY("key", record -> record.key()),
+VALUE("value", record -> record.value()),
+PARTITION("partition", record -> record.partition()),
+TOPIC("topic", record -> record.topic())
+;
+String fieldName;
+Function recordExtractor;
+
+FieldsConfiguration(String fieldName, Function 
recordExtractor) {
+  this.recordExtractor = recordExtractor;
+  this.fieldName = fieldName;
+}
+
+/**
+ * Return a list of the enums
+ * @param configs
+ * @return
+ */
+public static List toList(String... configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Return a list of the enums from their string representation.
+ * @param configs
+ * @return
+ */
+public static List toList(List configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Construct a Fields object from an iterable of enums.  These fields 
are the fields
+ * exposed in the Storm tuple emitted from the spout.
+ * @param configs
+ * @return
+ */
+public static Fields getFields(Iterable configs) {
+  List fields = new ArrayList<>();
+  for(FieldsConfiguration config : configs) {
+fields.add(config.fieldName);
+  }
+  return new Fields(fields);
+}
+  }
+
+  /**
+   * Build a tuple given the fields and the topic.  We want to use our 
FieldsConfiguration enum
+   * to define what this tuple looks like.
+   * @param  The key type in kafka
+   * @param  The

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread justinleet 
Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107533535
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.storm.kafka.flux;
+
+import com.google.common.base.Joiner;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.storm.kafka.spout.*;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.OutputFieldsGetter;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/**
+ * This is a convenience layer on top of the KafkaSpoutConfig.Builder 
available in storm-kafka-client.
+ * The justification for this class is two-fold.  First, there are a lot 
of moving parts and a simplified
+ * approach to constructing spouts is useful.  Secondly, and perhaps more 
importantly, the Builder pattern
+ * is decidedly unfriendly to use inside of Flux.  Finally, we can make 
things a bit more friendly by only requiring
+ * zookeeper and automatically figuring out the brokers for the bootstrap 
server.
+ *
+ * @param  The kafka key type
+ * @param  The kafka value type
+ */
+public class SimpleStormKafkaBuilder extends 
KafkaSpoutConfig.Builder {
+  final static String STREAM = "default";
+
+  /**
+   * The fields exposed by the kafka consumer.  These will show up in the 
Storm tuple.
+   */
+  public enum FieldsConfiguration {
+KEY("key", record -> record.key()),
+VALUE("value", record -> record.value()),
+PARTITION("partition", record -> record.partition()),
+TOPIC("topic", record -> record.topic())
+;
+String fieldName;
+Function recordExtractor;
+
+FieldsConfiguration(String fieldName, Function 
recordExtractor) {
+  this.recordExtractor = recordExtractor;
+  this.fieldName = fieldName;
+}
+
+/**
+ * Return a list of the enums
+ * @param configs
+ * @return
+ */
+public static List toList(String... configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Return a list of the enums from their string representation.
+ * @param configs
+ * @return
+ */
+public static List toList(List configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Construct a Fields object from an iterable of enums.  These fields 
are the fields
+ * exposed in the Storm tuple emitted from the spout.
+ * @param configs
+ * @return
+ */
+public static Fields getFields(Iterable configs) {
+  List fields = new ArrayList<>();
+  for(FieldsConfiguration config : configs) {
+fields.add(config.fieldName);
+  }
+  return new Fields(fields);
+}
+  }
+
+  /**
+   * Build a tuple given the fields and the topic.  We want to use our 
FieldsConfiguration enum
+   * to define what this tuple looks like.
+   * @param  The key type in kafka
+   * @param  The

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread justinleet 
Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107436680
  
--- Diff: 
metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
 ---
@@ -133,7 +138,25 @@ public void start() throws UnableToStartException {
   @Override
   public void stop() {
 if (stormCluster != null) {
-  stormCluster.shutdown();
+  try {
+stormCluster.shutdown();
+if(new File("logs/workers-artifacts").exists()) {
+  Path rootPath = Paths.get("logs");
+  Path destPath = Paths.get("target/logs");
+  try {
+Files.move(rootPath, destPath);
+Files.walk(destPath)
--- End diff --

Could this just be a FileUtils.deleteDirectory(destPath)?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] Final Board Resolution Draft V2

2017-03-22 Thread Billie Rinaldi 
Here are the remaining Apache IDs:
Kiran Komaravolu (kirankom)
Ray Urciuoli (rurcioli)

The list seems to be missing Jim Baker (jimbaker)? See
http://home.apache.org/phonebook?podling=metron. Also, it looks like Bryan
Taylor never submitted an ICLA, and has no Apache ID. I don't think he can
be on the initial PMC list, in that case.

On Mon, Mar 20, 2017 at 12:05 AM, James Sirota  wrote:

>
> - Removed affiliations
> - Added apache IDs where possible
> - Removed committers and only left PPMC members
>
> Hope this version holds up.  Please vote +1, -1, or 0 for neutral.  The
> vote will be open for 72 hours
>
>
> The incubating Apache Metron community believes it is time to graduate to
> TLP.
>
> Apache Metron entered incubation in December of 2015. Since then, we've
> overcome technical challenges to remove Category X dependencies, and made 3
> releases. Our most recent release contains binary convenience artifacts. We
> are a very helpful and engaged community, ready to answer all questions and
> feedback directed to us via the user list. Through our time in incubation
> we've added a number of committers and promoted some of them to PPMC
> membership. We are actively pursuing others. While we do still have issues
> to address raised by means of the maturity model, all projects are ongoing
> processes, and we believe we no longer need the incubator to continue
> addressing these issues.
>
> To inform the discussion, here is some basic project information:
>
> Project status:
>   http://incubator.apache.org/projects/metron.html
>
> Project website:
>   https://metron.incubator.apache.org/
>
> Project documentation:
>https://cwiki.apache.org/confluence/display/METRON/Documentation
>
> Maturity assessment:
>https://cwiki.apache.org/confluence/display/METRON/
> Apache+Project+Maturity+Model
>
> DRAFT of the board resolution is at the bottom of this email
>
> Proposed PMC size: 25 members
>
> Total number of committers: 6 members
>
>
> 516 commits on develop
> 34 contributors across all branches
>
> dev list averaged ~650 msgs/month for the last 3 months
>
>
> Resolution:
>
> Establish the Apache Metron Project
>
> WHEREAS, the Board of Directors deems it to be in the best
> interests of the Foundation and consistent with the
> Foundation's purpose to establish a Project Management
> Committee charged with the creation and maintenance of
> open-source software, for distribution at no charge to the
> public, related to a security analytics platform for big data use cases.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> Committee (PMC), to be known as the "Apache Metron Project",
> be and hereby is established pursuant to Bylaws of the
> Foundation; and be it further
>
> RESOLVED, that the Apache Metron Project be and hereby is
> responsible for the creation and maintenance of software
> related to:
> (a) A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.
> (b) Real time processing and application of enrichments
> (c) Efficient information storage
> (d) An interface that gives a security investigator a centralized view of
> data and alerts passed through the system.
>
> RESOLVED, that the office of "Vice President, Apache Metron" be
> and hereby is created, the person holding such office to
> serve at the direction of the Board of Directors as the chair
> of the Apache Metron Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache Metron Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache Metron Project:
>
>
> PPMC:
> Mark Bittmann (mbittmann)
> Sheetal Dolas (sheetal_dolas)
> Debo Dutta (ddutta)
> Discovery Gerdes (discovery)
> Andrew Hartnett (dev_warlord)
> Dave Hirko (dbhirko)
> Paul Kehrer (reaperhulk)
> Brad Kolarov (bjkolly)
> Kiran Komaravolu (UKNOWN)
> Larry McCay (lmccay)
> P. Taylor Goetz (ptgoetz)
> Ryan Merriman (rmerriman)
> Michael Perez (mperez)
> Charles Porter (cporter)
> Phillip Rhodes (prhodes)
> Sean Schulte (sirsean)
> James Sirota (jsirota)
> Casey Stella (cstella)
> Bryan Taylor (UKNOWN)
> Ray Urciuoli(UKNOWN)
> Vinod Kumar Vavilapalli (vinodkv)
> George Vetticaden (gvetticaden)
> Oskar Zabik (smogg)
> David Lyle (lyle)
> Nick Allen (nickallen)
>
>
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Casey Stella
> be appointed to the office of Vice President, Apache Metron, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed; and be it further
>
> RESOLVED, that the initial Apache Metron PMC be and hereby is
> tasked with the creation of a set of bylaws intended to
> encourage open development and increased participation in the
> Apache Metron

[DISCUSS][PROPOSAL] Maven Plugin to build packages with dependencies

2017-03-22 Thread Otto Fowler 
As we have discussion previously, I am working on a plugin architecture for
parsers and stellar ( down the road ), base on Nifi’s Nar format.

Part of using the Nar system is building the Nar itself.  This is done
using the nifi-nar-maven-plugin.  Some historical background - this plugin
used to live in the nifi
repo/tree, and they split it out to it’s own repo and published it to
apache mvn.

For Metron’s use of the nar we want the following:

1. To be able to change the archive’s extension from .nar to something else
specific for us
2. To be able to rename the manifest information generated so that it
doesn’t reference nar
3.  Possibly to be able to add more manifest entries and custom metron
specific metadata

My first preference, was to use the nifi plugin as is, but with just enough
modification to make
points 1 and 2 possible.

To that end I opened a jira and submitted a PR to the nifi project that did
just that, added the ability to configure the type produced by the plugin
and set the metadata prefix name.

The chair of nifi, has commented that issue suggesting that we fork or copy
the plugin, since he has rightly noticed that there is really no benefit for
them in accepting these changes and capabilities.

I wanted to have a discussion around what I see as the options we have at
this point..

1. As Nifi did, we request a new git repo to host our version of the
plugin, and do a release/publish of the plugin to apache mvn.  This would
include ‘rebranding’ the plugin
from nifi to something else.
2. We start as they did, with the plugin with my changes and as a separate
build step ( copy )
3. We fork and use git submodules to track that project

In cases 2 or 3 we will have to decide to rebrand or just use a custom
version scheme ( -METRON ) with the plugin.

To me, option 1 is the best option, although it will involve me getting
help from others to accomplish ( INFRA request, release manager etc etc ).
But, it is the cleanest, best option I think for a production case.
Thoughts?







for reference:
https://issues.apache.org/jira/browse/NIFI-3628
https://github.com/apache/nifi-maven/pull/2
http://mail-archives.apache.org/mod_mbox/nifi-dev/201508.mbox/%3CCALJK9a7xJW%2BG-dJSXAZ640xQdy4tpRZ%3DtEuHDgq8A%3DMY%2BOkf1g%40mail.gmail.com%3E
https://issues.apache.org/jira/browse/INFRA-10119

Re: [DISCUSS] Stepping down as release manager

2017-03-22 Thread Billie Rinaldi 
Agreed, expressions of support are always appreciated. :-) I only meant to
point out that an official vote is not always required when support has
already been expressed, not to discourage anyone from being supportive. +1
away!

On Wed, Mar 22, 2017 at 10:33 AM, Matt Foley  wrote:

> As Billie says, our bylaws don’t require a vote to assign a release
> manager.
> However, an RM certainly needs the support of the community to be
> effective,
> so all these +1’s are very much appreciated! :-)
> Thanks,
> --Matt
>
> On 3/22/17, 8:30 AM, "Billie Rinaldi"  wrote:
>
> Not everything needs to be voted upon. The ASF guidelines and Metron's
> bylaws specify which actions require a vote. Other types of things,
> like
> choosing a release manager, can be approved by lazy consensus. I just
> noticed that Metron's bylaws have slightly different definitions for
> the
> approval types than the ASF typically uses. I'd recommend changing
> these to
> the standard ASF definitions. Specifically, Lazy Consensus does not
> require
> a vote and thus no +1s are needed (https://www.apache.org/
> foundation/glossary.html#LazyConsensus) and Lazy Majority and Lazy 2/3
> Majority should be called Majority (https://www.apache.org/
> foundation/glossary.html#MajorityApproval) and 2/3 Majority.
>
> On Wed, Mar 22, 2017 at 7:23 AM, Justin Leet 
> wrote:
>
> > Right now it's just support, not a vote.  I assume, based on our past
> > practices, that there will be a separate [VOTE] thread.
> >
> > Justin
> >
> > On Wed, Mar 22, 2017 at 10:06 AM, Otto Fowler <
> ottobackwa...@gmail.com>
> > wrote:
> >
> > > +1 but is this explicitly an official vote?
> > >
> > >
> > > On March 21, 2017 at 13:51:16, Justin Leet (justinjl...@gmail.com)
> > wrote:
> > >
> > > +1 for Matt
> > >
> > > On Tue, Mar 21, 2017 at 12:21 PM, zeo...@gmail.com <
> zeo...@gmail.com>
> > > wrote:
> > >
> > > > +1 for mattf
> > > >
> > > > On Tue, Mar 21, 2017 at 11:04 AM Ryan Merriman <
> merrim...@gmail.com>
> > > > wrote:
> > > >
> > > > > +1 for Matt
> > > > >
> > > > > On Tue, Mar 21, 2017 at 9:44 AM, Matt Foley 
> > wrote:
> > > > >
> > > > > > Casey, you’ve been a great release manager. I know how much
> detail
> > > > > effort
> > > > > > goes into this role.
> > > > > >
> > > > > > I am willing to serve as RM for the next while, if the
> community
> > > would
> > > > > > like. I was the RM for Hadoop for about a year, and in fact
> was RM
> > > for
> > > > > its
> > > > > > 1.0 release. Granted that was a while ago, but overall
> process
> > > doesn’t
> > > > > seem
> > > > > > to have changed much :-)
> > > > > >
> > > > > > Cheers,
> > > > > > --Matt
> > > > > >
> > > > > > On 3/21/17, 7:32 AM, "Casey Stella" 
> wrote:
> > > > > >
> > > > > > Right, Billie is exactly right. Working with the community to
> > > > > > constructing
> > > > > > releases that conform to apache standards and policies is
> the main
> > > > > > duty.
> > > > > > This will (hopefully) be our first set of releases outside
> of the
> > > > > > incubator, so if I'm allowed to be biased, I'm hoping that
> someone
> > > > > with
> > > > > > previous release management experience in other projects will
> > > > > > volunteer.
> > > > > > We're leaving the nest a bit and having an experienced hand
> at the
> > > > > > tiller
> > > > > > would be advantageous.
> > > > > >
> > > > > >
> > > > > > On Tue, Mar 21, 2017 at 10:21 AM, Billie Rinaldi <
> > > > bil...@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > See http://www.apache.org/dev/release-publishing#release_
> manager
> > > > > and
> > > > > > > http://www.apache.org/legal/release-policy.html for
> information
> > > > on
> > > > > > the
> > > > > > > tasks that a release manager performs.
> > > > > > >
> > > > > > > On Tue, Mar 21, 2017 at 7:10 AM, Khurram Ahmed <
> > > > > > khurramah...@gmail.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Casey it would be helpful if you could outline the
> > > > > > responsibilities of a
> > > > > > > > release manager for the Metron project.
> > > > > > > >
> > > > > > > > On Mar 21, 2017 6:57 PM, "Casey Stella" <
> ceste...@gmail.com>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > I've been extremely honored to spend the last few
> months as
> > > > the
> > > > > > Metron
> > > > > > > > > Release Manager. That being said, my watch is ended
> and it's
> > > > > > time for
> > > > > > > > > another release manager to step into my place.
> > > > > > > > >
> > > > > > > > > Who would

Re: [DISCUSS][PROPOSAL] Acceptance Tests

2017-03-22 Thread Ryan Merriman 
I hear ya on Ansible.

On Wed, Mar 22, 2017 at 2:07 PM, David Lyle  wrote:

> I'm -1 on Ansible for testing for the same reasons I recommended reducing
> reliance on it for deployment. It simply isn't well suited to be a general
> purpose installer (or testing framework) for the variety of OS/Ansible
> versions that we find in the wild.
>
> I'd figure out how/what we want to test and then see what we need for a
> suitable framework. I'm certain we can find one that generalizes better
> than Ansible has for us so far.
>
> -D...
>
>
> On Wed, Mar 22, 2017 at 3:00 PM, Ryan Merriman 
> wrote:
>
> > I don't think a cluster installed by ansible is a prerequisite to using
> > ansible to integration test.  They would be completely separate modules
> > except maybe sharing some property or inventory files.  Just need to run
> > scripts and hit rest endpoints right?  Just an idea, maybe it's overkill.
> > I'm cool with rolling our own.
> >
> > On Wed, Mar 22, 2017 at 1:49 PM, Casey Stella 
> wrote:
> >
> > > Maybe, but I'd argue that we would want this to be run against a
> > > non-ansible installed cluster.  For a first pass, I'd recommend just a
> > set
> > > of shell scripts utilizing the REPL and the REST API along with shell
> > > commands.  Most of our capabilities are quite scriptable.
> > >
> > > On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman 
> > > wrote:
> > >
> > > > Bumping this thread.  Looks like we have several +1s so I propose we
> > move
> > > > to the next step.  I'm anxious to get this done because these tests
> > would
> > > > have saved me time over the last couple weeks.  The management UI in
> > > > https://github.com/apache/incubator-metron/pull/484 has a set of e2e
> > > tests
> > > > being maintained in another branch so those could also be included in
> > > this
> > > > test suite when the UI makes it into master.
> > > >
> > > > Ideas for an "Acceptance Testing Framework"?  Could Ansible be good
> fit
> > > for
> > > > this since we already have it in our stack?
> > > >
> > > > On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Ok, yes I agree. In my experience with e2e/acceptance tests,
> they're
> > > best
> > > > > kept general with an emphasis on verifying that all the plumbing
> > works
> > > > > together. So yes, there are definite edge cases I think we'll want
> to
> > > > test
> > > > > here, but I say that with the caveat that I think we should ideally
> > > cover
> > > > > as many non-happy-path cases in unit and integration tests as
> > possible.
> > > > As
> > > > > an example, I don't think it makes sense to cover most of the
> > profiler
> > > > > windowing DSL language edge cases in acceptance tests instead of or
> > in
> > > > > addition to unit/integration tests unless there is something
> specific
> > > to
> > > > > the integration with a given an environment that we think could be
> > > > > problematic.
> > > > >
> > > > > M
> > > > >
> > > > > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella 
> > > > wrote:
> > > > >
> > > > > > No, I'm saying that they shouldn't be restricted to real-world
> > > > use-cases.
> > > > > > The E2E tests I laid out weren't real-world, but they did
> exercise
> > > the
> > > > > > components similar to real-world use-cases.  They should also be
> > able
> > > > to
> > > > > be
> > > > > > able to tread outside of the happy-path for those use-cases.
> > > > > >
> > > > > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> > > > > > michael.miklav...@gmail.com> wrote:
> > > > > >
> > > > > > > "I don't think acceptance tests should loosely associate with
> > real
> > > > > uses,
> > > > > > > but they should
> > > > > > > be free to delve into weird non-happy-pathways."
> > > > > > >
> > > > > > > Not following - are you saying they should *tightly* associate
> > with
> > > > > real
> > > > > > > uses and additonally include non-happy-path?
> > > > > > >
> > > > > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <
> > ceste...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > It is absolutely not a naive question, Matt.  We don't have a
> > lot
> > > > (or
> > > > > > > any)
> > > > > > > > docs about our integration tests; it's more of a "follow the
> > > lead"
> > > > > type
> > > > > > > of
> > > > > > > > thing at the moment, but that should be rectified.
> > > > > > > >
> > > > > > > > The integration tests spin up and down infrastructure
> > in-process,
> > > > > some
> > > > > > of
> > > > > > > > which are real and some of which are mock versions of the
> > > services.
> > > > > > > These
> > > > > > > > are good for catching some types of bugs, but often things
> > sneak
> > > > > > through,
> > > > > > > > like:
> > > > > > > >
> > > > > > > >- Hbase and storm can't exist in the same JVM, so HBase is
> > > > mocked
> > > > > in
> > > > > > > >

Re: [DISCUSS][PROPOSAL] Acceptance Tests

2017-03-22 Thread David Lyle 
I'm -1 on Ansible for testing for the same reasons I recommended reducing
reliance on it for deployment. It simply isn't well suited to be a general
purpose installer (or testing framework) for the variety of OS/Ansible
versions that we find in the wild.

I'd figure out how/what we want to test and then see what we need for a
suitable framework. I'm certain we can find one that generalizes better
than Ansible has for us so far.

-D...


On Wed, Mar 22, 2017 at 3:00 PM, Ryan Merriman  wrote:

> I don't think a cluster installed by ansible is a prerequisite to using
> ansible to integration test.  They would be completely separate modules
> except maybe sharing some property or inventory files.  Just need to run
> scripts and hit rest endpoints right?  Just an idea, maybe it's overkill.
> I'm cool with rolling our own.
>
> On Wed, Mar 22, 2017 at 1:49 PM, Casey Stella  wrote:
>
> > Maybe, but I'd argue that we would want this to be run against a
> > non-ansible installed cluster.  For a first pass, I'd recommend just a
> set
> > of shell scripts utilizing the REPL and the REST API along with shell
> > commands.  Most of our capabilities are quite scriptable.
> >
> > On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman 
> > wrote:
> >
> > > Bumping this thread.  Looks like we have several +1s so I propose we
> move
> > > to the next step.  I'm anxious to get this done because these tests
> would
> > > have saved me time over the last couple weeks.  The management UI in
> > > https://github.com/apache/incubator-metron/pull/484 has a set of e2e
> > tests
> > > being maintained in another branch so those could also be included in
> > this
> > > test suite when the UI makes it into master.
> > >
> > > Ideas for an "Acceptance Testing Framework"?  Could Ansible be good fit
> > for
> > > this since we already have it in our stack?
> > >
> > > On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Ok, yes I agree. In my experience with e2e/acceptance tests, they're
> > best
> > > > kept general with an emphasis on verifying that all the plumbing
> works
> > > > together. So yes, there are definite edge cases I think we'll want to
> > > test
> > > > here, but I say that with the caveat that I think we should ideally
> > cover
> > > > as many non-happy-path cases in unit and integration tests as
> possible.
> > > As
> > > > an example, I don't think it makes sense to cover most of the
> profiler
> > > > windowing DSL language edge cases in acceptance tests instead of or
> in
> > > > addition to unit/integration tests unless there is something specific
> > to
> > > > the integration with a given an environment that we think could be
> > > > problematic.
> > > >
> > > > M
> > > >
> > > > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella 
> > > wrote:
> > > >
> > > > > No, I'm saying that they shouldn't be restricted to real-world
> > > use-cases.
> > > > > The E2E tests I laid out weren't real-world, but they did exercise
> > the
> > > > > components similar to real-world use-cases.  They should also be
> able
> > > to
> > > > be
> > > > > able to tread outside of the happy-path for those use-cases.
> > > > >
> > > > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > > > "I don't think acceptance tests should loosely associate with
> real
> > > > uses,
> > > > > > but they should
> > > > > > be free to delve into weird non-happy-pathways."
> > > > > >
> > > > > > Not following - are you saying they should *tightly* associate
> with
> > > > real
> > > > > > uses and additonally include non-happy-path?
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <
> ceste...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > It is absolutely not a naive question, Matt.  We don't have a
> lot
> > > (or
> > > > > > any)
> > > > > > > docs about our integration tests; it's more of a "follow the
> > lead"
> > > > type
> > > > > > of
> > > > > > > thing at the moment, but that should be rectified.
> > > > > > >
> > > > > > > The integration tests spin up and down infrastructure
> in-process,
> > > > some
> > > > > of
> > > > > > > which are real and some of which are mock versions of the
> > services.
> > > > > > These
> > > > > > > are good for catching some types of bugs, but often things
> sneak
> > > > > through,
> > > > > > > like:
> > > > > > >
> > > > > > >- Hbase and storm can't exist in the same JVM, so HBase is
> > > mocked
> > > > in
> > > > > > >those cases.
> > > > > > >- The FileSystem that we get for Hadoop is the
> > > LocalRawFileSystem,
> > > > > not
> > > > > > >truly HDFS.  There are differences and we've run into
> > > > > > them..hilariously
> > > > > > > at
> > > > > > >times. ;)
> > > > > > >- Things done statically in a bolt are shared across all
> bolts
> > > > > because
> > >

Re: [DISCUSS][PROPOSAL] Acceptance Tests

2017-03-22 Thread Casey Stella 
Agreed, we need to get the profiler, MaaS and the REST API deployed via the
mpack sooner rather than later.

On Wed, Mar 22, 2017 at 3:05 PM, Ryan Merriman  wrote:

> I think we'll have non manual rest/web deployment soon regardless of this
> discussion.
>
> On Wed, Mar 22, 2017 at 2:00 PM, Ryan Merriman 
> wrote:
>
> > I don't think a cluster installed by ansible is a prerequisite to using
> > ansible to integration test.  They would be completely separate modules
> > except maybe sharing some property or inventory files.  Just need to run
> > scripts and hit rest endpoints right?  Just an idea, maybe it's overkill.
> > I'm cool with rolling our own.
> >
> > On Wed, Mar 22, 2017 at 1:49 PM, Casey Stella 
> wrote:
> >
> >> Maybe, but I'd argue that we would want this to be run against a
> >> non-ansible installed cluster.  For a first pass, I'd recommend just a
> set
> >> of shell scripts utilizing the REPL and the REST API along with shell
> >> commands.  Most of our capabilities are quite scriptable.
> >>
> >> On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman 
> >> wrote:
> >>
> >> > Bumping this thread.  Looks like we have several +1s so I propose we
> >> move
> >> > to the next step.  I'm anxious to get this done because these tests
> >> would
> >> > have saved me time over the last couple weeks.  The management UI in
> >> > https://github.com/apache/incubator-metron/pull/484 has a set of e2e
> >> tests
> >> > being maintained in another branch so those could also be included in
> >> this
> >> > test suite when the UI makes it into master.
> >> >
> >> > Ideas for an "Acceptance Testing Framework"?  Could Ansible be good
> fit
> >> for
> >> > this since we already have it in our stack?
> >> >
> >> > On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
> >> > michael.miklav...@gmail.com> wrote:
> >> >
> >> > > Ok, yes I agree. In my experience with e2e/acceptance tests, they're
> >> best
> >> > > kept general with an emphasis on verifying that all the plumbing
> works
> >> > > together. So yes, there are definite edge cases I think we'll want
> to
> >> > test
> >> > > here, but I say that with the caveat that I think we should ideally
> >> cover
> >> > > as many non-happy-path cases in unit and integration tests as
> >> possible.
> >> > As
> >> > > an example, I don't think it makes sense to cover most of the
> profiler
> >> > > windowing DSL language edge cases in acceptance tests instead of or
> in
> >> > > addition to unit/integration tests unless there is something
> specific
> >> to
> >> > > the integration with a given an environment that we think could be
> >> > > problematic.
> >> > >
> >> > > M
> >> > >
> >> > > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella 
> >> > wrote:
> >> > >
> >> > > > No, I'm saying that they shouldn't be restricted to real-world
> >> > use-cases.
> >> > > > The E2E tests I laid out weren't real-world, but they did exercise
> >> the
> >> > > > components similar to real-world use-cases.  They should also be
> >> able
> >> > to
> >> > > be
> >> > > > able to tread outside of the happy-path for those use-cases.
> >> > > >
> >> > > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> >> > > > michael.miklav...@gmail.com> wrote:
> >> > > >
> >> > > > > "I don't think acceptance tests should loosely associate with
> real
> >> > > uses,
> >> > > > > but they should
> >> > > > > be free to delve into weird non-happy-pathways."
> >> > > > >
> >> > > > > Not following - are you saying they should *tightly* associate
> >> with
> >> > > real
> >> > > > > uses and additonally include non-happy-path?
> >> > > > >
> >> > > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <
> ceste...@gmail.com
> >> >
> >> > > > wrote:
> >> > > > >
> >> > > > > > It is absolutely not a naive question, Matt.  We don't have a
> >> lot
> >> > (or
> >> > > > > any)
> >> > > > > > docs about our integration tests; it's more of a "follow the
> >> lead"
> >> > > type
> >> > > > > of
> >> > > > > > thing at the moment, but that should be rectified.
> >> > > > > >
> >> > > > > > The integration tests spin up and down infrastructure
> >> in-process,
> >> > > some
> >> > > > of
> >> > > > > > which are real and some of which are mock versions of the
> >> services.
> >> > > > > These
> >> > > > > > are good for catching some types of bugs, but often things
> sneak
> >> > > > through,
> >> > > > > > like:
> >> > > > > >
> >> > > > > >- Hbase and storm can't exist in the same JVM, so HBase is
> >> > mocked
> >> > > in
> >> > > > > >those cases.
> >> > > > > >- The FileSystem that we get for Hadoop is the
> >> > LocalRawFileSystem,
> >> > > > not
> >> > > > > >truly HDFS.  There are differences and we've run into
> >> > > > > them..hilariously
> >> > > > > > at
> >> > > > > >times. ;)
> >> > > > > >- Things done statically in a bolt are shared across all
> >> bolts
> >> > > > because
> >> > >

Re: [DISCUSS][PROPOSAL] Acceptance Tests

2017-03-22 Thread Casey Stella 
I'd be ok with ansible, but we've run into such trouble with
incompatibilities between versions that I'm still a bit gun-shy about it.
That being said, I'm ok with it.

On Wed, Mar 22, 2017 at 3:00 PM, Ryan Merriman  wrote:

> I don't think a cluster installed by ansible is a prerequisite to using
> ansible to integration test.  They would be completely separate modules
> except maybe sharing some property or inventory files.  Just need to run
> scripts and hit rest endpoints right?  Just an idea, maybe it's overkill.
> I'm cool with rolling our own.
>
> On Wed, Mar 22, 2017 at 1:49 PM, Casey Stella  wrote:
>
> > Maybe, but I'd argue that we would want this to be run against a
> > non-ansible installed cluster.  For a first pass, I'd recommend just a
> set
> > of shell scripts utilizing the REPL and the REST API along with shell
> > commands.  Most of our capabilities are quite scriptable.
> >
> > On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman 
> > wrote:
> >
> > > Bumping this thread.  Looks like we have several +1s so I propose we
> move
> > > to the next step.  I'm anxious to get this done because these tests
> would
> > > have saved me time over the last couple weeks.  The management UI in
> > > https://github.com/apache/incubator-metron/pull/484 has a set of e2e
> > tests
> > > being maintained in another branch so those could also be included in
> > this
> > > test suite when the UI makes it into master.
> > >
> > > Ideas for an "Acceptance Testing Framework"?  Could Ansible be good fit
> > for
> > > this since we already have it in our stack?
> > >
> > > On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > Ok, yes I agree. In my experience with e2e/acceptance tests, they're
> > best
> > > > kept general with an emphasis on verifying that all the plumbing
> works
> > > > together. So yes, there are definite edge cases I think we'll want to
> > > test
> > > > here, but I say that with the caveat that I think we should ideally
> > cover
> > > > as many non-happy-path cases in unit and integration tests as
> possible.
> > > As
> > > > an example, I don't think it makes sense to cover most of the
> profiler
> > > > windowing DSL language edge cases in acceptance tests instead of or
> in
> > > > addition to unit/integration tests unless there is something specific
> > to
> > > > the integration with a given an environment that we think could be
> > > > problematic.
> > > >
> > > > M
> > > >
> > > > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella 
> > > wrote:
> > > >
> > > > > No, I'm saying that they shouldn't be restricted to real-world
> > > use-cases.
> > > > > The E2E tests I laid out weren't real-world, but they did exercise
> > the
> > > > > components similar to real-world use-cases.  They should also be
> able
> > > to
> > > > be
> > > > > able to tread outside of the happy-path for those use-cases.
> > > > >
> > > > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> > > > > michael.miklav...@gmail.com> wrote:
> > > > >
> > > > > > "I don't think acceptance tests should loosely associate with
> real
> > > > uses,
> > > > > > but they should
> > > > > > be free to delve into weird non-happy-pathways."
> > > > > >
> > > > > > Not following - are you saying they should *tightly* associate
> with
> > > > real
> > > > > > uses and additonally include non-happy-path?
> > > > > >
> > > > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <
> ceste...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > > > It is absolutely not a naive question, Matt.  We don't have a
> lot
> > > (or
> > > > > > any)
> > > > > > > docs about our integration tests; it's more of a "follow the
> > lead"
> > > > type
> > > > > > of
> > > > > > > thing at the moment, but that should be rectified.
> > > > > > >
> > > > > > > The integration tests spin up and down infrastructure
> in-process,
> > > > some
> > > > > of
> > > > > > > which are real and some of which are mock versions of the
> > services.
> > > > > > These
> > > > > > > are good for catching some types of bugs, but often things
> sneak
> > > > > through,
> > > > > > > like:
> > > > > > >
> > > > > > >- Hbase and storm can't exist in the same JVM, so HBase is
> > > mocked
> > > > in
> > > > > > >those cases.
> > > > > > >- The FileSystem that we get for Hadoop is the
> > > LocalRawFileSystem,
> > > > > not
> > > > > > >truly HDFS.  There are differences and we've run into
> > > > > > them..hilariously
> > > > > > > at
> > > > > > >times. ;)
> > > > > > >- Things done statically in a bolt are shared across all
> bolts
> > > > > because
> > > > > > >they all are threads in the same process
> > > > > > >
> > > > > > > It's good, it catches bugs, it lets us debug things easily, it
> > runs
> > > > > with
> > > > > > > every single build automatically via travis.
> > > > > > > It's bad because it's awkward

Re: [DISCUSS][PROPOSAL] Acceptance Tests

2017-03-22 Thread Ryan Merriman 
I think we'll have non manual rest/web deployment soon regardless of this
discussion.

On Wed, Mar 22, 2017 at 2:00 PM, Ryan Merriman  wrote:

> I don't think a cluster installed by ansible is a prerequisite to using
> ansible to integration test.  They would be completely separate modules
> except maybe sharing some property or inventory files.  Just need to run
> scripts and hit rest endpoints right?  Just an idea, maybe it's overkill.
> I'm cool with rolling our own.
>
> On Wed, Mar 22, 2017 at 1:49 PM, Casey Stella  wrote:
>
>> Maybe, but I'd argue that we would want this to be run against a
>> non-ansible installed cluster.  For a first pass, I'd recommend just a set
>> of shell scripts utilizing the REPL and the REST API along with shell
>> commands.  Most of our capabilities are quite scriptable.
>>
>> On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman 
>> wrote:
>>
>> > Bumping this thread.  Looks like we have several +1s so I propose we
>> move
>> > to the next step.  I'm anxious to get this done because these tests
>> would
>> > have saved me time over the last couple weeks.  The management UI in
>> > https://github.com/apache/incubator-metron/pull/484 has a set of e2e
>> tests
>> > being maintained in another branch so those could also be included in
>> this
>> > test suite when the UI makes it into master.
>> >
>> > Ideas for an "Acceptance Testing Framework"?  Could Ansible be good fit
>> for
>> > this since we already have it in our stack?
>> >
>> > On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
>> > michael.miklav...@gmail.com> wrote:
>> >
>> > > Ok, yes I agree. In my experience with e2e/acceptance tests, they're
>> best
>> > > kept general with an emphasis on verifying that all the plumbing works
>> > > together. So yes, there are definite edge cases I think we'll want to
>> > test
>> > > here, but I say that with the caveat that I think we should ideally
>> cover
>> > > as many non-happy-path cases in unit and integration tests as
>> possible.
>> > As
>> > > an example, I don't think it makes sense to cover most of the profiler
>> > > windowing DSL language edge cases in acceptance tests instead of or in
>> > > addition to unit/integration tests unless there is something specific
>> to
>> > > the integration with a given an environment that we think could be
>> > > problematic.
>> > >
>> > > M
>> > >
>> > > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella 
>> > wrote:
>> > >
>> > > > No, I'm saying that they shouldn't be restricted to real-world
>> > use-cases.
>> > > > The E2E tests I laid out weren't real-world, but they did exercise
>> the
>> > > > components similar to real-world use-cases.  They should also be
>> able
>> > to
>> > > be
>> > > > able to tread outside of the happy-path for those use-cases.
>> > > >
>> > > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
>> > > > michael.miklav...@gmail.com> wrote:
>> > > >
>> > > > > "I don't think acceptance tests should loosely associate with real
>> > > uses,
>> > > > > but they should
>> > > > > be free to delve into weird non-happy-pathways."
>> > > > >
>> > > > > Not following - are you saying they should *tightly* associate
>> with
>> > > real
>> > > > > uses and additonally include non-happy-path?
>> > > > >
>> > > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella > >
>> > > > wrote:
>> > > > >
>> > > > > > It is absolutely not a naive question, Matt.  We don't have a
>> lot
>> > (or
>> > > > > any)
>> > > > > > docs about our integration tests; it's more of a "follow the
>> lead"
>> > > type
>> > > > > of
>> > > > > > thing at the moment, but that should be rectified.
>> > > > > >
>> > > > > > The integration tests spin up and down infrastructure
>> in-process,
>> > > some
>> > > > of
>> > > > > > which are real and some of which are mock versions of the
>> services.
>> > > > > These
>> > > > > > are good for catching some types of bugs, but often things sneak
>> > > > through,
>> > > > > > like:
>> > > > > >
>> > > > > >- Hbase and storm can't exist in the same JVM, so HBase is
>> > mocked
>> > > in
>> > > > > >those cases.
>> > > > > >- The FileSystem that we get for Hadoop is the
>> > LocalRawFileSystem,
>> > > > not
>> > > > > >truly HDFS.  There are differences and we've run into
>> > > > > them..hilariously
>> > > > > > at
>> > > > > >times. ;)
>> > > > > >- Things done statically in a bolt are shared across all
>> bolts
>> > > > because
>> > > > > >they all are threads in the same process
>> > > > > >
>> > > > > > It's good, it catches bugs, it lets us debug things easily, it
>> runs
>> > > > with
>> > > > > > every single build automatically via travis.
>> > > > > > It's bad because it's awkward to get the dependencies isolated
>> > > > > sufficiently
>> > > > > > for all of these components to get them to play nice in the same
>> > JVM.
>> > > > > >
>> > > > > > Acceptance tests would

Re: [DISCUSS][PROPOSAL] Acceptance Tests

2017-03-22 Thread Otto Fowler 
Does this mean we’ll have non manual rest/web deployment soon?


On March 22, 2017 at 14:47:34, Ryan Merriman (merrim...@gmail.com) wrote:

Bumping this thread. Looks like we have several +1s so I propose we move
to the next step. I'm anxious to get this done because these tests would
have saved me time over the last couple weeks. The management UI in
https://github.com/apache/incubator-metron/pull/484 has a set of e2e tests
being maintained in another branch so those could also be included in this
test suite when the UI makes it into master.

Ideas for an "Acceptance Testing Framework"? Could Ansible be good fit for
this since we already have it in our stack?

On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Ok, yes I agree. In my experience with e2e/acceptance tests, they're best
> kept general with an emphasis on verifying that all the plumbing works
> together. So yes, there are definite edge cases I think we'll want to
test
> here, but I say that with the caveat that I think we should ideally cover
> as many non-happy-path cases in unit and integration tests as possible.
As
> an example, I don't think it makes sense to cover most of the profiler
> windowing DSL language edge cases in acceptance tests instead of or in
> addition to unit/integration tests unless there is something specific to
> the integration with a given an environment that we think could be
> problematic.
>
> M
>
> On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella  wrote:
>
> > No, I'm saying that they shouldn't be restricted to real-world
use-cases.
> > The E2E tests I laid out weren't real-world, but they did exercise the
> > components similar to real-world use-cases. They should also be able to
> be
> > able to tread outside of the happy-path for those use-cases.
> >
> > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > "I don't think acceptance tests should loosely associate with real
> uses,
> > > but they should
> > > be free to delve into weird non-happy-pathways."
> > >
> > > Not following - are you saying they should *tightly* associate with
> real
> > > uses and additonally include non-happy-path?
> > >
> > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella 
> > wrote:
> > >
> > > > It is absolutely not a naive question, Matt. We don't have a lot
(or
> > > any)
> > > > docs about our integration tests; it's more of a "follow the lead"
> type
> > > of
> > > > thing at the moment, but that should be rectified.
> > > >
> > > > The integration tests spin up and down infrastructure in-process,
> some
> > of
> > > > which are real and some of which are mock versions of the services.
> > > These
> > > > are good for catching some types of bugs, but often things sneak
> > through,
> > > > like:
> > > >
> > > > - Hbase and storm can't exist in the same JVM, so HBase is mocked
> in
> > > > those cases.
> > > > - The FileSystem that we get for Hadoop is the LocalRawFileSystem,
> > not
> > > > truly HDFS. There are differences and we've run into
> > > them..hilariously
> > > > at
> > > > times. ;)
> > > > - Things done statically in a bolt are shared across all bolts
> > because
> > > > they all are threads in the same process
> > > >
> > > > It's good, it catches bugs, it lets us debug things easily, it runs
> > with
> > > > every single build automatically via travis.
> > > > It's bad because it's awkward to get the dependencies isolated
> > > sufficiently
> > > > for all of these components to get them to play nice in the same
JVM.
> > > >
> > > > Acceptance tests would be run against a real cluster, so they
would:
> > > >
> > > > - run against real components, not testing or mock components
> > > > - run against multiple nodes
> > > >
> > > > I can imagine a world where we can unify the two to a certain
degree
> in
> > > > many cases if we could spin up a docker version of Metron to run as
> > part
> > > of
> > > > the build, but I think in the meantime, we should focus on
providing
> > > both.
> > > >
> > > > I suspect the reference application is possibly inspiring my
> > suggestions
> > > > here, but I think the main difference here is that the reference
> > > > application is intended to be informational from a end-user
> > perspective:
> > > > it's detailing a use-case that users will understand. I don't think
> > > > acceptance tests should loosely associate with real uses, but they
> > should
> > > > be free to delve into weird non-happy-pathways.
> > > >
> > > > On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley 
wrote:
> > > >
> > > > > Automating stuff that now has to be done manually gets a big +1.
> > > > >
> > > > > But, Casey, could you please clarify the relationship between
what
> > you
> > > > > plan to do and the current “integration test” framework? Will
this
> > be
> > > in
> > > > > the form of additional integration tests? Or a different test
> > > framework?
> > > > > Can

Re: [DISCUSS][PROPOSAL] Acceptance Tests

2017-03-22 Thread Ryan Merriman 
I don't think a cluster installed by ansible is a prerequisite to using
ansible to integration test.  They would be completely separate modules
except maybe sharing some property or inventory files.  Just need to run
scripts and hit rest endpoints right?  Just an idea, maybe it's overkill.
I'm cool with rolling our own.

On Wed, Mar 22, 2017 at 1:49 PM, Casey Stella  wrote:

> Maybe, but I'd argue that we would want this to be run against a
> non-ansible installed cluster.  For a first pass, I'd recommend just a set
> of shell scripts utilizing the REPL and the REST API along with shell
> commands.  Most of our capabilities are quite scriptable.
>
> On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman 
> wrote:
>
> > Bumping this thread.  Looks like we have several +1s so I propose we move
> > to the next step.  I'm anxious to get this done because these tests would
> > have saved me time over the last couple weeks.  The management UI in
> > https://github.com/apache/incubator-metron/pull/484 has a set of e2e
> tests
> > being maintained in another branch so those could also be included in
> this
> > test suite when the UI makes it into master.
> >
> > Ideas for an "Acceptance Testing Framework"?  Could Ansible be good fit
> for
> > this since we already have it in our stack?
> >
> > On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > Ok, yes I agree. In my experience with e2e/acceptance tests, they're
> best
> > > kept general with an emphasis on verifying that all the plumbing works
> > > together. So yes, there are definite edge cases I think we'll want to
> > test
> > > here, but I say that with the caveat that I think we should ideally
> cover
> > > as many non-happy-path cases in unit and integration tests as possible.
> > As
> > > an example, I don't think it makes sense to cover most of the profiler
> > > windowing DSL language edge cases in acceptance tests instead of or in
> > > addition to unit/integration tests unless there is something specific
> to
> > > the integration with a given an environment that we think could be
> > > problematic.
> > >
> > > M
> > >
> > > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella 
> > wrote:
> > >
> > > > No, I'm saying that they shouldn't be restricted to real-world
> > use-cases.
> > > > The E2E tests I laid out weren't real-world, but they did exercise
> the
> > > > components similar to real-world use-cases.  They should also be able
> > to
> > > be
> > > > able to tread outside of the happy-path for those use-cases.
> > > >
> > > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > "I don't think acceptance tests should loosely associate with real
> > > uses,
> > > > > but they should
> > > > > be free to delve into weird non-happy-pathways."
> > > > >
> > > > > Not following - are you saying they should *tightly* associate with
> > > real
> > > > > uses and additonally include non-happy-path?
> > > > >
> > > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella 
> > > > wrote:
> > > > >
> > > > > > It is absolutely not a naive question, Matt.  We don't have a lot
> > (or
> > > > > any)
> > > > > > docs about our integration tests; it's more of a "follow the
> lead"
> > > type
> > > > > of
> > > > > > thing at the moment, but that should be rectified.
> > > > > >
> > > > > > The integration tests spin up and down infrastructure in-process,
> > > some
> > > > of
> > > > > > which are real and some of which are mock versions of the
> services.
> > > > > These
> > > > > > are good for catching some types of bugs, but often things sneak
> > > > through,
> > > > > > like:
> > > > > >
> > > > > >- Hbase and storm can't exist in the same JVM, so HBase is
> > mocked
> > > in
> > > > > >those cases.
> > > > > >- The FileSystem that we get for Hadoop is the
> > LocalRawFileSystem,
> > > > not
> > > > > >truly HDFS.  There are differences and we've run into
> > > > > them..hilariously
> > > > > > at
> > > > > >times. ;)
> > > > > >- Things done statically in a bolt are shared across all bolts
> > > > because
> > > > > >they all are threads in the same process
> > > > > >
> > > > > > It's good, it catches bugs, it lets us debug things easily, it
> runs
> > > > with
> > > > > > every single build automatically via travis.
> > > > > > It's bad because it's awkward to get the dependencies isolated
> > > > > sufficiently
> > > > > > for all of these components to get them to play nice in the same
> > JVM.
> > > > > >
> > > > > > Acceptance tests would be run against a real cluster, so they
> > would:
> > > > > >
> > > > > >- run against real components, not testing or mock components
> > > > > >- run against multiple nodes
> > > > > >
> > > > > > I can imagine a world where we can unify the two to a certain
> > degree
> > > in
> > > > > > many cases if we

Re: [DISCUSS][PROPOSAL] Acceptance Tests

2017-03-22 Thread Casey Stella 
Maybe, but I'd argue that we would want this to be run against a
non-ansible installed cluster.  For a first pass, I'd recommend just a set
of shell scripts utilizing the REPL and the REST API along with shell
commands.  Most of our capabilities are quite scriptable.

On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman  wrote:

> Bumping this thread.  Looks like we have several +1s so I propose we move
> to the next step.  I'm anxious to get this done because these tests would
> have saved me time over the last couple weeks.  The management UI in
> https://github.com/apache/incubator-metron/pull/484 has a set of e2e tests
> being maintained in another branch so those could also be included in this
> test suite when the UI makes it into master.
>
> Ideas for an "Acceptance Testing Framework"?  Could Ansible be good fit for
> this since we already have it in our stack?
>
> On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
> michael.miklav...@gmail.com> wrote:
>
> > Ok, yes I agree. In my experience with e2e/acceptance tests, they're best
> > kept general with an emphasis on verifying that all the plumbing works
> > together. So yes, there are definite edge cases I think we'll want to
> test
> > here, but I say that with the caveat that I think we should ideally cover
> > as many non-happy-path cases in unit and integration tests as possible.
> As
> > an example, I don't think it makes sense to cover most of the profiler
> > windowing DSL language edge cases in acceptance tests instead of or in
> > addition to unit/integration tests unless there is something specific to
> > the integration with a given an environment that we think could be
> > problematic.
> >
> > M
> >
> > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella 
> wrote:
> >
> > > No, I'm saying that they shouldn't be restricted to real-world
> use-cases.
> > > The E2E tests I laid out weren't real-world, but they did exercise the
> > > components similar to real-world use-cases.  They should also be able
> to
> > be
> > > able to tread outside of the happy-path for those use-cases.
> > >
> > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> > > michael.miklav...@gmail.com> wrote:
> > >
> > > > "I don't think acceptance tests should loosely associate with real
> > uses,
> > > > but they should
> > > > be free to delve into weird non-happy-pathways."
> > > >
> > > > Not following - are you saying they should *tightly* associate with
> > real
> > > > uses and additonally include non-happy-path?
> > > >
> > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella 
> > > wrote:
> > > >
> > > > > It is absolutely not a naive question, Matt.  We don't have a lot
> (or
> > > > any)
> > > > > docs about our integration tests; it's more of a "follow the lead"
> > type
> > > > of
> > > > > thing at the moment, but that should be rectified.
> > > > >
> > > > > The integration tests spin up and down infrastructure in-process,
> > some
> > > of
> > > > > which are real and some of which are mock versions of the services.
> > > > These
> > > > > are good for catching some types of bugs, but often things sneak
> > > through,
> > > > > like:
> > > > >
> > > > >- Hbase and storm can't exist in the same JVM, so HBase is
> mocked
> > in
> > > > >those cases.
> > > > >- The FileSystem that we get for Hadoop is the
> LocalRawFileSystem,
> > > not
> > > > >truly HDFS.  There are differences and we've run into
> > > > them..hilariously
> > > > > at
> > > > >times. ;)
> > > > >- Things done statically in a bolt are shared across all bolts
> > > because
> > > > >they all are threads in the same process
> > > > >
> > > > > It's good, it catches bugs, it lets us debug things easily, it runs
> > > with
> > > > > every single build automatically via travis.
> > > > > It's bad because it's awkward to get the dependencies isolated
> > > > sufficiently
> > > > > for all of these components to get them to play nice in the same
> JVM.
> > > > >
> > > > > Acceptance tests would be run against a real cluster, so they
> would:
> > > > >
> > > > >- run against real components, not testing or mock components
> > > > >- run against multiple nodes
> > > > >
> > > > > I can imagine a world where we can unify the two to a certain
> degree
> > in
> > > > > many cases if we could spin up a docker version of Metron to run as
> > > part
> > > > of
> > > > > the build, but I think in the meantime, we should focus on
> providing
> > > > both.
> > > > >
> > > > > I suspect the reference application is possibly inspiring my
> > > suggestions
> > > > > here, but I think the main difference here is that the reference
> > > > > application is intended to be informational from a end-user
> > > perspective:
> > > > > it's detailing a use-case that users will understand.  I don't
> think
> > > > > acceptance tests should loosely associate with real uses, but they
> > > should
> > > > > be free to delve into weird

Re: [DISCUSS][PROPOSAL] Acceptance Tests

2017-03-22 Thread Ryan Merriman 
Bumping this thread.  Looks like we have several +1s so I propose we move
to the next step.  I'm anxious to get this done because these tests would
have saved me time over the last couple weeks.  The management UI in
https://github.com/apache/incubator-metron/pull/484 has a set of e2e tests
being maintained in another branch so those could also be included in this
test suite when the UI makes it into master.

Ideas for an "Acceptance Testing Framework"?  Could Ansible be good fit for
this since we already have it in our stack?

On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
michael.miklav...@gmail.com> wrote:

> Ok, yes I agree. In my experience with e2e/acceptance tests, they're best
> kept general with an emphasis on verifying that all the plumbing works
> together. So yes, there are definite edge cases I think we'll want to test
> here, but I say that with the caveat that I think we should ideally cover
> as many non-happy-path cases in unit and integration tests as possible. As
> an example, I don't think it makes sense to cover most of the profiler
> windowing DSL language edge cases in acceptance tests instead of or in
> addition to unit/integration tests unless there is something specific to
> the integration with a given an environment that we think could be
> problematic.
>
> M
>
> On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella  wrote:
>
> > No, I'm saying that they shouldn't be restricted to real-world use-cases.
> > The E2E tests I laid out weren't real-world, but they did exercise the
> > components similar to real-world use-cases.  They should also be able to
> be
> > able to tread outside of the happy-path for those use-cases.
> >
> > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> > michael.miklav...@gmail.com> wrote:
> >
> > > "I don't think acceptance tests should loosely associate with real
> uses,
> > > but they should
> > > be free to delve into weird non-happy-pathways."
> > >
> > > Not following - are you saying they should *tightly* associate with
> real
> > > uses and additonally include non-happy-path?
> > >
> > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella 
> > wrote:
> > >
> > > > It is absolutely not a naive question, Matt.  We don't have a lot (or
> > > any)
> > > > docs about our integration tests; it's more of a "follow the lead"
> type
> > > of
> > > > thing at the moment, but that should be rectified.
> > > >
> > > > The integration tests spin up and down infrastructure in-process,
> some
> > of
> > > > which are real and some of which are mock versions of the services.
> > > These
> > > > are good for catching some types of bugs, but often things sneak
> > through,
> > > > like:
> > > >
> > > >- Hbase and storm can't exist in the same JVM, so HBase is mocked
> in
> > > >those cases.
> > > >- The FileSystem that we get for Hadoop is the LocalRawFileSystem,
> > not
> > > >truly HDFS.  There are differences and we've run into
> > > them..hilariously
> > > > at
> > > >times. ;)
> > > >- Things done statically in a bolt are shared across all bolts
> > because
> > > >they all are threads in the same process
> > > >
> > > > It's good, it catches bugs, it lets us debug things easily, it runs
> > with
> > > > every single build automatically via travis.
> > > > It's bad because it's awkward to get the dependencies isolated
> > > sufficiently
> > > > for all of these components to get them to play nice in the same JVM.
> > > >
> > > > Acceptance tests would be run against a real cluster, so they would:
> > > >
> > > >- run against real components, not testing or mock components
> > > >- run against multiple nodes
> > > >
> > > > I can imagine a world where we can unify the two to a certain degree
> in
> > > > many cases if we could spin up a docker version of Metron to run as
> > part
> > > of
> > > > the build, but I think in the meantime, we should focus on providing
> > > both.
> > > >
> > > > I suspect the reference application is possibly inspiring my
> > suggestions
> > > > here, but I think the main difference here is that the reference
> > > > application is intended to be informational from a end-user
> > perspective:
> > > > it's detailing a use-case that users will understand.  I don't think
> > > > acceptance tests should loosely associate with real uses, but they
> > should
> > > > be free to delve into weird non-happy-pathways.
> > > >
> > > > On Fri, Mar 3, 2017 at 2:16 PM, Matt Foley  wrote:
> > > >
> > > > > Automating stuff that now has to be done manually gets a big +1.
> > > > >
> > > > > But, Casey, could you please clarify the relationship between what
> > you
> > > > > plan to do and the current “integration test” framework?  Will this
> > be
> > > in
> > > > > the form of additional integration tests? Or a different test
> > > framework?
> > > > > Can it be done in the integration test framework, rather than
> > creating
> > > > new
> > > > > mechanism?

Re: [DISCUSS] Stepping down as release manager

2017-03-22 Thread Matt Foley 
As Billie says, our bylaws don’t require a vote to assign a release manager.
However, an RM certainly needs the support of the community to be effective,
so all these +1’s are very much appreciated! :-)
Thanks,
--Matt

On 3/22/17, 8:30 AM, "Billie Rinaldi"  wrote:

Not everything needs to be voted upon. The ASF guidelines and Metron's
bylaws specify which actions require a vote. Other types of things, like
choosing a release manager, can be approved by lazy consensus. I just
noticed that Metron's bylaws have slightly different definitions for the
approval types than the ASF typically uses. I'd recommend changing these to
the standard ASF definitions. Specifically, Lazy Consensus does not require
a vote and thus no +1s are needed (https://www.apache.org/
foundation/glossary.html#LazyConsensus) and Lazy Majority and Lazy 2/3
Majority should be called Majority (https://www.apache.org/
foundation/glossary.html#MajorityApproval) and 2/3 Majority.

On Wed, Mar 22, 2017 at 7:23 AM, Justin Leet  wrote:

> Right now it's just support, not a vote.  I assume, based on our past
> practices, that there will be a separate [VOTE] thread.
>
> Justin
>
> On Wed, Mar 22, 2017 at 10:06 AM, Otto Fowler 
> wrote:
>
> > +1 but is this explicitly an official vote?
> >
> >
> > On March 21, 2017 at 13:51:16, Justin Leet (justinjl...@gmail.com)
> wrote:
> >
> > +1 for Matt
> >
> > On Tue, Mar 21, 2017 at 12:21 PM, zeo...@gmail.com 
> > wrote:
> >
> > > +1 for mattf
> > >
> > > On Tue, Mar 21, 2017 at 11:04 AM Ryan Merriman 
> > > wrote:
> > >
> > > > +1 for Matt
> > > >
> > > > On Tue, Mar 21, 2017 at 9:44 AM, Matt Foley 
> wrote:
> > > >
> > > > > Casey, you’ve been a great release manager. I know how much detail
> > > > effort
> > > > > goes into this role.
> > > > >
> > > > > I am willing to serve as RM for the next while, if the community
> > would
> > > > > like. I was the RM for Hadoop for about a year, and in fact was RM
> > for
> > > > its
> > > > > 1.0 release. Granted that was a while ago, but overall process
> > doesn’t
> > > > seem
> > > > > to have changed much :-)
> > > > >
> > > > > Cheers,
> > > > > --Matt
> > > > >
> > > > > On 3/21/17, 7:32 AM, "Casey Stella"  wrote:
> > > > >
> > > > > Right, Billie is exactly right. Working with the community to
> > > > > constructing
> > > > > releases that conform to apache standards and policies is the main
> > > > > duty.
> > > > > This will (hopefully) be our first set of releases outside of the
> > > > > incubator, so if I'm allowed to be biased, I'm hoping that someone
> > > > with
> > > > > previous release management experience in other projects will
> > > > > volunteer.
> > > > > We're leaving the nest a bit and having an experienced hand at the
> > > > > tiller
> > > > > would be advantageous.
> > > > >
> > > > >
> > > > > On Tue, Mar 21, 2017 at 10:21 AM, Billie Rinaldi <
> > > bil...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > See http://www.apache.org/dev/release-publishing#release_manager
> > > > and
> > > > > > http://www.apache.org/legal/release-policy.html for information
> > > on
> > > > > the
> > > > > > tasks that a release manager performs.
> > > > > >
> > > > > > On Tue, Mar 21, 2017 at 7:10 AM, Khurram Ahmed <
> > > > > khurramah...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Casey it would be helpful if you could outline the
> > > > > responsibilities of a
> > > > > > > release manager for the Metron project.
> > > > > > >
> > > > > > > On Mar 21, 2017 6:57 PM, "Casey Stella" 
> > > > > wrote:
> > > > > > >
> > > > > > > > I've been extremely honored to spend the last few months as
> > > the
> > > > > Metron
> > > > > > > > Release Manager. That being said, my watch is ended and it's
> > > > > time for
> > > > > > > > another release manager to step into my place.
> > > > > > > >
> > > > > > > > Who would like to volunteer to be release manager for the
> > > next
> > > > > release
> > > > > > of
> > > > > > > > Metron?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Casey
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > --
> > >
> > > Jon
> > >
> >
> >
>

[GitHub] incubator-metron issue #487: METRON-792: Quick Dev should remove/replace RPM...

2017-03-22 Thread cestella 
Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/487
  
+1 by inspection, this one bit me last night.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #487: METRON-792: Quick Dev should remove/repl...

2017-03-22 Thread dlyle65535 
GitHub user dlyle65535 opened a pull request:

https://github.com/apache/incubator-metron/pull/487

METRON-792: Quick Dev should remove/replace RPM packages

## Contributor Comments
While building out METRON-671, I inadvertently omitted recreation of the 
RPM database for Quick Dev. As a result, stale RPMs are installed so, Quick Dev 
removes and then re-installs the same bits. 

This fixes that. Now it clears out /localrepo and re-runs createrepo.

To test:

- Merge this PR
- Run _vagrant up_ from _project 
directory/metron-deployment/vagrant/quick-dev-platform_. 
- Once deployment is complete, _vagrant ssh_ to node1.
_project directory/metron-deployment/vagrant/quick-dev-platform_. 
- Once deployment is complete, _vagrant ssh_ to node1.
- Verify that the RPMs in /localrepo are timestamped the same as those in 
_project directory/metron-deployment/packaging/docker/rpm-docker/RPMS/noarch_

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron (Incubating).  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [X] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [X] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [X] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [N/A] Have you included steps to reproduce the behavior or problem that 
is being changed or addressed?
- [X] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [X] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [N/A] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [N/A] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [X] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [N/A] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  bin/generate-md.sh
  mvn site:site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommened that [travis-ci](https://travis-ci.org) is set up for 
your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dlyle65535/incubator-metron METRON-792

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/incubator-metron/pull/487.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #487


commit f8e8e999096aa6963fa1ca96e186c14320afbc43
Author: David Lyle 
Date:   2017-03-22T16:46:48Z

METRON-792: Quick Dev should remove/replace RPM packages




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[MENTORS] initiating the TLP vote

2017-03-22 Thread James Sirota 
Mentors,

Looks like we have everything we need to initiate the incubator board vote to 
leave the incubator.  Can you provide some guidance as to what is the best way 
to approach this?

--- 
Thank you,

James Sirota
PPMC- Apache Metron (Incubating)
jsirota AT apache DOT org

[RESULT][VOTE] Final Board Resolution Draft V2

2017-03-22 Thread James Sirota 
Vote passes with 4 binding +1s (casey, david, kyle, james)

on to the next steps...

22.03.2017, 09:50, "James Sirota" :
> + 1 (binding)
>
> 22.03.2017, 08:30, "Casey Stella" :
>>  +1 binding
>>
>>  On Wed, Mar 22, 2017 at 11:23 AM, David Lyle  wrote:
>>
>>>   +1 binding
>>>
>>>   On Wed, Mar 22, 2017 at 10:49 AM, Kyle Richardson <
>>>   kylerichards...@gmail.com
>>>   > wrote:
>>>
>>>   > +1 (binding)
>>>   >
>>>   > On Mon, Mar 20, 2017 at 3:05 AM, James Sirota 
>>>   wrote:
>>>   >
>>>   > >
>>>   > > - Removed affiliations
>>>   > > - Added apache IDs where possible
>>>   > > - Removed committers and only left PPMC members
>>>   > >
>>>   > > Hope this version holds up. Please vote +1, -1, or 0 for neutral. The
>>>   > > vote will be open for 72 hours
>>>   > >
>>>   > >
>>>   > > The incubating Apache Metron community believes it is time to graduate
>>>   to
>>>   > > TLP.
>>>   > >
>>>   > > Apache Metron entered incubation in December of 2015. Since then, 
>>> we've
>>>   > > overcome technical challenges to remove Category X dependencies, and
>>>   > made 3
>>>   > > releases. Our most recent release contains binary convenience
>>>   artifacts.
>>>   > We
>>>   > > are a very helpful and engaged community, ready to answer all 
>>> questions
>>>   > and
>>>   > > feedback directed to us via the user list. Through our time in
>>>   incubation
>>>   > > we've added a number of committers and promoted some of them to PPMC
>>>   > > membership. We are actively pursuing others. While we do still have
>>>   > issues
>>>   > > to address raised by means of the maturity model, all projects are
>>>   > ongoing
>>>   > > processes, and we believe we no longer need the incubator to continue
>>>   > > addressing these issues.
>>>   > >
>>>   > > To inform the discussion, here is some basic project information:
>>>   > >
>>>   > > Project status:
>>>   > > http://incubator.apache.org/projects/metron.html
>>>   > >
>>>   > > Project website:
>>>   > > https://metron.incubator.apache.org/
>>>   > >
>>>   > > Project documentation:
>>>   > > https://cwiki.apache.org/confluence/display/METRON/Documentation
>>>   > >
>>>   > > Maturity assessment:
>>>   > > https://cwiki.apache.org/confluence/display/METRON/
>>>   > > Apache+Project+Maturity+Model
>>>   > >
>>>   > > DRAFT of the board resolution is at the bottom of this email
>>>   > >
>>>   > > Proposed PMC size: 25 members
>>>   > >
>>>   > > Total number of committers: 6 members
>>>   > >
>>>   > >
>>>   > > 516 commits on develop
>>>   > > 34 contributors across all branches
>>>   > >
>>>   > > dev list averaged ~650 msgs/month for the last 3 months
>>>   > >
>>>   > >
>>>   > > Resolution:
>>>   > >
>>>   > > Establish the Apache Metron Project
>>>   > >
>>>   > > WHEREAS, the Board of Directors deems it to be in the best
>>>   > > interests of the Foundation and consistent with the
>>>   > > Foundation's purpose to establish a Project Management
>>>   > > Committee charged with the creation and maintenance of
>>>   > > open-source software, for distribution at no charge to the
>>>   > > public, related to a security analytics platform for big data use
>>>   cases.
>>>   > >
>>>   > > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>>   > > Committee (PMC), to be known as the "Apache Metron Project",
>>>   > > be and hereby is established pursuant to Bylaws of the
>>>   > > Foundation; and be it further
>>>   > >
>>>   > > RESOLVED, that the Apache Metron Project be and hereby is
>>>   > > responsible for the creation and maintenance of software
>>>   > > related to:
>>>   > > (a) A mechanism to capture, store, and normalize any type of security
>>>   > > telemetry at extremely high rates.
>>>   > > (b) Real time processing and application of enrichments
>>>   > > (c) Efficient information storage
>>>   > > (d) An interface that gives a security investigator a centralized view
>>>   of
>>>   > > data and alerts passed through the system.
>>>   > >
>>>   > > RESOLVED, that the office of "Vice President, Apache Metron" be
>>>   > > and hereby is created, the person holding such office to
>>>   > > serve at the direction of the Board of Directors as the chair
>>>   > > of the Apache Metron Project, and to have primary responsibility
>>>   > > for management of the projects within the scope of
>>>   > > responsibility of the Apache Metron Project; and be it further
>>>   > >
>>>   > > RESOLVED, that the persons listed immediately below be and
>>>   > > hereby are appointed to serve as the initial members of the
>>>   > > Apache Metron Project:
>>>   > >
>>>   > >
>>>   > > PPMC:
>>>   > > Mark Bittmann (mbittmann)
>>>   > > Sheetal Dolas (sheetal_dolas)
>>>   > > Debo Dutta (ddutta)
>>>   > > Discovery Gerdes (discovery)
>>>   > > Andrew Hartnett (dev_warlord)
>>>   > > Dave Hirko (dbhirko)
>>>   > > Paul Kehrer (reaperhulk)
>>>   > > Brad Kolarov (bjkolly)
>>>   > > Kiran

Re: [VOTE] Final Board Resolution Draft V2

2017-03-22 Thread Matt Foley 
+1 (non-binding)

On 3/22/17, 8:30 AM, "Casey Stella"  wrote:

+1 binding

On Wed, Mar 22, 2017 at 11:23 AM, David Lyle  wrote:

> +1 binding
>
> On Wed, Mar 22, 2017 at 10:49 AM, Kyle Richardson <
> kylerichards...@gmail.com
> > wrote:
>
> > +1 (binding)
> >
> > On Mon, Mar 20, 2017 at 3:05 AM, James Sirota 
> wrote:
> >
> > >
> > > - Removed affiliations
> > > - Added apache IDs where possible
> > > - Removed committers and only left PPMC members
> > >
> > > Hope this version holds up.  Please vote +1, -1, or 0 for neutral.  
The
> > > vote will be open for 72 hours
> > >
> > >
> > > The incubating Apache Metron community believes it is time to graduate
> to
> > > TLP.
> > >
> > > Apache Metron entered incubation in December of 2015. Since then, 
we've
> > > overcome technical challenges to remove Category X dependencies, and
> > made 3
> > > releases. Our most recent release contains binary convenience
> artifacts.
> > We
> > > are a very helpful and engaged community, ready to answer all 
questions
> > and
> > > feedback directed to us via the user list. Through our time in
> incubation
> > > we've added a number of committers and promoted some of them to PPMC
> > > membership. We are actively pursuing others. While we do still have
> > issues
> > > to address raised by means of the maturity model, all projects are
> > ongoing
> > > processes, and we believe we no longer need the incubator to continue
> > > addressing these issues.
> > >
> > > To inform the discussion, here is some basic project information:
> > >
> > > Project status:
> > >   http://incubator.apache.org/projects/metron.html
> > >
> > > Project website:
> > >   https://metron.incubator.apache.org/
> > >
> > > Project documentation:
> > >https://cwiki.apache.org/confluence/display/METRON/Documentation
> > >
> > > Maturity assessment:
> > >https://cwiki.apache.org/confluence/display/METRON/
> > > Apache+Project+Maturity+Model
> > >
> > > DRAFT of the board resolution is at the bottom of this email
> > >
> > > Proposed PMC size: 25 members
> > >
> > > Total number of committers: 6 members
> > >
> > >
> > > 516 commits on develop
> > > 34 contributors across all branches
> > >
> > > dev list averaged ~650 msgs/month for the last 3 months
> > >
> > >
> > > Resolution:
> > >
> > > Establish the Apache Metron Project
> > >
> > > WHEREAS, the Board of Directors deems it to be in the best
> > > interests of the Foundation and consistent with the
> > > Foundation's purpose to establish a Project Management
> > > Committee charged with the creation and maintenance of
> > > open-source software, for distribution at no charge to the
> > > public, related to a security analytics platform for big data use
> cases.
> > >
> > > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > > Committee (PMC), to be known as the "Apache Metron Project",
> > > be and hereby is established pursuant to Bylaws of the
> > > Foundation; and be it further
> > >
> > > RESOLVED, that the Apache Metron Project be and hereby is
> > > responsible for the creation and maintenance of software
> > > related to:
> > > (a) A mechanism to capture, store, and normalize any type of security
> > > telemetry at extremely high rates.
> > > (b) Real time processing and application of enrichments
> > > (c) Efficient information storage
> > > (d) An interface that gives a security investigator a centralized view
> of
> > > data and alerts passed through the system.
> > >
> > > RESOLVED, that the office of "Vice President, Apache Metron" be
> > > and hereby is created, the person holding such office to
> > > serve at the direction of the Board of Directors as the chair
> > > of the Apache Metron Project, and to have primary responsibility
> > > for management of the projects within the scope of
> > > responsibility of the Apache Metron Project; and be it further
> > >
> > > RESOLVED, that the persons listed immediately below be and
> > > hereby are appointed to serve as the initial members of the
> > > Apache Metron Project:
> > >
> > >
> > > PPMC:
> > > Mark Bittmann (mbittmann)
> > > Sheetal Dolas (sheetal_dolas)
> > > Debo Dutta (ddutta)
> > > Discovery Gerdes (discovery)
> > > Andrew Hartnett (dev_warlord)
> > > Dave Hirko (dbhirko)
> > > Paul Kehrer (reaperhulk)
> > > Brad Kolarov (bjkolly)
> > > Kiran Komaravolu (UKNOWN)
> > > Larry McCay (lmccay)
> > > P. Taylor Goetz (ptgoetz)
> > > Ryan Merriman

Re: [VOTE] Final Board Resolution Draft V2

2017-03-22 Thread James Sirota 
+ 1 (binding)


22.03.2017, 08:30, "Casey Stella" :
> +1 binding
>
> On Wed, Mar 22, 2017 at 11:23 AM, David Lyle  wrote:
>
>>  +1 binding
>>
>>  On Wed, Mar 22, 2017 at 10:49 AM, Kyle Richardson <
>>  kylerichards...@gmail.com
>>  > wrote:
>>
>>  > +1 (binding)
>>  >
>>  > On Mon, Mar 20, 2017 at 3:05 AM, James Sirota 
>>  wrote:
>>  >
>>  > >
>>  > > - Removed affiliations
>>  > > - Added apache IDs where possible
>>  > > - Removed committers and only left PPMC members
>>  > >
>>  > > Hope this version holds up. Please vote +1, -1, or 0 for neutral. The
>>  > > vote will be open for 72 hours
>>  > >
>>  > >
>>  > > The incubating Apache Metron community believes it is time to graduate
>>  to
>>  > > TLP.
>>  > >
>>  > > Apache Metron entered incubation in December of 2015. Since then, we've
>>  > > overcome technical challenges to remove Category X dependencies, and
>>  > made 3
>>  > > releases. Our most recent release contains binary convenience
>>  artifacts.
>>  > We
>>  > > are a very helpful and engaged community, ready to answer all questions
>>  > and
>>  > > feedback directed to us via the user list. Through our time in
>>  incubation
>>  > > we've added a number of committers and promoted some of them to PPMC
>>  > > membership. We are actively pursuing others. While we do still have
>>  > issues
>>  > > to address raised by means of the maturity model, all projects are
>>  > ongoing
>>  > > processes, and we believe we no longer need the incubator to continue
>>  > > addressing these issues.
>>  > >
>>  > > To inform the discussion, here is some basic project information:
>>  > >
>>  > > Project status:
>>  > > http://incubator.apache.org/projects/metron.html
>>  > >
>>  > > Project website:
>>  > > https://metron.incubator.apache.org/
>>  > >
>>  > > Project documentation:
>>  > > https://cwiki.apache.org/confluence/display/METRON/Documentation
>>  > >
>>  > > Maturity assessment:
>>  > > https://cwiki.apache.org/confluence/display/METRON/
>>  > > Apache+Project+Maturity+Model
>>  > >
>>  > > DRAFT of the board resolution is at the bottom of this email
>>  > >
>>  > > Proposed PMC size: 25 members
>>  > >
>>  > > Total number of committers: 6 members
>>  > >
>>  > >
>>  > > 516 commits on develop
>>  > > 34 contributors across all branches
>>  > >
>>  > > dev list averaged ~650 msgs/month for the last 3 months
>>  > >
>>  > >
>>  > > Resolution:
>>  > >
>>  > > Establish the Apache Metron Project
>>  > >
>>  > > WHEREAS, the Board of Directors deems it to be in the best
>>  > > interests of the Foundation and consistent with the
>>  > > Foundation's purpose to establish a Project Management
>>  > > Committee charged with the creation and maintenance of
>>  > > open-source software, for distribution at no charge to the
>>  > > public, related to a security analytics platform for big data use
>>  cases.
>>  > >
>>  > > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
>>  > > Committee (PMC), to be known as the "Apache Metron Project",
>>  > > be and hereby is established pursuant to Bylaws of the
>>  > > Foundation; and be it further
>>  > >
>>  > > RESOLVED, that the Apache Metron Project be and hereby is
>>  > > responsible for the creation and maintenance of software
>>  > > related to:
>>  > > (a) A mechanism to capture, store, and normalize any type of security
>>  > > telemetry at extremely high rates.
>>  > > (b) Real time processing and application of enrichments
>>  > > (c) Efficient information storage
>>  > > (d) An interface that gives a security investigator a centralized view
>>  of
>>  > > data and alerts passed through the system.
>>  > >
>>  > > RESOLVED, that the office of "Vice President, Apache Metron" be
>>  > > and hereby is created, the person holding such office to
>>  > > serve at the direction of the Board of Directors as the chair
>>  > > of the Apache Metron Project, and to have primary responsibility
>>  > > for management of the projects within the scope of
>>  > > responsibility of the Apache Metron Project; and be it further
>>  > >
>>  > > RESOLVED, that the persons listed immediately below be and
>>  > > hereby are appointed to serve as the initial members of the
>>  > > Apache Metron Project:
>>  > >
>>  > >
>>  > > PPMC:
>>  > > Mark Bittmann (mbittmann)
>>  > > Sheetal Dolas (sheetal_dolas)
>>  > > Debo Dutta (ddutta)
>>  > > Discovery Gerdes (discovery)
>>  > > Andrew Hartnett (dev_warlord)
>>  > > Dave Hirko (dbhirko)
>>  > > Paul Kehrer (reaperhulk)
>>  > > Brad Kolarov (bjkolly)
>>  > > Kiran Komaravolu (UKNOWN)
>>  > > Larry McCay (lmccay)
>>  > > P. Taylor Goetz (ptgoetz)
>>  > > Ryan Merriman (rmerriman)
>>  > > Michael Perez (mperez)
>>  > > Charles Porter (cporter)
>>  > > Phillip Rhodes (prhodes)
>>  > > Sean Schulte (sirsean)
>>  > > James Sirota (jsirota)
>>  > > Casey Stella (cstella)
>>  > > Bryan Taylor (UKNOWN)
>>  > > Ray Urciuoli(UKNOWN)
>>  > > Vinod Kumar Vavilapalli

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

2017-03-22 Thread cestella 
Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
# Testing Plan
## Preliminaries

* Please perform the following tests on the `full-dev` vagrant environment.
* Set an environment variable to indicate `METRON_HOME`:
`export METRON_HOME=/usr/metron/0.3.1` 


## Ensure Data Flows from the Indices
Ensure that with a basic full-dev we get data into the elasticsearch
indices and into HDFS.

## (Optional) Free Up Space on the virtual machine

First, let's free up some headroom on the virtual machine.  If you are 
running this on a
multinode cluster, you would not have to do this.
* Stop and disable Metron in Ambari
* Kill monit via `service monit stop`
* Kill tcpreplay via `for i in $(ps -ef | grep tcpreplay | awk '{print 
$2}');do kill -9 $i;done`
* Kill yaf via `for i in $(ps -ef | grep yaf | awk '{print $2}');do kill -9 
$i;done`
* Kill bro via `for i in $(ps -ef | grep bro | awk '{print $2}');do kill -9 
$i;done`

## Test the PCAP topology

A new kafka spout necessitates testing pcap.
### Install and start pycapa 
```
# set env vars
export PYCAPA_HOME=/opt/pycapa
export PYTHON27_HOME=/opt/rh/python27/root

# Install these packages via yum (RHEL, CentOS)
yum -y install epel-release centos-release-scl 
yum -y install "@Development tools" python27 python27-scldevel 
python27-python-virtualenv libpcap-devel libselinux-python

# Setup directories
mkdir $PYCAPA_HOME && chmod 755 $PYCAPA_HOME

# Create virtualenv
export LD_LIBRARY_PATH="/opt/rh/python27/root/usr/lib64"
${PYTHON27_HOME}/usr/bin/virtualenv pycapa-venv

# Copy pycapa
# copy incubator-metron/metron-sensors/pycapa from the Metron source tree 
into $PYCAPA_HOME on the node you would like to install pycapa on.

# Build it
cd ${PYCAPA_HOME}/pycapa
# activate the virtualenv
source ${PYCAPA_HOME}/pycapa-venv/bin/activate
pip install -r requirements.txt
python setup.py install

# Run it
cd ${PYCAPA_HOME}/pycapa-venv/bin
pycapa --producer --topic pcap -i eth1 -k node1:6667
```
### Ensure pycapa can write to HDFS
* Ensure that `/apps/metron/pcap` exists and can be written to by the
  storm user.  If not, then:
```
sudo su - hdfs
hadoop fs -mkdir -p /apps/metron/pcap
hadoop fs -chown metron:hadoop /apps/metron/pcap
hadoop fs -chmod 775 /apps/metron/pcap
``` 
* Start the pcap topology via `$METRON_HOME/bin/start_pcap_topology.sh`
* Start the pycapa packet capture producer on eth1 via `/usr/bin/pycapa 
--producer --topic pcap -i eth1 -k node1:6667`
* Watch the topology in the Storm UI and kill the packet capture utility 
from before, when the number of packets ingested is over 3k.  Ensure that at at 
least 3 files exist on HDFS by running `hadoop fs -ls /apps/metron/pcap`
* Choose a file (denoted by $FILE) and dump a few of the contents using the 
pcap_inspector utility via `$METRON_HOME/bin/pcap_inspector.sh -i $FILE -n 5`
* Choose one of the lines and note the protocol.
  * Note that when you run the commands below, the resulting file will be 
placed in the execution directory where you kicked off the job from.
* Run a Stellar query filter query by executing a command similar to the 
following, with the values noted above (match your start_time format to the 
date format provided - default is to use millis since epoch):
```
$METRON_HOME/bin/pcap_query.sh query -st "20160617" -df "MMdd" -query 
"protocol == 6" -rpf 500
```
* Verify the MR job finishes successfully. Upon completion, you should see 
multiple files named with relatively current datestamps in your current 
directory, e.g. pcap-data-20160617160549737+.pcap
* Copy the files to your local machine and verify you can them it in 
Wireshark. I chose a middle file and the last file. The middle file should have 
500 records (per the records_per_file option), and the last one will likely 
have a number of records <= 500.

## Test the Profiler

### Setup
* Ensure that Metron is stopped and put in maintenance mode in Ambari
* Create the profiler hbase table
`echo "create 'profiler', 'P'" | hbase shell`

* Open `~/rand_gen.py` and paste the following:
```
#!/usr/bin/python
import random
import sys
import time
def main():
  mu = float(sys.argv[1])
  sigma = float(sys.argv[2])
  freq_s = int(sys.argv[3])
  while True:
out = '{ "value" : ' + str(random.gauss(mu, sigma)) + ' }'
print out
sys.stdout.flush()
time.sleep(freq_s)

if __name__ == '__main__':
  main()
```
This will generate random JSON maps with a numeric field called `value`

* Set the profiler to use 1 minute tick durations:
  * Edit

Re: [VOTE] Final Board Resolution Draft V2

2017-03-22 Thread David Lyle 
+1 binding

On Wed, Mar 22, 2017 at 10:49 AM, Kyle Richardson  wrote:

> +1 (binding)
>
> On Mon, Mar 20, 2017 at 3:05 AM, James Sirota  wrote:
>
> >
> > - Removed affiliations
> > - Added apache IDs where possible
> > - Removed committers and only left PPMC members
> >
> > Hope this version holds up.  Please vote +1, -1, or 0 for neutral.  The
> > vote will be open for 72 hours
> >
> >
> > The incubating Apache Metron community believes it is time to graduate to
> > TLP.
> >
> > Apache Metron entered incubation in December of 2015. Since then, we've
> > overcome technical challenges to remove Category X dependencies, and
> made 3
> > releases. Our most recent release contains binary convenience artifacts.
> We
> > are a very helpful and engaged community, ready to answer all questions
> and
> > feedback directed to us via the user list. Through our time in incubation
> > we've added a number of committers and promoted some of them to PPMC
> > membership. We are actively pursuing others. While we do still have
> issues
> > to address raised by means of the maturity model, all projects are
> ongoing
> > processes, and we believe we no longer need the incubator to continue
> > addressing these issues.
> >
> > To inform the discussion, here is some basic project information:
> >
> > Project status:
> >   http://incubator.apache.org/projects/metron.html
> >
> > Project website:
> >   https://metron.incubator.apache.org/
> >
> > Project documentation:
> >https://cwiki.apache.org/confluence/display/METRON/Documentation
> >
> > Maturity assessment:
> >https://cwiki.apache.org/confluence/display/METRON/
> > Apache+Project+Maturity+Model
> >
> > DRAFT of the board resolution is at the bottom of this email
> >
> > Proposed PMC size: 25 members
> >
> > Total number of committers: 6 members
> >
> >
> > 516 commits on develop
> > 34 contributors across all branches
> >
> > dev list averaged ~650 msgs/month for the last 3 months
> >
> >
> > Resolution:
> >
> > Establish the Apache Metron Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best
> > interests of the Foundation and consistent with the
> > Foundation's purpose to establish a Project Management
> > Committee charged with the creation and maintenance of
> > open-source software, for distribution at no charge to the
> > public, related to a security analytics platform for big data use cases.
> >
> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > Committee (PMC), to be known as the "Apache Metron Project",
> > be and hereby is established pursuant to Bylaws of the
> > Foundation; and be it further
> >
> > RESOLVED, that the Apache Metron Project be and hereby is
> > responsible for the creation and maintenance of software
> > related to:
> > (a) A mechanism to capture, store, and normalize any type of security
> > telemetry at extremely high rates.
> > (b) Real time processing and application of enrichments
> > (c) Efficient information storage
> > (d) An interface that gives a security investigator a centralized view of
> > data and alerts passed through the system.
> >
> > RESOLVED, that the office of "Vice President, Apache Metron" be
> > and hereby is created, the person holding such office to
> > serve at the direction of the Board of Directors as the chair
> > of the Apache Metron Project, and to have primary responsibility
> > for management of the projects within the scope of
> > responsibility of the Apache Metron Project; and be it further
> >
> > RESOLVED, that the persons listed immediately below be and
> > hereby are appointed to serve as the initial members of the
> > Apache Metron Project:
> >
> >
> > PPMC:
> > Mark Bittmann (mbittmann)
> > Sheetal Dolas (sheetal_dolas)
> > Debo Dutta (ddutta)
> > Discovery Gerdes (discovery)
> > Andrew Hartnett (dev_warlord)
> > Dave Hirko (dbhirko)
> > Paul Kehrer (reaperhulk)
> > Brad Kolarov (bjkolly)
> > Kiran Komaravolu (UKNOWN)
> > Larry McCay (lmccay)
> > P. Taylor Goetz (ptgoetz)
> > Ryan Merriman (rmerriman)
> > Michael Perez (mperez)
> > Charles Porter (cporter)
> > Phillip Rhodes (prhodes)
> > Sean Schulte (sirsean)
> > James Sirota (jsirota)
> > Casey Stella (cstella)
> > Bryan Taylor (UKNOWN)
> > Ray Urciuoli(UKNOWN)
> > Vinod Kumar Vavilapalli (vinodkv)
> > George Vetticaden (gvetticaden)
> > Oskar Zabik (smogg)
> > David Lyle (lyle)
> > Nick Allen (nickallen)
> >
> >
> >
> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Casey Stella
> > be appointed to the office of Vice President, Apache Metron, to
> > serve in accordance with and subject to the direction of the
> > Board of Directors and the Bylaws of the Foundation until
> > death, resignation, retirement, removal or disqualification,
> > or until a successor is appointed; and be it further
> >
> > RESOLVED, that the initial Apache Metron PMC be and hereby is
> > tasked with the creation of a set of bylaws intended to
> > encourage open development and

Re: [VOTE] Final Board Resolution Draft V2

2017-03-22 Thread Kyle Richardson 
+1 (binding)

On Mon, Mar 20, 2017 at 3:05 AM, James Sirota  wrote:

>
> - Removed affiliations
> - Added apache IDs where possible
> - Removed committers and only left PPMC members
>
> Hope this version holds up.  Please vote +1, -1, or 0 for neutral.  The
> vote will be open for 72 hours
>
>
> The incubating Apache Metron community believes it is time to graduate to
> TLP.
>
> Apache Metron entered incubation in December of 2015. Since then, we've
> overcome technical challenges to remove Category X dependencies, and made 3
> releases. Our most recent release contains binary convenience artifacts. We
> are a very helpful and engaged community, ready to answer all questions and
> feedback directed to us via the user list. Through our time in incubation
> we've added a number of committers and promoted some of them to PPMC
> membership. We are actively pursuing others. While we do still have issues
> to address raised by means of the maturity model, all projects are ongoing
> processes, and we believe we no longer need the incubator to continue
> addressing these issues.
>
> To inform the discussion, here is some basic project information:
>
> Project status:
>   http://incubator.apache.org/projects/metron.html
>
> Project website:
>   https://metron.incubator.apache.org/
>
> Project documentation:
>https://cwiki.apache.org/confluence/display/METRON/Documentation
>
> Maturity assessment:
>https://cwiki.apache.org/confluence/display/METRON/
> Apache+Project+Maturity+Model
>
> DRAFT of the board resolution is at the bottom of this email
>
> Proposed PMC size: 25 members
>
> Total number of committers: 6 members
>
>
> 516 commits on develop
> 34 contributors across all branches
>
> dev list averaged ~650 msgs/month for the last 3 months
>
>
> Resolution:
>
> Establish the Apache Metron Project
>
> WHEREAS, the Board of Directors deems it to be in the best
> interests of the Foundation and consistent with the
> Foundation's purpose to establish a Project Management
> Committee charged with the creation and maintenance of
> open-source software, for distribution at no charge to the
> public, related to a security analytics platform for big data use cases.
>
> NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> Committee (PMC), to be known as the "Apache Metron Project",
> be and hereby is established pursuant to Bylaws of the
> Foundation; and be it further
>
> RESOLVED, that the Apache Metron Project be and hereby is
> responsible for the creation and maintenance of software
> related to:
> (a) A mechanism to capture, store, and normalize any type of security
> telemetry at extremely high rates.
> (b) Real time processing and application of enrichments
> (c) Efficient information storage
> (d) An interface that gives a security investigator a centralized view of
> data and alerts passed through the system.
>
> RESOLVED, that the office of "Vice President, Apache Metron" be
> and hereby is created, the person holding such office to
> serve at the direction of the Board of Directors as the chair
> of the Apache Metron Project, and to have primary responsibility
> for management of the projects within the scope of
> responsibility of the Apache Metron Project; and be it further
>
> RESOLVED, that the persons listed immediately below be and
> hereby are appointed to serve as the initial members of the
> Apache Metron Project:
>
>
> PPMC:
> Mark Bittmann (mbittmann)
> Sheetal Dolas (sheetal_dolas)
> Debo Dutta (ddutta)
> Discovery Gerdes (discovery)
> Andrew Hartnett (dev_warlord)
> Dave Hirko (dbhirko)
> Paul Kehrer (reaperhulk)
> Brad Kolarov (bjkolly)
> Kiran Komaravolu (UKNOWN)
> Larry McCay (lmccay)
> P. Taylor Goetz (ptgoetz)
> Ryan Merriman (rmerriman)
> Michael Perez (mperez)
> Charles Porter (cporter)
> Phillip Rhodes (prhodes)
> Sean Schulte (sirsean)
> James Sirota (jsirota)
> Casey Stella (cstella)
> Bryan Taylor (UKNOWN)
> Ray Urciuoli(UKNOWN)
> Vinod Kumar Vavilapalli (vinodkv)
> George Vetticaden (gvetticaden)
> Oskar Zabik (smogg)
> David Lyle (lyle)
> Nick Allen (nickallen)
>
>
>
> NOW, THEREFORE, BE IT FURTHER RESOLVED, that Casey Stella
> be appointed to the office of Vice President, Apache Metron, to
> serve in accordance with and subject to the direction of the
> Board of Directors and the Bylaws of the Foundation until
> death, resignation, retirement, removal or disqualification,
> or until a successor is appointed; and be it further
>
> RESOLVED, that the initial Apache Metron PMC be and hereby is
> tasked with the creation of a set of bylaws intended to
> encourage open development and increased participation in the
> Apache Metron Project; and be it further
>
> RESOLVED, that the Apache Metron Project be and hereby
> is tasked with the migration and rationalization of the Apache
> Incubator Metron podling; and be it further
>
> RESOLVED, that all responsibilities pertaining to the Apache
> Incubator Metron podling encumbered upon the Apache Incubator
>

Re: [DISCUSS] Stepping down as release manager

2017-03-22 Thread Kyle Richardson 
+1 for Matt

On Wed, Mar 22, 2017 at 10:23 AM, Justin Leet  wrote:

> Right now it's just support, not a vote.  I assume, based on our past
> practices, that there will be a separate [VOTE] thread.
>
> Justin
>
> On Wed, Mar 22, 2017 at 10:06 AM, Otto Fowler 
> wrote:
>
> > +1 but is this explicitly an official vote?
> >
> >
> > On March 21, 2017 at 13:51:16, Justin Leet (justinjl...@gmail.com)
> wrote:
> >
> > +1 for Matt
> >
> > On Tue, Mar 21, 2017 at 12:21 PM, zeo...@gmail.com 
> > wrote:
> >
> > > +1 for mattf
> > >
> > > On Tue, Mar 21, 2017 at 11:04 AM Ryan Merriman 
> > > wrote:
> > >
> > > > +1 for Matt
> > > >
> > > > On Tue, Mar 21, 2017 at 9:44 AM, Matt Foley 
> wrote:
> > > >
> > > > > Casey, you’ve been a great release manager. I know how much detail
> > > > effort
> > > > > goes into this role.
> > > > >
> > > > > I am willing to serve as RM for the next while, if the community
> > would
> > > > > like. I was the RM for Hadoop for about a year, and in fact was RM
> > for
> > > > its
> > > > > 1.0 release. Granted that was a while ago, but overall process
> > doesn’t
> > > > seem
> > > > > to have changed much :-)
> > > > >
> > > > > Cheers,
> > > > > --Matt
> > > > >
> > > > > On 3/21/17, 7:32 AM, "Casey Stella"  wrote:
> > > > >
> > > > > Right, Billie is exactly right. Working with the community to
> > > > > constructing
> > > > > releases that conform to apache standards and policies is the main
> > > > > duty.
> > > > > This will (hopefully) be our first set of releases outside of the
> > > > > incubator, so if I'm allowed to be biased, I'm hoping that someone
> > > > with
> > > > > previous release management experience in other projects will
> > > > > volunteer.
> > > > > We're leaving the nest a bit and having an experienced hand at the
> > > > > tiller
> > > > > would be advantageous.
> > > > >
> > > > >
> > > > > On Tue, Mar 21, 2017 at 10:21 AM, Billie Rinaldi <
> > > bil...@apache.org>
> > > > > wrote:
> > > > >
> > > > > > See http://www.apache.org/dev/release-publishing#release_manager
> > > > and
> > > > > > http://www.apache.org/legal/release-policy.html for information
> > > on
> > > > > the
> > > > > > tasks that a release manager performs.
> > > > > >
> > > > > > On Tue, Mar 21, 2017 at 7:10 AM, Khurram Ahmed <
> > > > > khurramah...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Casey it would be helpful if you could outline the
> > > > > responsibilities of a
> > > > > > > release manager for the Metron project.
> > > > > > >
> > > > > > > On Mar 21, 2017 6:57 PM, "Casey Stella" 
> > > > > wrote:
> > > > > > >
> > > > > > > > I've been extremely honored to spend the last few months as
> > > the
> > > > > Metron
> > > > > > > > Release Manager. That being said, my watch is ended and it's
> > > > > time for
> > > > > > > > another release manager to step into my place.
> > > > > > > >
> > > > > > > > Who would like to volunteer to be release manager for the
> > > next
> > > > > release
> > > > > > of
> > > > > > > > Metron?
> > > > > > > >
> > > > > > > > Best,
> > > > > > > >
> > > > > > > > Casey
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > > >
> > > >
> > > --
> > >
> > > Jon
> > >
> >
> >
>

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread cestella 
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107434843
  
--- Diff: 
metron-analytics/metron-profiler/src/main/config/profiler.properties ---
@@ -39,4 +39,4 @@ profiler.hbase.flush.interval.seconds=30
 
 kafka.zk=node1:2181
 kafka.broker=node1:6667
-kafka.start=WHERE_I_LEFT_OFF
+kafka.start=UNCOMMITTED_EARLIEST
--- End diff --

This needs documentation.  We had our own abstractions around the kafka 
offsets which we no longer need because the new spout has better ones.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread cestella 
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107428876
  
--- Diff: 
metron-platform/metron-integration-test/src/main/java/org/apache/metron/integration/components/FluxTopologyComponent.java
 ---
@@ -133,7 +138,25 @@ public void start() throws UnableToStartException {
   @Override
   public void stop() {
 if (stormCluster != null) {
-  stormCluster.shutdown();
+  try {
--- End diff --

Note here, we moved to storm 1.0.3 in order to have a single API that would 
work between storm 1.0.1 in the HDP profile as well as the default profile.  
Storm 1.0.3 writes out worker artifact logs and they need to be cleaned up.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

2017-03-22 Thread cestella 
Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
I owe:
* Better Javadocs around some of the pcap infrastructure
* An acceptance test plan


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread cestella 
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107430400
  
--- Diff: pom.xml ---
@@ -67,20 +67,44 @@
 
 1.0.1
 1.0.1
-0.10.0.1
+0.10.0
 2.7.1
 1.1.1
-1.8.0
 1.5.2
 
+1.8.0
 4.5
 3.7
 2.7.1
 3.3
-${base_storm_version}
+1.0.3
+
+
1.0.1.2.5.0.0-1245
--- End diff --

The comment above this line is worth pointing out.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread cestella 
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107429992
  
--- Diff: 
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/Deserializers.java
 ---
@@ -16,31 +16,35 @@
  * limitations under the License.
  */
 
-package org.apache.metron.spout.pcap.scheme;
+package org.apache.metron.spout.pcap.deserializer;
 
-import org.apache.storm.spout.MultiScheme;
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
 import org.apache.metron.common.utils.timestamp.TimestampConverter;
-import org.apache.storm.kafka.KeyValueSchemeAsMultiScheme;
 
-public enum TimestampScheme {
-   FROM_KEY( converter -> new KeyValueSchemeAsMultiScheme(new 
FromKeyScheme().withTimestampConverter(converter)))
-  ,FROM_PACKET(converter -> new 
FromPacketScheme().withTimestampConverter(converter));
+import java.util.function.Function;
+
+public enum Deserializers {
--- End diff --

It's worth noting here that the new spout does not use multischemes.  We 
are doing a fair bit of byte munging on data coming into the pcap topology in 
those multischemes, but the abstraction was clunky.  This makes things a bit 
clearer.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

2017-03-22 Thread cestella 
Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107428522
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/dsl/functions/resolver/ClasspathFunctionResolver.java
 ---
@@ -235,8 +235,16 @@ public void initialize(Context context) {
 }
 
 FilterBuilder filterBuilder = new FilterBuilder();
-excludes.forEach(excl -> filterBuilder.exclude(excl));
-includes.forEach(incl -> filterBuilder.include(incl));
+excludes.forEach(excl -> {
--- End diff --

This is an intermittent test bug that I noticed and fixed in order to get 
the tests to run consistently.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Stepping down as release manager

2017-03-22 Thread Justin Leet 
Right now it's just support, not a vote.  I assume, based on our past
practices, that there will be a separate [VOTE] thread.

Justin

On Wed, Mar 22, 2017 at 10:06 AM, Otto Fowler 
wrote:

> +1 but is this explicitly an official vote?
>
>
> On March 21, 2017 at 13:51:16, Justin Leet (justinjl...@gmail.com) wrote:
>
> +1 for Matt
>
> On Tue, Mar 21, 2017 at 12:21 PM, zeo...@gmail.com 
> wrote:
>
> > +1 for mattf
> >
> > On Tue, Mar 21, 2017 at 11:04 AM Ryan Merriman 
> > wrote:
> >
> > > +1 for Matt
> > >
> > > On Tue, Mar 21, 2017 at 9:44 AM, Matt Foley  wrote:
> > >
> > > > Casey, you’ve been a great release manager. I know how much detail
> > > effort
> > > > goes into this role.
> > > >
> > > > I am willing to serve as RM for the next while, if the community
> would
> > > > like. I was the RM for Hadoop for about a year, and in fact was RM
> for
> > > its
> > > > 1.0 release. Granted that was a while ago, but overall process
> doesn’t
> > > seem
> > > > to have changed much :-)
> > > >
> > > > Cheers,
> > > > --Matt
> > > >
> > > > On 3/21/17, 7:32 AM, "Casey Stella"  wrote:
> > > >
> > > > Right, Billie is exactly right. Working with the community to
> > > > constructing
> > > > releases that conform to apache standards and policies is the main
> > > > duty.
> > > > This will (hopefully) be our first set of releases outside of the
> > > > incubator, so if I'm allowed to be biased, I'm hoping that someone
> > > with
> > > > previous release management experience in other projects will
> > > > volunteer.
> > > > We're leaving the nest a bit and having an experienced hand at the
> > > > tiller
> > > > would be advantageous.
> > > >
> > > >
> > > > On Tue, Mar 21, 2017 at 10:21 AM, Billie Rinaldi <
> > bil...@apache.org>
> > > > wrote:
> > > >
> > > > > See http://www.apache.org/dev/release-publishing#release_manager
> > > and
> > > > > http://www.apache.org/legal/release-policy.html for information
> > on
> > > > the
> > > > > tasks that a release manager performs.
> > > > >
> > > > > On Tue, Mar 21, 2017 at 7:10 AM, Khurram Ahmed <
> > > > khurramah...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Casey it would be helpful if you could outline the
> > > > responsibilities of a
> > > > > > release manager for the Metron project.
> > > > > >
> > > > > > On Mar 21, 2017 6:57 PM, "Casey Stella" 
> > > > wrote:
> > > > > >
> > > > > > > I've been extremely honored to spend the last few months as
> > the
> > > > Metron
> > > > > > > Release Manager. That being said, my watch is ended and it's
> > > > time for
> > > > > > > another release manager to step into my place.
> > > > > > >
> > > > > > > Who would like to volunteer to be release manager for the
> > next
> > > > release
> > > > > of
> > > > > > > Metron?
> > > > > > >
> > > > > > > Best,
> > > > > > >
> > > > > > > Casey
> > > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > >
> > >
> > --
> >
> > Jon
> >
>
>

Re: [DISCUSS] Stepping down as release manager

2017-03-22 Thread Otto Fowler 
+1 but is this explicitly an official vote?


On March 21, 2017 at 13:51:16, Justin Leet (justinjl...@gmail.com) wrote:

+1 for Matt

On Tue, Mar 21, 2017 at 12:21 PM, zeo...@gmail.com 
wrote:

> +1 for mattf
>
> On Tue, Mar 21, 2017 at 11:04 AM Ryan Merriman 
> wrote:
>
> > +1 for Matt
> >
> > On Tue, Mar 21, 2017 at 9:44 AM, Matt Foley  wrote:
> >
> > > Casey, you’ve been a great release manager. I know how much detail
> > effort
> > > goes into this role.
> > >
> > > I am willing to serve as RM for the next while, if the community
would
> > > like. I was the RM for Hadoop for about a year, and in fact was RM
for
> > its
> > > 1.0 release. Granted that was a while ago, but overall process
doesn’t
> > seem
> > > to have changed much :-)
> > >
> > > Cheers,
> > > --Matt
> > >
> > > On 3/21/17, 7:32 AM, "Casey Stella"  wrote:
> > >
> > > Right, Billie is exactly right. Working with the community to
> > > constructing
> > > releases that conform to apache standards and policies is the main
> > > duty.
> > > This will (hopefully) be our first set of releases outside of the
> > > incubator, so if I'm allowed to be biased, I'm hoping that someone
> > with
> > > previous release management experience in other projects will
> > > volunteer.
> > > We're leaving the nest a bit and having an experienced hand at the
> > > tiller
> > > would be advantageous.
> > >
> > >
> > > On Tue, Mar 21, 2017 at 10:21 AM, Billie Rinaldi <
> bil...@apache.org>
> > > wrote:
> > >
> > > > See http://www.apache.org/dev/release-publishing#release_manager
> > and
> > > > http://www.apache.org/legal/release-policy.html for information
> on
> > > the
> > > > tasks that a release manager performs.
> > > >
> > > > On Tue, Mar 21, 2017 at 7:10 AM, Khurram Ahmed <
> > > khurramah...@gmail.com>
> > > > wrote:
> > > >
> > > > > Casey it would be helpful if you could outline the
> > > responsibilities of a
> > > > > release manager for the Metron project.
> > > > >
> > > > > On Mar 21, 2017 6:57 PM, "Casey Stella" 
> > > wrote:
> > > > >
> > > > > > I've been extremely honored to spend the last few months as
> the
> > > Metron
> > > > > > Release Manager. That being said, my watch is ended and it's
> > > time for
> > > > > > another release manager to step into my place.
> > > > > >
> > > > > > Who would like to volunteer to be release manager for the
> next
> > > release
> > > > of
> > > > > > Metron?
> > > > > >
> > > > > > Best,
> > > > > >
> > > > > > Casey
> > > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > >
> > >
> >
> --
>
> Jon
>