[GitHub] incubator-metron pull request #481: METRON-322 Global Batching and Flushing

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/481#discussion_r107781590
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
 ---
@@ -92,22 +177,51 @@ public void prepare(Map stormConf, TopologyContext 
context, OutputCollector coll
   configurationTransformation = x -> x;
 }
 try {
-  bulkMessageWriter.init(stormConf
-, configurationTransformation.apply(new 
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
-);
+  WriterConfiguration writerconf = configurationTransformation.apply(
+  new IndexingWriterConfiguration(bulkMessageWriter.getName(), 
getConfigurations()));
+  if (defaultBatchTimeout == 0) {
+//This means getComponentConfiguration was never called to 
initialize defaultBatchTimeout,
+//probably because we are in a unit test scenario.  So calculate 
it here.
+BatchTimeoutHelper timeoutHelper = new 
BatchTimeoutHelper(writerconf::getAllConfiguredTimeouts, batchTimeoutDivisor);
+defaultBatchTimeout = timeoutHelper.getDefaultBatchTimeout();
+  }
+  writerComponent.setDefaultBatchTimeout(defaultBatchTimeout);
+  bulkMessageWriter.init(stormConf, writerconf);
 } catch (Exception e) {
   throw new RuntimeException(e);
 }
   }
 
+  /**
+   * Used only for unit testing.
+   */
+  public void prepare(Map stormConf, TopologyContext context, 
OutputCollector collector, Clock clock) {
+prepare(stormConf, context, collector);
+writerComponent.withClock(clock);
+  }
+
   @SuppressWarnings("unchecked")
   @Override
   public void execute(Tuple tuple) {
-JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
-String sensorType = MessageUtils.getSensorType(message);
 try
 {
-  WriterConfiguration writerConfiguration = 
configurationTransformation.apply(new 
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()));
+  if (isTick(tuple)) {
+if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+  //WriterToBulkWriter doesn't allow batching, so no need to flush 
on Tick.
+  LOG.debug("Flushing message queues older than their 
batchTimeouts");
+  writerComponent.flushTimeouts(bulkMessageWriter, 
configurationTransformation.apply(
+  new 
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+  , messageGetStrategy);
+}
+collector.ack(tuple);
--- End diff --

Good idea, will change to separate "try" wrappers around the tick tuple vs 
message tuple processing.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #481: METRON-322 Global Batching and Flushing

[mattf-horton]

Github user mattf-horton commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/481#discussion_r107779736
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/IndexingConfigurations.java
 ---
@@ -73,7 +74,30 @@ public boolean isDefault(String sensorName, String 
writerName) {
   }
 
   public int getBatchSize(String sensorName, String writerName ) {
- return getBatchSize(getSensorIndexingConfig(sensorName, writerName));
+return getBatchSize(getSensorIndexingConfig(sensorName, writerName));
+  }
+
+  public int getBatchTimeout(String sensorName, String writerName ) {
+return getBatchTimeout(getSensorIndexingConfig(sensorName, 
writerName));
+  }
+
+  public List getAllConfiguredTimeouts(String writerName) {
+//The configuration infrastructure was not designed to enumerate 
sensors, so we synthesize.
+//The use of SENSOR_MARKER_STRING assumes the config key for a sensor 
name consists of a
+//prefix string (dependent on the ConfigurationType) followed by the 
sensor name.
+//This then allows us to determine the keyPrefixString, and thence 
which of all the
+//configurations.keySet are actually sensor keys.  Then we look under 
those for timeouts.
+String markerKeyString = getKey(SENSOR_MARKER_STRING);
--- End diff --

Yup, this is unnecessarily complex.  I'm going to eliminate 
SENSOR_MARKER_STRING and replace:
```
String markerKeyString = getKey(SENSOR_MARKER_STRING);
int prefixStringLength = markerKeyString.indexOf(SENSOR_MARKER_STRING);
String keyPrefixString = markerKeyString.substring(0, 
prefixStringLength);
```
with just:
```
String keyPrefixString = getKey("");
```



---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #481: METRON-322 Global Batching and Flushing

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/481#discussion_r107765415
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/bolt/BulkMessageWriterBolt.java
 ---
@@ -92,22 +177,51 @@ public void prepare(Map stormConf, TopologyContext 
context, OutputCollector coll
   configurationTransformation = x -> x;
 }
 try {
-  bulkMessageWriter.init(stormConf
-, configurationTransformation.apply(new 
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
-);
+  WriterConfiguration writerconf = configurationTransformation.apply(
+  new IndexingWriterConfiguration(bulkMessageWriter.getName(), 
getConfigurations()));
+  if (defaultBatchTimeout == 0) {
+//This means getComponentConfiguration was never called to 
initialize defaultBatchTimeout,
+//probably because we are in a unit test scenario.  So calculate 
it here.
+BatchTimeoutHelper timeoutHelper = new 
BatchTimeoutHelper(writerconf::getAllConfiguredTimeouts, batchTimeoutDivisor);
+defaultBatchTimeout = timeoutHelper.getDefaultBatchTimeout();
+  }
+  writerComponent.setDefaultBatchTimeout(defaultBatchTimeout);
+  bulkMessageWriter.init(stormConf, writerconf);
 } catch (Exception e) {
   throw new RuntimeException(e);
 }
   }
 
+  /**
+   * Used only for unit testing.
+   */
+  public void prepare(Map stormConf, TopologyContext context, 
OutputCollector collector, Clock clock) {
+prepare(stormConf, context, collector);
+writerComponent.withClock(clock);
+  }
+
   @SuppressWarnings("unchecked")
   @Override
   public void execute(Tuple tuple) {
-JSONObject message = (JSONObject) messageGetStrategy.get(tuple);
-String sensorType = MessageUtils.getSensorType(message);
 try
 {
-  WriterConfiguration writerConfiguration = 
configurationTransformation.apply(new 
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()));
+  if (isTick(tuple)) {
+if (!(bulkMessageWriter instanceof WriterToBulkWriter)) {
+  //WriterToBulkWriter doesn't allow batching, so no need to flush 
on Tick.
+  LOG.debug("Flushing message queues older than their 
batchTimeouts");
+  writerComponent.flushTimeouts(bulkMessageWriter, 
configurationTransformation.apply(
+  new 
IndexingWriterConfiguration(bulkMessageWriter.getName(), getConfigurations()))
+  , messageGetStrategy);
+}
+collector.ack(tuple);
--- End diff --

Should this be in a finally?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #481: METRON-322 Global Batching and Flushing

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/481#discussion_r107766918
  
--- Diff: 
metron-platform/metron-common/src/main/java/org/apache/metron/common/configuration/IndexingConfigurations.java
 ---
@@ -73,7 +74,30 @@ public boolean isDefault(String sensorName, String 
writerName) {
   }
 
   public int getBatchSize(String sensorName, String writerName ) {
- return getBatchSize(getSensorIndexingConfig(sensorName, writerName));
+return getBatchSize(getSensorIndexingConfig(sensorName, writerName));
+  }
+
+  public int getBatchTimeout(String sensorName, String writerName ) {
+return getBatchTimeout(getSensorIndexingConfig(sensorName, 
writerName));
+  }
+
+  public List getAllConfiguredTimeouts(String writerName) {
+//The configuration infrastructure was not designed to enumerate 
sensors, so we synthesize.
+//The use of SENSOR_MARKER_STRING assumes the config key for a sensor 
name consists of a
+//prefix string (dependent on the ConfigurationType) followed by the 
sensor name.
+//This then allows us to determine the keyPrefixString, and thence 
which of all the
+//configurations.keySet are actually sensor keys.  Then we look under 
those for timeouts.
+String markerKeyString = getKey(SENSOR_MARKER_STRING);
--- End diff --

Could you perhaps go a bit into why you're using this 
`SENSOR_MARKER_STRING` again?  I read the comment and I am still not entirely 
sure.  I think, at a high level, what you're trying to do is, for a given 
writer (e.g. `hdfs`) you want a list of all of the configured timeouts across 
all sensors, right?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
I just finished running some parser tests in a Kerberized environment to 
confirm that this will address both kerberized and non-kerberized 
configurations. Here is what I had to do to get this running. Note, I'm using 
Ambari to configure everything for Hadoop, but not the Metron components. There 
is separate work that will need to be done for Metron in the MPack for Ambari.

1. Spin up full dev as normal (ensure that the profile is correctly set to 
-P HDP-2.5.0.0 in metron-deployment/roles/metron-builder/tasks/main.yml). As of 
this writing, the current full-dev ansible install will do a complete build as 
part of the deployment.
2. service stop monit and kill all the topologies. You can go ahead and 
kill the other sensors per the commands from @cestella  as well (bro, pycapa, 
etc.). We'll spin up our own simple test topology and populate the kafka topic 
manually to test this out.
3. Setup symlinks
```
ln -s /usr/metron/0.3.1/lib/metron-*-uber.jar 
/usr/hdp/2.5.3.0-37/storm/extlib-daemon/
ln -s /usr/hdp/current/hadoop-client/conf/core-site.xml /etc/storm/conf
ln -s /usr/hdp/current/hbase-client/conf/hbase-site.xml /etc/storm/conf
ln -s 
/usr/hdp/2.5.3.0-37/storm/contrib/storm-hbase/storm-hbase-1.0.1.2.5.3.0-37.jar 
/usr/hdp/2.5.3.0-37/storm/extlib
ln -s 
/usr/hdp/2.5.3.0-37/storm/contrib/storm-hdfs/storm-hdfs-1.0.1.2.5.3.0-37.jar 
/usr/hdp/2.5.3.0-37/storm/extlib
```
4. Check that the jce security is setup in 
/usr/jdk64/jdk1.8.0_77/jre/lib/security/. If not, then you'll want to run 
through the following steps:
4.1 Download the jce policy from: 
http://www.oracle.com/technetwork/java/javase/downloads/jce8-download-2133166.html
 and copy it to your vm.
4.2 unzip -o -j -q /path-to-the-copied-zip/jce_policy-8.zip -d 
/usr/jdk64/jdk1.8.0_77/jre/lib/security/
5. Configure MIT Kerberos
```
yum -y install krb5-server krb5-libs krb5-workstation
sed -i 's/kerberos.example.com/node1/g' /etc/krb5.conf
cp /etc/krb5.conf /var/lib/ambari-server/resources/scripts
kdb5_util create -s
/etc/rc.d/init.d/krb5kdc start
/etc/rc.d/init.d/kadmin start
chkconfig krb5kdc on
chkconfig kadmin on

```
6. Setup admin and personal user principals. You'll kinit as the "{your 
user name}" user. For my tests, I did "addprinc mike". I just made the 
passwords "password" for the sake of easy testing.
```
kadmin.local -q "addprinc admin/admin"
kadmin.local -q "addprinc {your user name}"
```
7. In Ambari, setup AutoHDFS by adding the following properties to 
custom-storm-site
```

nimbus.autocredential.plugins.classes=['org.apache.storm.hdfs.common.security.AutoHDFS','org.apache.storm.hbase.security.AutoHBase']

nimbus.credential.renewers.classes=['org.apache.storm.hdfs.common.security.AutoHDFS','org.apache.storm.hbase.security.AutoHBase']
hdfs.keytab.file=/etc/security/keytabs/hdfs.headless.keytab
hdfs.kerberos.principal=hdfs-metron_clus...@example.com
hbase.keytab.file=/etc/security/keytabs/hbase.headless.keytab
hbase.kerberos.principal=hbase-metron_clus...@example.com
nimbus.credential.renewers.freq.secs=82800
```
8. Kerberize the cluster via Ambari. The wizard is fairly straightforward, 
but further documentation can be found 
[here](http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.5.3/bk_security/content/_enabling_kerberos_security_in_ambari.html).
 For this exercise, choose existing MIT KDC (this is what we setup and 
installed in previous steps.) Realm is EXAMPLE.COM. The admin principal will 
end up as admin/ad...@example.com when testing the KDC.
9. Let the cluster spin up, but don't worry about starting up Metron via 
Ambari - we're going to run the parsers manually against the rest of the Hadoop 
cluster Kerberized.
10. Kinit and provide the password you chose from earlier.
```
kinit 
```
11. Create a kafka topic for the jsonMap parser.
```
/usr/hdp/current/kafka-broker/bin/kafka-topics.sh --zookeeper node1:2181 
--create --topic jsonMap --replication-factor 1 --partitions 1
```
12. Setup the ACLs for the topic AND the consumer group. The consumer group 
config is new in this API as far as I can tell because we didn't have to do 
this in the "storm-kafka" version of the api. Note that the docs around kafka 
0.10 will suggest using a "--consumer-group" option, but in HDP 2.5.x the 
option needs to be "--group". Also make sure you use your username in 
"--allow-principal User:" instead of "mike". 
```
/usr/hdp/current/kafka-broker/bin/kafka-acls.sh --authorizer 
kafka.security.auth.SimpleAclAuthorizer --authorizer-properties 
zookeeper.connect=node1:2181 --add --allow-principal User:storm-metron_cluster 
--allow-principal User:mike --topic jsonMap
/usr/hdp/current/kafka-broker/bin/kafka-acls.sh 

Re: [VOTE] Final Board Resolution Draft V2

[Nick Allen]

+1 (binding)  I want to make sure my support is on the record, even though
the vote already passed.  Very exciting.

On Wed, Mar 22, 2017 at 3:34 PM, Billie Rinaldi  wrote:

> Here are the remaining Apache IDs:
> Kiran Komaravolu (kirankom)
> Ray Urciuoli (rurcioli)
>
> The list seems to be missing Jim Baker (jimbaker)? See
> http://home.apache.org/phonebook?podling=metron. Also, it looks like Bryan
> Taylor never submitted an ICLA, and has no Apache ID. I don't think he can
> be on the initial PMC list, in that case.
>
> On Mon, Mar 20, 2017 at 12:05 AM, James Sirota  wrote:
>
> >
> > - Removed affiliations
> > - Added apache IDs where possible
> > - Removed committers and only left PPMC members
> >
> > Hope this version holds up.  Please vote +1, -1, or 0 for neutral.  The
> > vote will be open for 72 hours
> >
> >
> > The incubating Apache Metron community believes it is time to graduate to
> > TLP.
> >
> > Apache Metron entered incubation in December of 2015. Since then, we've
> > overcome technical challenges to remove Category X dependencies, and
> made 3
> > releases. Our most recent release contains binary convenience artifacts.
> We
> > are a very helpful and engaged community, ready to answer all questions
> and
> > feedback directed to us via the user list. Through our time in incubation
> > we've added a number of committers and promoted some of them to PPMC
> > membership. We are actively pursuing others. While we do still have
> issues
> > to address raised by means of the maturity model, all projects are
> ongoing
> > processes, and we believe we no longer need the incubator to continue
> > addressing these issues.
> >
> > To inform the discussion, here is some basic project information:
> >
> > Project status:
> >   http://incubator.apache.org/projects/metron.html
> >
> > Project website:
> >   https://metron.incubator.apache.org/
> >
> > Project documentation:
> >https://cwiki.apache.org/confluence/display/METRON/Documentation
> >
> > Maturity assessment:
> >https://cwiki.apache.org/confluence/display/METRON/
> > Apache+Project+Maturity+Model
> >
> > DRAFT of the board resolution is at the bottom of this email
> >
> > Proposed PMC size: 25 members
> >
> > Total number of committers: 6 members
> >
> >
> > 516 commits on develop
> > 34 contributors across all branches
> >
> > dev list averaged ~650 msgs/month for the last 3 months
> >
> >
> > Resolution:
> >
> > Establish the Apache Metron Project
> >
> > WHEREAS, the Board of Directors deems it to be in the best
> > interests of the Foundation and consistent with the
> > Foundation's purpose to establish a Project Management
> > Committee charged with the creation and maintenance of
> > open-source software, for distribution at no charge to the
> > public, related to a security analytics platform for big data use cases.
> >
> > NOW, THEREFORE, BE IT RESOLVED, that a Project Management
> > Committee (PMC), to be known as the "Apache Metron Project",
> > be and hereby is established pursuant to Bylaws of the
> > Foundation; and be it further
> >
> > RESOLVED, that the Apache Metron Project be and hereby is
> > responsible for the creation and maintenance of software
> > related to:
> > (a) A mechanism to capture, store, and normalize any type of security
> > telemetry at extremely high rates.
> > (b) Real time processing and application of enrichments
> > (c) Efficient information storage
> > (d) An interface that gives a security investigator a centralized view of
> > data and alerts passed through the system.
> >
> > RESOLVED, that the office of "Vice President, Apache Metron" be
> > and hereby is created, the person holding such office to
> > serve at the direction of the Board of Directors as the chair
> > of the Apache Metron Project, and to have primary responsibility
> > for management of the projects within the scope of
> > responsibility of the Apache Metron Project; and be it further
> >
> > RESOLVED, that the persons listed immediately below be and
> > hereby are appointed to serve as the initial members of the
> > Apache Metron Project:
> >
> >
> > PPMC:
> > Mark Bittmann (mbittmann)
> > Sheetal Dolas (sheetal_dolas)
> > Debo Dutta (ddutta)
> > Discovery Gerdes (discovery)
> > Andrew Hartnett (dev_warlord)
> > Dave Hirko (dbhirko)
> > Paul Kehrer (reaperhulk)
> > Brad Kolarov (bjkolly)
> > Kiran Komaravolu (UKNOWN)
> > Larry McCay (lmccay)
> > P. Taylor Goetz (ptgoetz)
> > Ryan Merriman (rmerriman)
> > Michael Perez (mperez)
> > Charles Porter (cporter)
> > Phillip Rhodes (prhodes)
> > Sean Schulte (sirsean)
> > James Sirota (jsirota)
> > Casey Stella (cstella)
> > Bryan Taylor (UKNOWN)
> > Ray Urciuoli(UKNOWN)
> > Vinod Kumar Vavilapalli (vinodkv)
> > George Vetticaden (gvetticaden)
> > Oskar Zabik (smogg)
> > David Lyle (lyle)
> > Nick Allen (nickallen)
> >
> >
> >
> > NOW, THEREFORE, BE IT FURTHER RESOLVED, that Casey Stella
> > be appointed to the office of Vice President, Apache 

Re: [MENTORS] initiating the TLP vote

[Casey Stella]

Sounds good, thanks Taylor.  I'll write up the email and start the
discussion on incubator general.

On Thu, Mar 23, 2017 at 1:58 PM, P. Taylor Goetz  wrote:

> Anyone can do this, it doesn’t need to be a mentor or the proposed VP.
> It’s much like a release vote: Forward a copy of the proposed resolution
> and a link to the PPMC graduation VOTE.
>
> So the procedure is:
>
> 1. Start a DISCUSS thread on general@incubator. Once that discussion dies
> down, and if there seems to be positive consensus…
> 2. Start a VOTE thread for the same.
>
> -Taylor
>
>
> > On Mar 22, 2017, at 12:53 PM, James Sirota  wrote:
> >
> > Mentors,
> >
> > Looks like we have everything we need to initiate the incubator board
> vote to leave the incubator.  Can you provide some guidance as to what is
> the best way to approach this?
> >
> > ---
> > Thank you,
> >
> > James Sirota
> > PPMC- Apache Metron (Incubating)
> > jsirota AT apache DOT org
>
>

Re: [MENTORS] initiating the TLP vote

[P. Taylor Goetz]

Anyone can do this, it doesn’t need to be a mentor or the proposed VP. It’s 
much like a release vote: Forward a copy of the proposed resolution and a link 
to the PPMC graduation VOTE.

So the procedure is:

1. Start a DISCUSS thread on general@incubator. Once that discussion dies down, 
and if there seems to be positive consensus…
2. Start a VOTE thread for the same.

-Taylor


> On Mar 22, 2017, at 12:53 PM, James Sirota  wrote:
> 
> Mentors,
> 
> Looks like we have everything we need to initiate the incubator board vote to 
> leave the incubator.  Can you provide some guidance as to what is the best 
> way to approach this?
> 
> --- 
> Thank you,
> 
> James Sirota
> PPMC- Apache Metron (Incubating)
> jsirota AT apache DOT org

possibly useful info re kafka and kafka connector versions, with Storm

[Matt Foley]

Repost from user@storm:

 

From: Harsha Chintalapani 
Reply-To: "u...@storm.apache.org" 
Date: Thursday, March 23, 2017 at 8:51 AM
To: "u...@storm.apache.org" 
Subject: Re: Valid Kafka version for Storm 1.0.2

 

Hi Anis,

We've two kafka connectors now. One uses old consumer API (Simple 
Consumer) https://github.com/apache/storm/tree/master/external/storm-kafka

and there is new storm-kafka-client which uses new consumer API 
https://github.com/apache/storm/tree/master/external/storm-kafka-client

The way we build connectors are not necessarily tied to a specific version , as 
long as the client APIs are backward compatible you are free to use any version.

In case of 1, You can use kafka version from 0.8.x to 0.10.x

In case of 2, We recommend you use 0.10.0.1 onwards as thats when the kafka 
client APIs are stabilized and had critical bug-fixes.

Thanks,

Harsha

On Tue, Mar 21, 2017 at 6:45 PM Anis Nasir  wrote:

Dear all, 

 

Can anyone suggest me a working version of Kafka for Storm 1.0.2.

 

Thanking you in advance.

 

Regards,

Anis

 

 

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107687487
  
--- Diff: 
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
 ---
@@ -96,16 +109,18 @@ public HDFSWriterCallback withConfig(HDFSWriterConfig 
config) {
 this.config = config;
 return this;
 }
+
 @Override
 public List apply(List tuple, EmitContext context) {
-
-List keyValue = (List) tuple.get(0);
-LongWritable ts = (LongWritable) keyValue.get(0);
-BytesWritable rawPacket = (BytesWritable)keyValue.get(1);
+byte[] key = (byte[]) tuple.get(0);
+byte[] value = (byte[]) tuple.get(1);
+if(!config.getDeserializer().deserializeKeyValue(key, value, 
KeyValue.key.get(), KeyValue.value.get())) {
+LOG.debug("Dropping malformed packet...");
--- End diff --

yeah, you're right.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
Follow-on JIRA to move back to apache storm for storm-kafka-client is at 
https://issues.apache.org/jira/browse/METRON-794


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107683860
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.storm.kafka.flux;
+
+import com.google.common.base.Joiner;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.storm.kafka.spout.*;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.OutputFieldsGetter;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/**
+ * This is a convenience layer on top of the KafkaSpoutConfig.Builder 
available in storm-kafka-client.
+ * The justification for this class is two-fold.  First, there are a lot 
of moving parts and a simplified
+ * approach to constructing spouts is useful.  Secondly, and perhaps more 
importantly, the Builder pattern
+ * is decidedly unfriendly to use inside of Flux.  Finally, we can make 
things a bit more friendly by only requiring
+ * zookeeper and automatically figuring out the brokers for the bootstrap 
server.
+ *
+ * @param  The kafka key type
+ * @param  The kafka value type
+ */
+public class SimpleStormKafkaBuilder extends 
KafkaSpoutConfig.Builder {
+  final static String STREAM = "default";
+
+  /**
+   * The fields exposed by the kafka consumer.  These will show up in the 
Storm tuple.
+   */
+  public enum FieldsConfiguration {
+KEY("key", record -> record.key()),
+VALUE("value", record -> record.value()),
+PARTITION("partition", record -> record.partition()),
+TOPIC("topic", record -> record.topic())
+;
+String fieldName;
+Function recordExtractor;
+
+FieldsConfiguration(String fieldName, Function 
recordExtractor) {
+  this.recordExtractor = recordExtractor;
+  this.fieldName = fieldName;
+}
+
+/**
+ * Return a list of the enums
+ * @param configs
+ * @return
+ */
+public static List toList(String... configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Return a list of the enums from their string representation.
+ * @param configs
+ * @return
+ */
+public static List toList(List configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Construct a Fields object from an iterable of enums.  These fields 
are the fields
+ * exposed in the Storm tuple emitted from the spout.
+ * @param configs
+ * @return
+ */
+public static Fields getFields(Iterable configs) {
+  List fields = new ArrayList<>();
+  for(FieldsConfiguration config : configs) {
+fields.add(config.fieldName);
+  }
+  return new Fields(fields);
+}
+  }
+
+  /**
+   * Build a tuple given the fields and the topic.  We want to use our 
FieldsConfiguration enum
+   * to define what this tuple looks like.
+   * @param  The key type in kafka
+   * @param  The value 

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107683873
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -0,0 +1,234 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.storm.kafka.flux;
+
+import com.google.common.base.Joiner;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.metron.common.utils.KafkaUtils;
+import org.apache.storm.kafka.spout.*;
+import org.apache.storm.spout.SpoutOutputCollector;
+import org.apache.storm.topology.OutputFieldsDeclarer;
+import org.apache.storm.topology.OutputFieldsGetter;
+import org.apache.storm.tuple.Fields;
+import org.apache.storm.tuple.Values;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Map;
+import java.util.function.Function;
+
+/**
+ * This is a convenience layer on top of the KafkaSpoutConfig.Builder 
available in storm-kafka-client.
+ * The justification for this class is two-fold.  First, there are a lot 
of moving parts and a simplified
+ * approach to constructing spouts is useful.  Secondly, and perhaps more 
importantly, the Builder pattern
+ * is decidedly unfriendly to use inside of Flux.  Finally, we can make 
things a bit more friendly by only requiring
+ * zookeeper and automatically figuring out the brokers for the bootstrap 
server.
+ *
+ * @param  The kafka key type
+ * @param  The kafka value type
+ */
+public class SimpleStormKafkaBuilder extends 
KafkaSpoutConfig.Builder {
+  final static String STREAM = "default";
+
+  /**
+   * The fields exposed by the kafka consumer.  These will show up in the 
Storm tuple.
+   */
+  public enum FieldsConfiguration {
+KEY("key", record -> record.key()),
+VALUE("value", record -> record.value()),
+PARTITION("partition", record -> record.partition()),
+TOPIC("topic", record -> record.topic())
+;
+String fieldName;
+Function recordExtractor;
+
+FieldsConfiguration(String fieldName, Function 
recordExtractor) {
+  this.recordExtractor = recordExtractor;
+  this.fieldName = fieldName;
+}
+
+/**
+ * Return a list of the enums
+ * @param configs
+ * @return
+ */
+public static List toList(String... configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Return a list of the enums from their string representation.
+ * @param configs
+ * @return
+ */
+public static List toList(List configs) {
+  List ret = new ArrayList<>();
+  for(String config : configs) {
+ret.add(FieldsConfiguration.valueOf(config.toUpperCase()));
+  }
+  return ret;
+}
+
+/**
+ * Construct a Fields object from an iterable of enums.  These fields 
are the fields
+ * exposed in the Storm tuple emitted from the spout.
+ * @param configs
+ * @return
+ */
+public static Fields getFields(Iterable configs) {
+  List fields = new ArrayList<>();
+  for(FieldsConfiguration config : configs) {
+fields.add(config.fieldName);
+  }
+  return new Fields(fields);
+}
+  }
+
+  /**
+   * Build a tuple given the fields and the topic.  We want to use our 
FieldsConfiguration enum
+   * to define what this tuple looks like.
+   * @param  The key type in kafka
+   * @param  The value 

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107683392
  
--- Diff: 
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
 ---
@@ -96,16 +109,18 @@ public HDFSWriterCallback withConfig(HDFSWriterConfig 
config) {
 this.config = config;
 return this;
 }
+
 @Override
 public List apply(List tuple, EmitContext context) {
-
-List keyValue = (List) tuple.get(0);
-LongWritable ts = (LongWritable) keyValue.get(0);
-BytesWritable rawPacket = (BytesWritable)keyValue.get(1);
+byte[] key = (byte[]) tuple.get(0);
+byte[] value = (byte[]) tuple.get(1);
+if(!config.getDeserializer().deserializeKeyValue(key, value, 
KeyValue.key.get(), KeyValue.value.get())) {
+LOG.debug("Dropping malformed packet...");
--- End diff --

I'm good with that. I had mostly discarded the worry about size because 
this is in debugging statements anyway and typically with storm you're setting 
reasonable timeouts on logging levels anyway.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107683042
  
--- Diff: 
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/deserializer/Deserializers.java
 ---
@@ -0,0 +1,59 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.spout.pcap.deserializer;
+
+import org.apache.metron.common.utils.timestamp.TimestampConverters;
+import org.apache.metron.common.utils.timestamp.TimestampConverter;
+
+import java.util.function.Function;
+
+/**
+ * Deserializers take the raw bytes from kafka key and value and construct 
the timestamp and raw bytes for PCAP.
+ */
+public enum Deserializers {
+  /**
+   * Extract the timestamp from the key and the raw packet 
(global-headerless) from the value
+   */
+   FROM_KEY( converter -> new FromKeyDeserializer(converter))
+  /**
+   * Ignore the key and pull the timestamp directly from the packet 
itself.  Also, assume that the packet isn't global-headerless.
+   */
+  ,FROM_PACKET(converter -> new FromPacketDeserializer());
+  ;
+  Function creator;
+  Deserializers(Function creator)
+  {
+this.creator = creator;
+  }
+
+  public static KeyValueDeserializer create(String scheme, 
TimestampConverter converter) {
+try {
+  Deserializers ts = Deserializers.valueOf(scheme.toUpperCase());
+  return ts.creator.apply(converter);
+}
+catch(IllegalArgumentException iae) {
+  return Deserializers.FROM_KEY.creator.apply(converter);
+}
+  }
+
+  public static KeyValueDeserializer create(String scheme, String 
converter) {
+return create(scheme, 
TimestampConverters.valueOf(converter.toUpperCase()));
--- End diff --

yep


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107682535
  
--- Diff: 
metron-platform/metron-pcap-backend/src/main/java/org/apache/metron/spout/pcap/HDFSWriterCallback.java
 ---
@@ -96,16 +109,18 @@ public HDFSWriterCallback withConfig(HDFSWriterConfig 
config) {
 this.config = config;
 return this;
 }
+
 @Override
 public List apply(List tuple, EmitContext context) {
-
-List keyValue = (List) tuple.get(0);
-LongWritable ts = (LongWritable) keyValue.get(0);
-BytesWritable rawPacket = (BytesWritable)keyValue.get(1);
+byte[] key = (byte[]) tuple.get(0);
+byte[] value = (byte[]) tuple.get(1);
+if(!config.getDeserializer().deserializeKeyValue(key, value, 
KeyValue.key.get(), KeyValue.value.get())) {
+LOG.debug("Dropping malformed packet...");
--- End diff --

I could, but, of course, they're raw bytes and the value could be rather 
big.  How about I compromise and make this a bit more helpful by including:
* Their lengths
* The key itself in hex


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #486: METRON-793: Migrate to storm-kafka-clien...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/incubator-metron/pull/486#discussion_r107681982
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/topology/ParserTopologyBuilder.java
 ---
@@ -106,19 +105,26 @@ public static TopologyBuilder build(String 
zookeeperUrl,
   /**
* Create a spout that consumes tuples from a Kafka topic.
*
-   * @param zookeeperUrlZookeeper URL
+   * @param zkQuorum Zookeeper URL
* @param sensorType  Type of sensor
-   * @param offset  Kafka topic offset where the topology 
will start; BEGINNING, END, WHERE_I_LEFT_OFF
-   * @param kafkaSpoutConfigOptions Configuration options for the kafka 
spout
+   * @param kafkaConfigOptional Configuration options for the kafka 
spout
* @param parserConfigConfiguration for the parser
* @return
*/
-  private static KafkaSpout createKafkaSpout(String zookeeperUrl, String 
sensorType, SpoutConfig.Offset offset, EnumMap 
kafkaSpoutConfigOptions, SensorParserConfig parserConfig) {
-
+  private static StormKafkaSpout createKafkaSpout(String zkQuorum, String 
sensorType, Optional> kafkaConfigOptional, 
SensorParserConfig parserConfig) {
+Map kafkaSpoutConfigOptions = 
kafkaConfigOptional.orElse(new HashMap<>());
 String inputTopic = parserConfig.getSensorTopic() != null ? 
parserConfig.getSensorTopic() : sensorType;
-SpoutConfig spoutConfig = new SpoutConfig(new ZkHosts(zookeeperUrl), 
inputTopic, "", inputTopic).from(offset);
-SpoutConfigOptions.configure(spoutConfig, kafkaSpoutConfigOptions);
-return new KafkaSpout(spoutConfig);
+
if(!kafkaSpoutConfigOptions.containsKey(SpoutConfiguration.FIRST_POLL_OFFSET_STRATEGY.key))
 {
--- End diff --

Yep


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron pull request #487: METRON-792: Quick Dev should remove/repl...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/incubator-metron/pull/487


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #487: METRON-792: Quick Dev should remove/replace RPM...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/487
  
I'm +1 by inspection


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] incubator-metron issue #486: METRON-793: Migrate to storm-kafka-client kafka...

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/incubator-metron/pull/486
  
@cestella I'm good with keeping the extension points especially after the 
points you made. I think the TODOs are valuable, I just wanted to know the 
thought behind potentially building it out.

Given the API instability, unfortunately it seems like our dependencies 
aren't going to provide that insulation layer.  I'd rather have that be 
provided in a stable manner upstream from us, but that's not something we have 
any control over.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS][PROPOSAL] Acceptance Tests

[Nick Allen]

Although I disagree, I completely understand the negative feelings towards
Ansible.  I think Ansible has its problems, but it also has many advantages
too.

I would be in favor of Ansible over something with less structure like a
set of Bash scripts.  At the same time, I would probably favor a number of
alternatives over Ansible, but I just don't know what those are yet.

On Wed, Mar 22, 2017 at 3:07 PM, David Lyle  wrote:

> I'm -1 on Ansible for testing for the same reasons I recommended reducing
> reliance on it for deployment. It simply isn't well suited to be a general
> purpose installer (or testing framework) for the variety of OS/Ansible
> versions that we find in the wild.
>
> I'd figure out how/what we want to test and then see what we need for a
> suitable framework. I'm certain we can find one that generalizes better
> than Ansible has for us so far.
>
> -D...
>
>
> On Wed, Mar 22, 2017 at 3:00 PM, Ryan Merriman 
> wrote:
>
> > I don't think a cluster installed by ansible is a prerequisite to using
> > ansible to integration test.  They would be completely separate modules
> > except maybe sharing some property or inventory files.  Just need to run
> > scripts and hit rest endpoints right?  Just an idea, maybe it's overkill.
> > I'm cool with rolling our own.
> >
> > On Wed, Mar 22, 2017 at 1:49 PM, Casey Stella 
> wrote:
> >
> > > Maybe, but I'd argue that we would want this to be run against a
> > > non-ansible installed cluster.  For a first pass, I'd recommend just a
> > set
> > > of shell scripts utilizing the REPL and the REST API along with shell
> > > commands.  Most of our capabilities are quite scriptable.
> > >
> > > On Wed, Mar 22, 2017 at 2:47 PM, Ryan Merriman 
> > > wrote:
> > >
> > > > Bumping this thread.  Looks like we have several +1s so I propose we
> > move
> > > > to the next step.  I'm anxious to get this done because these tests
> > would
> > > > have saved me time over the last couple weeks.  The management UI in
> > > > https://github.com/apache/incubator-metron/pull/484 has a set of e2e
> > > tests
> > > > being maintained in another branch so those could also be included in
> > > this
> > > > test suite when the UI makes it into master.
> > > >
> > > > Ideas for an "Acceptance Testing Framework"?  Could Ansible be good
> fit
> > > for
> > > > this since we already have it in our stack?
> > > >
> > > > On Mon, Mar 6, 2017 at 1:01 PM, Michael Miklavcic <
> > > > michael.miklav...@gmail.com> wrote:
> > > >
> > > > > Ok, yes I agree. In my experience with e2e/acceptance tests,
> they're
> > > best
> > > > > kept general with an emphasis on verifying that all the plumbing
> > works
> > > > > together. So yes, there are definite edge cases I think we'll want
> to
> > > > test
> > > > > here, but I say that with the caveat that I think we should ideally
> > > cover
> > > > > as many non-happy-path cases in unit and integration tests as
> > possible.
> > > > As
> > > > > an example, I don't think it makes sense to cover most of the
> > profiler
> > > > > windowing DSL language edge cases in acceptance tests instead of or
> > in
> > > > > addition to unit/integration tests unless there is something
> specific
> > > to
> > > > > the integration with a given an environment that we think could be
> > > > > problematic.
> > > > >
> > > > > M
> > > > >
> > > > > On Mon, Mar 6, 2017 at 11:32 AM, Casey Stella 
> > > > wrote:
> > > > >
> > > > > > No, I'm saying that they shouldn't be restricted to real-world
> > > > use-cases.
> > > > > > The E2E tests I laid out weren't real-world, but they did
> exercise
> > > the
> > > > > > components similar to real-world use-cases.  They should also be
> > able
> > > > to
> > > > > be
> > > > > > able to tread outside of the happy-path for those use-cases.
> > > > > >
> > > > > > On Mon, Mar 6, 2017 at 6:30 PM, Michael Miklavcic <
> > > > > > michael.miklav...@gmail.com> wrote:
> > > > > >
> > > > > > > "I don't think acceptance tests should loosely associate with
> > real
> > > > > uses,
> > > > > > > but they should
> > > > > > > be free to delve into weird non-happy-pathways."
> > > > > > >
> > > > > > > Not following - are you saying they should *tightly* associate
> > with
> > > > > real
> > > > > > > uses and additonally include non-happy-path?
> > > > > > >
> > > > > > > On Fri, Mar 3, 2017 at 12:57 PM, Casey Stella <
> > ceste...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > It is absolutely not a naive question, Matt.  We don't have a
> > lot
> > > > (or
> > > > > > > any)
> > > > > > > > docs about our integration tests; it's more of a "follow the
> > > lead"
> > > > > type
> > > > > > > of
> > > > > > > > thing at the moment, but that should be rectified.
> > > > > > > >
> > > > > > > > The integration tests spin up and down infrastructure
> > in-process,
> > > > > some
> > > > > > of
> > > > > > > > which are real