[GitHub] metron pull request #590: METRON-953: Complete migration away from incubator...

[justinleet]

GitHub user justinleet opened a pull request:

https://github.com/apache/metron/pull/590

METRON-953: Complete migration away from incubator-metron to metron for 
links and docs

## Contributor Comments
Moving "incubator-metron" to "metron" everywhere other than the 
current-book directory.  Now that infra is setup everywhere, all links should 
work with just metron.  In addition, references to the incubator-metron 
directory are changed to metron (because a clean clone should result in a 
metron dir now).

At this point, it should only appear in the current-book.  If any of these 
should survive, let me know.

I still need to run up full dev to validate, but I've opened the links 
manually and had them resolve.

To validate, make sure there aren't any incubator-metron instances 
leftover.  Ensure everything still builds and that the site-book looks good 
(other than anything derived from current-book).  Make sure the various links 
aren't broken. One Zeppelin dashboard (pcap) is effected by this, and to 
validate on full dev requires setting up per 
https://github.com/apache/metron/pull/559.

Any other validation needs are welcome.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [ ] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron github_travis

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/590.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #590


commit e7c9a408b39f159f097551363e94a80ab905
Author: justinjleet 
Date:   2017-05-14T23:31:52Z

Changing the various github links and such

commit d9045423ae57b74f6e09d87ea988467393b05b05
Author: justinjleet 
Date:   2017-05-15T14:23:20Z

adding more stuff




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #584: METRON-950: Migrate storm-kafka-client to 1.1

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/584
  
@nickwallen Yeah, that's where I landed. Looking in our pcap helper class, 
I see the following.
```
  public static MetronEthernetDecoder createDecoder() {
MetronEthernetDecoder ethernetDecoder = new MetronEthernetDecoder();
IpDecoder ipDecoder = new IpDecoder();
ethernetDecoder.register(EthernetType.IPV4, ipDecoder);
return ethernetDecoder;
  }
```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #584: METRON-950: Migrate storm-kafka-client to 1.1

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/584
  
Does `pcap_inspector` only print out for TCP packets?  The first 5 of those 
are all UDP based (NTP & DNS).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #584: METRON-950: Migrate storm-kafka-client to 1.1

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/584
  
When I spin this up in full dev, half the records don't output with the 
pcap_inspector. I'm running:
`pycapa --producer --interface eth0 --kafka-broker node1:6667 --kafka-topic 
pcap --max-packets 10`

And I get 10 records, as expected. 

/usr/metron/0.4.0/bin/pcap_inspector.sh -i 
/apps/metron/pcap/pcap_pcap_1494886095377022000_0_pcap-7-1494887672
```
TS: May 15, 2017 10:08:15 PM UTC
TS: May 15, 2017 10:08:17 PM UTC
TS: May 15, 2017 10:08:17 PM UTC
TS: May 15, 2017 10:08:20 PM UTC
TS: May 15, 2017 10:08:20 PM UTC
TS: May 15, 2017 10:08:22 PM UTC,ip_src_addr: 10.0.2.2,ip_src_port: 
58390,ip_dst_addr: 10.0.2.15,ip_dst_port: 22,protocol: 6
TS: May 15, 2017 10:08:22 PM UTC,ip_src_addr: 10.0.2.15,ip_src_port: 
22,ip_dst_addr: 10.0.2.2,ip_dst_port: 58390,protocol: 6
TS: May 15, 2017 10:08:25 PM UTC,ip_src_addr: 10.0.2.2,ip_src_port: 
58390,ip_dst_addr: 10.0.2.15,ip_dst_port: 22,protocol: 6
TS: May 15, 2017 10:08:25 PM UTC,ip_src_addr: 10.0.2.15,ip_src_port: 
22,ip_dst_addr: 10.0.2.2,ip_dst_port: 58390,protocol: 6
TS: May 15, 2017 10:08:27 PM UTC,ip_src_addr: 10.0.2.2,ip_src_port: 
58390,ip_dst_addr: 10.0.2.15,ip_dst_port: 22,protocol: 6
```
Wireshark seems to be OK.

![image](https://cloud.githubusercontent.com/assets/658443/26083299/29d5c0fe-3992-11e7-8139-ed9e103c4384.png)

I'm looking into this and will update as I have more info.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: mvn building errors with 0.3.1

[Ryan Merriman]

Kevin, I think your error is related to npm.  Can you attach the full log
file?

On Mon, May 15, 2017 at 5:23 PM, Kevin Waterson 
wrote:

> I am getting something similar..
>
> [INFO] BUILD FAILURE
> [INFO]
> 
> [INFO] Total time: 08:08 min
> [INFO] Finished at: 2017-05-15T13:04:00+10:00
> [INFO] Final Memory: 220M/3873M
> [INFO]
> 
> [ERROR] Failed to execute goal
> com.github.eirslett:frontend-maven-plugin:1.3:npm (ng build) on project
> metron-config: Failed to run task: 'npm run build' failed. (error code 1)
> -> [Help 1]
>
> Looking back through the code I can see
> [ERROR] at
> /home/kevin/metron/docs/incubator-metron/metron-
> interface/metron-config/node_modules/resolve/lib/async.js:24:24
> [ERROR] at FSReqWrap.oncomplete (fs.js:117:15)
> [ERROR]
> [ERROR] npm ERR! Linux 4.4.0-75-generic
> [ERROR] npm ERR! argv
> "/home/kevin/metron/docs/incubator-metron/metron-
> interface/metron-config/node/node"
> "/home/kevin/metron/docs/incubator-metron/metron-
> interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
> "run" "build"
> [ERROR] npm ERR! node v6.2.0
> [ERROR] npm ERR! npm  v3.8.9
> [ERROR] npm ERR! code ELIFECYCLE
> [ERROR] npm ERR! metron-management-ui@0.4.0 build:
> `./node_modules/angular-cli/bin/ng build -prod`
> [ERROR] npm ERR! Exit status 1
> [ERROR] npm ERR!
> [ERROR] npm ERR! Failed at the metron-management-ui@0.4.0 build script
> './node_modules/angular-cli/bin/ng build -prod'.
> [ERROR] npm ERR! Make sure you have the latest version of node.js and npm
> installed.
> [ERROR] npm ERR! If you do, this is most likely a problem with the
> metron-management-ui package,
> [ERROR] npm ERR! not with npm itself.
> [ERROR] npm ERR! Tell the author that this fails on your system:
> [ERROR] npm ERR! ./node_modules/angular-cli/bin/ng build -prod
> [ERROR] npm ERR! You can get information on how to open an issue for this
> project with:
> [ERROR] npm ERR! npm bugs metron-management-ui
> [ERROR] npm ERR! Or if that isn't available, you can get their info via:
> [ERROR] npm ERR! npm owner ls metron-management-ui
> [ERROR] npm ERR! There is likely additional logging output above.
> [ERROR]
>
>
> Kevin
>
> On Tue, May 16, 2017 at 4:32 AM, Dima Kovalyov 
> wrote:
>
> > Hello Metron devs,
> >
> > I am trying to merge Apache Metron branch Metron_0.3.1 into our Metron
> > fork. I resolved few conflicts in regards to pom and travis xml files.
> > When I build it, I receive following error:
> > [INFO] Compiling 55 source files to
> > /home/redacted/sst-metron/metron-platform/metron-
> enrichment/target/classes
> > /home/redacted/sst-metron/metron-platform/metron-
> > enrichment/src/main/java/org/apache/metron/enrichment/
> > adapters/simplehbase/SimpleHBaseAdapter.java:90:
> > warning: [unchecked] unchecked call to put(K,V) as a member of the raw
> > type HashMap
> >   enriched.put(kv.getKey().type + "." + values.getKey(),
> > values.getValue());
> >   ^
> >   where K,V are type-variables:
> > K extends Object declared in class HashMap
> > V extends Object declared in class HashMap
> > ...
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile
> > (default-compile) on project metron-enrichment: Compilation failure ->
> > [Help 1]
> >
> > Attached full file for check.
> >
> > Both repos building fine separately.
> >
> > Can you please advise where I can find file that causes this?
> > Thank you.
> >
> > - Dima
> >
>

Re: mvn building errors with 0.3.1

[Kevin Waterson]

I am getting something similar..

[INFO] BUILD FAILURE
[INFO]

[INFO] Total time: 08:08 min
[INFO] Finished at: 2017-05-15T13:04:00+10:00
[INFO] Final Memory: 220M/3873M
[INFO]

[ERROR] Failed to execute goal
com.github.eirslett:frontend-maven-plugin:1.3:npm (ng build) on project
metron-config: Failed to run task: 'npm run build' failed. (error code 1)
-> [Help 1]

Looking back through the code I can see
[ERROR] at
/home/kevin/metron/docs/incubator-metron/metron-interface/metron-config/node_modules/resolve/lib/async.js:24:24
[ERROR] at FSReqWrap.oncomplete (fs.js:117:15)
[ERROR]
[ERROR] npm ERR! Linux 4.4.0-75-generic
[ERROR] npm ERR! argv
"/home/kevin/metron/docs/incubator-metron/metron-interface/metron-config/node/node"
"/home/kevin/metron/docs/incubator-metron/metron-interface/metron-config/node/node_modules/npm/bin/npm-cli.js"
"run" "build"
[ERROR] npm ERR! node v6.2.0
[ERROR] npm ERR! npm  v3.8.9
[ERROR] npm ERR! code ELIFECYCLE
[ERROR] npm ERR! metron-management-ui@0.4.0 build:
`./node_modules/angular-cli/bin/ng build -prod`
[ERROR] npm ERR! Exit status 1
[ERROR] npm ERR!
[ERROR] npm ERR! Failed at the metron-management-ui@0.4.0 build script
'./node_modules/angular-cli/bin/ng build -prod'.
[ERROR] npm ERR! Make sure you have the latest version of node.js and npm
installed.
[ERROR] npm ERR! If you do, this is most likely a problem with the
metron-management-ui package,
[ERROR] npm ERR! not with npm itself.
[ERROR] npm ERR! Tell the author that this fails on your system:
[ERROR] npm ERR! ./node_modules/angular-cli/bin/ng build -prod
[ERROR] npm ERR! You can get information on how to open an issue for this
project with:
[ERROR] npm ERR! npm bugs metron-management-ui
[ERROR] npm ERR! Or if that isn't available, you can get their info via:
[ERROR] npm ERR! npm owner ls metron-management-ui
[ERROR] npm ERR! There is likely additional logging output above.
[ERROR]


Kevin

On Tue, May 16, 2017 at 4:32 AM, Dima Kovalyov 
wrote:

> Hello Metron devs,
>
> I am trying to merge Apache Metron branch Metron_0.3.1 into our Metron
> fork. I resolved few conflicts in regards to pom and travis xml files.
> When I build it, I receive following error:
> [INFO] Compiling 55 source files to
> /home/redacted/sst-metron/metron-platform/metron-enrichment/target/classes
> /home/redacted/sst-metron/metron-platform/metron-
> enrichment/src/main/java/org/apache/metron/enrichment/
> adapters/simplehbase/SimpleHBaseAdapter.java:90:
> warning: [unchecked] unchecked call to put(K,V) as a member of the raw
> type HashMap
>   enriched.put(kv.getKey().type + "." + values.getKey(),
> values.getValue());
>   ^
>   where K,V are type-variables:
> K extends Object declared in class HashMap
> V extends Object declared in class HashMap
> ...
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.5.1:compile
> (default-compile) on project metron-enrichment: Compilation failure ->
> [Help 1]
>
> Attached full file for check.
>
> Both repos building fine separately.
>
> Can you please advise where I can find file that causes this?
> Thank you.
>
> - Dima
>

[GitHub] metron pull request #584: METRON-950: Migrate storm-kafka-client to 1.1

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/584#discussion_r116584769
  
--- Diff: 
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
 ---
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka.spout.internal;
+
+import java.util.concurrent.TimeUnit;
+import org.apache.storm.utils.Time;
+
+public class Timer {
+  private final long delay;
+  private final long period;
+  private final TimeUnit timeUnit;
+  private final long periodNanos;
+  private long start;
+
+  public Timer(long delay, long period, TimeUnit timeUnit) {
+this.delay = delay;
+this.period = period;
+this.timeUnit = timeUnit;
+this.periodNanos = timeUnit.toNanos(period);
+this.start = System.nanoTime() + timeUnit.toNanos(delay);
+  }
+
+  public long period() {
+return this.period;
+  }
+
+  public long delay() {
+return this.delay;
+  }
+
+  public TimeUnit getTimeUnit() {
+return this.timeUnit;
+  }
+
+  public boolean isExpiredResetOnTrue() {
--- End diff --

Ok, done, added test.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #588: METRON-954: Create ability to change output topic ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/588#discussion_r116583655
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/SquidIntegrationTest.java
 ---
@@ -29,6 +29,11 @@ String getSensorType() {
   }
 
   @Override
--- End diff --

haha ok, you got it.  I'll adjust it tomorrow to be a different type of 
test and not piggyback on squid.  You're right, the parser tests are for 
testing the parsers, not testing the ancillary stuff, like kafka writers.  
Thanks for keeping me honest, @ottobackwards 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #588: METRON-954: Create ability to change output topic ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/588#discussion_r116583237
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/SquidIntegrationTest.java
 ---
@@ -29,6 +29,11 @@ String getSensorType() {
   }
 
   @Override
--- End diff --

I have a couple of things that come to mind, not all relevant:

* With 777 and 942, squid is a real parser extension, installed and usable 
etc.  So throwing in a test of something outside the parser into it's 
integration test, and making it _diffferent_ than the others seems not right to 
me.

* Same for any parser really.  The parsers tests are for testing the 
parsers.  You guys did a lot of work making the tests alike, and 
understandable, we should maintain that.

* If you don't change that, I don't think I'll have to do conflict work in 
777, which I continue to have to maintain ;) ( disregard this )


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #589: METRON-955: Make the default sync policy for HDFS ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/589#discussion_r116582720
  
--- Diff: 
metron-platform/metron-writer/src/main/java/org/apache/metron/writer/hdfs/HdfsWriter.java
 ---
@@ -111,6 +121,16 @@ public BulkWriterResponse write(String sourceType
 return response;
   }
 
+  public SyncPolicy createSyncPolicy(int batchSize) {
--- End diff --

yep


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #588: METRON-954: Create ability to change output topic ...

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/588#discussion_r116582169
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/SquidIntegrationTest.java
 ---
@@ -29,6 +29,11 @@ String getSensorType() {
   }
 
   @Override
--- End diff --

We could spin up a new integration test, but I thought this did the trick 
and didn't add the cost of another integration test to the build time.  

I do have a test ensuring the value gets passed through to the parser.  
This just completed the loop.  If we don't feel that an integration test is 
necessary here, then I can remove it too.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #588: METRON-954: Create ability to change output topic ...

[ottobackwards]

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/588#discussion_r116581479
  
--- Diff: 
metron-platform/metron-parsers/src/test/java/org/apache/metron/parsers/integration/SquidIntegrationTest.java
 ---
@@ -29,6 +29,11 @@ String getSensorType() {
   }
 
   @Override
--- End diff --

Is this required?  Could we write a test just for testing the setting?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #584: METRON-950: Migrate storm-kafka-client to 1.1

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/584#discussion_r116581475
  
--- Diff: 
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
 ---
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka.spout.internal;
+
+import java.util.concurrent.TimeUnit;
+import org.apache.storm.utils.Time;
+
+public class Timer {
+  private final long delay;
+  private final long period;
+  private final TimeUnit timeUnit;
+  private final long periodNanos;
+  private long start;
+
+  public Timer(long delay, long period, TimeUnit timeUnit) {
+this.delay = delay;
+this.period = period;
+this.timeUnit = timeUnit;
+this.periodNanos = timeUnit.toNanos(period);
+this.start = System.nanoTime() + timeUnit.toNanos(delay);
+  }
+
+  public long period() {
+return this.period;
+  }
+
+  public long delay() {
+return this.delay;
+  }
+
+  public TimeUnit getTimeUnit() {
+return this.timeUnit;
+  }
+
+  public boolean isExpiredResetOnTrue() {
--- End diff --

It is precisely the 1.1.0 version of the code with the exception of calling 
`System.nanoTime` instead of `Time.nanoTime` (see 
[here](https://github.com/apache/storm/blob/a4afacd9617d620f50cf026fc599821f7ac25c79/external/storm-kafka-client/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java)
 ).

It should be noted that `Time.nanoTime` calls `System.nanoTime` (see 
[here](https://github.com/apache/storm/blob/a4afacd9617d620f50cf026fc599821f7ac25c79/storm-client/src/jvm/org/apache/storm/utils/Time.java#L162)
 ) in a production scenario (i.e. non-simulation).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #584: METRON-950: Migrate storm-kafka-client to 1.1

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/584
  
OK, works for me


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #584: METRON-950: Migrate storm-kafka-client to 1.1

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/584#discussion_r116579916
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -202,31 +240,4 @@ public String getTopic() {
 return new StormKafkaSpout<>(builder);
   }
 
-  private static Map modifyKafkaProps(Map 
props, String zkQuorum) {
-try {
-  if(!props.containsKey(KafkaSpoutConfig.Consumer.BOOTSTRAP_SERVERS)) {
-//this isn't a putIfAbsent because I only want to pull the brokers 
from zk if it's absent.
-List brokers = 
KafkaUtils.INSTANCE.getBrokersFromZookeeper(zkQuorum);
-props.put(KafkaSpoutConfig.Consumer.BOOTSTRAP_SERVERS, 
Joiner.on(",").join(brokers));
-  }
-  props.putIfAbsent(KafkaSpoutConfig.Consumer.KEY_DESERIALIZER, 
ByteArrayDeserializer.class.getName());
-  props.putIfAbsent(KafkaSpoutConfig.Consumer.VALUE_DESERIALIZER, 
ByteArrayDeserializer.class.getName());
-
-} catch (Exception e) {
-  throw new IllegalStateException("Unable to retrieve brokers from 
zookeeper: " + e.getMessage(), e);
-}
-return props;
-  }
-
-  private static  KafkaSpoutTuplesBuilder 
createTuplesBuilder(List config, String topic) {
-TupleBuilder tb =  new TupleBuilder(topic, 
FieldsConfiguration.toList(config));
-return new KafkaSpoutTuplesBuilderNamedTopics.Builder<>(tb).build();
-  }
-
-
-  private static KafkaSpoutStreams createStreams(List config, 
String topic) {
-final Fields fields = 
FieldsConfiguration.getFields(FieldsConfiguration.toList(config));
-return new KafkaSpoutStreamsNamedTopics.Builder(fields, STREAM, new 
String[] { topic} ).build();
--- End diff --

Yep


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #584: METRON-950: Migrate storm-kafka-client to 1.1

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/584
  
@mmiklavc it is.  To give guarantees beyond that, would require us to shade 
and relocate the `metron-kafka-spout` project, which I did not want to do 
without some good justification.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #584: METRON-950: Migrate storm-kafka-client to 1.1

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/584
  
Just to clarify for others looking at this, there is a `Timer` class in 
`storm-kafka-client-1.1.0` which depends on a `Time` class (no 'r') in 
storm-core. In 1.0.3, the main version of Storm we depend on, there is no 
`nanoTime()` method. This was introduced in 1.1. @cestella is handling this via 
the temporary module `metron-storm-kafka-override` via `System.nanoTime()`.

We're only recommending upgrading the client currently because, first and 
foremost, it provides the desired performance improvements, and second the 
impact is significantly less than upgrading the whole of Storm.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #584: METRON-950: Migrate storm-kafka-client to 1.1

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/584#discussion_r116574125
  
--- Diff: 
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
 ---
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka.spout.internal;
+
+import java.util.concurrent.TimeUnit;
+import org.apache.storm.utils.Time;
+
+public class Timer {
+  private final long delay;
+  private final long period;
+  private final TimeUnit timeUnit;
+  private final long periodNanos;
+  private long start;
+
+  public Timer(long delay, long period, TimeUnit timeUnit) {
+this.delay = delay;
+this.period = period;
+this.timeUnit = timeUnit;
+this.periodNanos = timeUnit.toNanos(period);
+this.start = System.nanoTime() + timeUnit.toNanos(delay);
+  }
+
+  public long period() {
+return this.period;
+  }
+
+  public long delay() {
+return this.delay;
+  }
+
+  public TimeUnit getTimeUnit() {
+return this.timeUnit;
+  }
+
+  public boolean isExpiredResetOnTrue() {
--- End diff --

That's a good catch.  It'd be nice to have that tested.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #584: METRON-950: Migrate storm-kafka-client to 1.1

[cestella]

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/584#discussion_r116574075
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -165,13 +177,39 @@ public SimpleStormKafkaBuilder( Map 
kafkaProps
 , List fieldsConfiguration
 )
   {
-super( modifyKafkaProps(kafkaProps, zkQuorum)
- , createStreams(fieldsConfiguration, topic)
- , createTuplesBuilder(fieldsConfiguration, topic)
- );
+super( getBootstrapServers(zkQuorum, kafkaProps)
+ , 
createDeserializer(Optional.ofNullable((String)kafkaProps.get(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG)),
 DEFAULT_DESERIALIZER)
+ , 
createDeserializer(Optional.ofNullable((String)kafkaProps.get(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG)),
 DEFAULT_DESERIALIZER)
+ , topic
+);
+setProp(kafkaProps);
+setRecordTranslator(new 
SpoutRecordTranslator<>(FieldsConfiguration.toList(fieldsConfiguration)));
 this.topic = topic;
   }
 
+  private static  Class createDeserializer( 
Optional deserializerClass
--- End diff --

I'm not super keen on passing in possibly null values around.  I rather 
like the notion of passing in that something is optional in the type signature.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #584: METRON-950: Migrate storm-kafka-client to 1.1

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/584#discussion_r116564604
  
--- Diff: 
metron-platform/metron-storm-kafka-override/src/main/java/org/apache/storm/kafka/spout/internal/Timer.java
 ---
@@ -0,0 +1,58 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.storm.kafka.spout.internal;
+
+import java.util.concurrent.TimeUnit;
+import org.apache.storm.utils.Time;
+
+public class Timer {
+  private final long delay;
+  private final long period;
+  private final TimeUnit timeUnit;
+  private final long periodNanos;
+  private long start;
+
+  public Timer(long delay, long period, TimeUnit timeUnit) {
+this.delay = delay;
+this.period = period;
+this.timeUnit = timeUnit;
+this.periodNanos = timeUnit.toNanos(period);
+this.start = System.nanoTime() + timeUnit.toNanos(delay);
+  }
+
+  public long period() {
+return this.period;
+  }
+
+  public long delay() {
+return this.delay;
+  }
+
+  public TimeUnit getTimeUnit() {
+return this.timeUnit;
+  }
+
+  public boolean isExpiredResetOnTrue() {
--- End diff --

I'm not sure how complicated the original implementation is vs this one.  
Could we grab any unit tests on the original Timer and make sure that this 
Timer meets the same expectations?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #584: METRON-950: Migrate storm-kafka-client to 1.1

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/584#discussion_r116566613
  
--- Diff: 
metron-platform/metron-storm-kafka/src/main/java/org/apache/metron/storm/kafka/flux/SimpleStormKafkaBuilder.java
 ---
@@ -202,31 +240,4 @@ public String getTopic() {
 return new StormKafkaSpout<>(builder);
   }
 
-  private static Map modifyKafkaProps(Map 
props, String zkQuorum) {
-try {
-  if(!props.containsKey(KafkaSpoutConfig.Consumer.BOOTSTRAP_SERVERS)) {
-//this isn't a putIfAbsent because I only want to pull the brokers 
from zk if it's absent.
-List brokers = 
KafkaUtils.INSTANCE.getBrokersFromZookeeper(zkQuorum);
-props.put(KafkaSpoutConfig.Consumer.BOOTSTRAP_SERVERS, 
Joiner.on(",").join(brokers));
-  }
-  props.putIfAbsent(KafkaSpoutConfig.Consumer.KEY_DESERIALIZER, 
ByteArrayDeserializer.class.getName());
-  props.putIfAbsent(KafkaSpoutConfig.Consumer.VALUE_DESERIALIZER, 
ByteArrayDeserializer.class.getName());
-
-} catch (Exception e) {
-  throw new IllegalStateException("Unable to retrieve brokers from 
zookeeper: " + e.getMessage(), e);
-}
-return props;
-  }
-
-  private static  KafkaSpoutTuplesBuilder 
createTuplesBuilder(List config, String topic) {
-TupleBuilder tb =  new TupleBuilder(topic, 
FieldsConfiguration.toList(config));
-return new KafkaSpoutTuplesBuilderNamedTopics.Builder<>(tb).build();
-  }
-
-
-  private static KafkaSpoutStreams createStreams(List config, 
String topic) {
-final Fields fields = 
FieldsConfiguration.getFields(FieldsConfiguration.toList(config));
-return new KafkaSpoutStreamsNamedTopics.Builder(fields, STREAM, new 
String[] { topic} ).build();
--- End diff --

STREAM isn't used anymore, right? Can we kill the field (and no, I can't 
comment on the field because it's not technically part of the diff, sigh).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #584: METRON-950: Migrate storm-kafka-client to 1.1

[justinleet]

Github user justinleet commented on a diff in the pull request:

https://github.com/apache/metron/pull/584#discussion_r116568707
  
--- Diff: pom.xml ---
@@ -106,7 +122,7 @@
  but I justify it by noting that this should be able to be 
removed when we migrate to
  Storm 1.1.x, which properly supports Kafka 0.10.x.
   -->
-
1.0.1.2.5.0.0-1245
+1.1.0
--- End diff --

Couldn't the whole block above this be dropped/edited?  It's pure HDP 
profile now, right?  Or better yet, replaced with an explanation of the new 
1.1.0 + Timer explanation.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #589: METRON-955: Make the default sync policy for HDFS ...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/589

METRON-955: Make the default sync policy for HDFS Writer be based on the 
batch size

## Contributor Comments
Right now, we do a sync per record written in HDFS. This has performance 
penalties as it requires a Namenode contact. We get around this by letting the 
user define the sync policy in Flux, but that's less convenient and more 
sensible defaults could be created. We should use the batch size as a sensible 
default for the sync policy (i.e. sync every batch).

Testing plan pending

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [ ] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron hdfs_index_sync

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/589.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #589


commit 3fcb332c50cac72b5322d8c006028311246a42b9
Author: cstella 
Date:   2017-05-15T18:29:17Z

METRON-955: Make the default sync policy for HDFS Writer be based on the 
batch size




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #588: METRON-954: Create ability to change output topic of pars...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/588
  
Yep, testing being a main one.  YOu want to try out a parser without having 
it shove its data into `enrichment`


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #588: METRON-954: Create ability to change output topic of pars...

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/588
  
Casey, what is the use case for this?  Testing?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #588: METRON-954: Create ability to change output topic ...

[cestella]

GitHub user cestella opened a pull request:

https://github.com/apache/metron/pull/588

METRON-954: Create ability to change output topic of parsers from the CLI

## Contributor Comments
Right now, changing the output of the kafka topic for the parsers involves 
either adjusting source code or adjusting parser configs in zookeeper.  It 
seems sensible to add the output topic to the CLI params as a convenience 
approach.

To test this, 
* Create a topic called `my_topic` in kafka
* start up a parser topology of your choosing with `-ot my_topic`
* feed data into it
* use the console consumer to validate output is being piped into `my_topic`

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root incubating-metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/cestella/incubator-metron parser_output_topic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/588.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #588


commit 630727c9ce3e3deb92f21ae78742210810de0f48
Author: cstella 
Date:   2017-05-15T16:59:30Z

METRON-954: Create ability to change output topic of parsers from the CLI




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Remember remember the 29th of April — Matt’s instructions for handling the github rename

[Matt Foley]

So Apache infra confirmed that they use github’s “rename” capability, so no one 
needs to worry about losing connection with their remote repositories.  That 
means that doing any of the below is strictly voluntary, and B and D (changing 
upstream pointers) are completely unnecessary.

If you want to do A and C, renaming your personal repos as “metron” instead of 
“incubating-metron”, the below instructions are still valid.

Cheers,
--Matt

On 5/13/17, 5:55 AM, "Otto Fowler"  wrote:

Resending this, so that will land on top and be more findable for folks.


———
A brief guide to changing our github remotes from 
“apache/metron-incubating” to “apache/metron” 
[also applies to “incubator-metron-bro-plugin-kafka” → 
“metron-bro-plugin-kafka” ] 

Hey all, as a result of un-incubating, the github apache repo names will 
change. Some people have expressed concerns whether this will impact their 
development environments. This potentially affects four areas: 
A. Changing local repo names to match 
B. Changing local repos’ upstream pointers to point at the new apache repo 
names 
C. Changing the names of personal fork repos in Github to match 
D. Changing the upstream “source” apache repo that your personal fork repos 
in Github point at for purposes of Pull Requests 

‘B’ is the only important one, but we’ll talk about them in order. 

‘A’ is the least important. You can call a local repo anything you want, 
because the name of the local clone is just a directory name, and not strictly 
part of the repository at all. If you want to change its name, just `mv` the 
base directory name. It doesn’t matter. You probably already call them 
something other than “incubator-metron” anyway :-) 

‘B’ may or may not be important, depending on how Apache INFRA goes about 
changing the name of the Github apache/metron mirror repo. If they use the 
standard Github owner’s command for renaming a repo, then Github maintains 
semantics for users of the prior name. Then, you wouldn’t have to do anything 
about the name change unless you wanted to. 

However, if they use a technique that doesn’t respect Github’s protocol (or 
if that protocol doesn’t work with mirror repos), or if you just want to keep 
things neat and avoid confusion, here is how you change local repos’ upstream 
pointers to point at the new apache repo names: 

1. cd to the base directory of the local git repo 
2. Give the command `git remote -v` 
3. You’ll probably see something like: 

apache  https://github.com/apache/incubator-metron.git (fetch) 
apache  https://github.com/apache/incubator-metron.git (push) 
origin  https://github.com/mattf-horton/incubator-metron.git (fetch) 
origin  https://github.com/mattf-horton/incubator-metron.git (push) 

4. In this case, you could give the command: 
`git remote set-url apache https://github.com/apache/metron.git` 
5. Now the command `git remote -v` will show: 

apache  https://github.com/apache/metron.git (fetch) 
apache  https://github.com/apache/metron.git (push) 
origin  https://github.com/mattf-horton/incubator-metron.git (fetch) 
origin  https://github.com/mattf-horton/incubator-metron.git (push) 

and thereafter, every reference to the remote named “apache” would refer to 
apache/metron instead of apache/incubator-metron. 

‘C’ is if you want to rename your personal fork repositories in Github. 
Again it doesn’t matter that much, but you may wish to do so for appearance’ 
sake, or to avoid confusion. Instructions for renaming a personal repo are 
here: https://help.github.com/articles/renaming-a-repository/ . (Note you must 
have “owner” privs for the repo.) The instructions say that if you follow these 
instructions, and don’t create a new repo with the old name, then Github will 
maintain semantics for users of the prior name, so you don’t have to do any 
other adjustments. Nevertheless, you would then probably want to do something 
like this command within the above example repo: 
`git remote set-url origin https://github.com/mattf-horton/metron.git` 
Thereafter references to “origin” in your local repo would refer directly 
to your renamed fork repo in Github. 

‘D’ finally is perhaps the most difficult question. Every fork repo in 
Github “knows” what repo it was forked from, and uses that as the default for 
Pull Requests generated from the fork. Can this be changed? 

If Apache INFRA respects the Github protocol for repo renaming, then we 
don’t need to change anything. We can just rely on the semantic preservation in 
the protocol and new PRs generated will automatically point at the renamed 
Apache repository, directly or indirectly. That would be best, because Github 
does NOT provide any easy means for changing the “fork upstream” pointer. 

If the Github protocol is 

[GitHub] metron issue #481: METRON-322 Global Batching and Flushing

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/481
  
This is simply waiting for me to have time to return to it.  Would like to 
leave it open, if agreeable to community.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #425: METRON-609 Enhance Mpack to handle single-node and small-...

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/425
  
Withdrawn.  Much of this has been incorporated in other PRs already 
accepted.  The remaining part is in METRON-865, formerly a sub-task of 
METRON-609, but now raised to a regular Bug jira ticket.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #408: METRON-608 Mpack to install a single-node test cluster

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/408
  
Withdrawn.  The current installers adequately install a single node system.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #408: METRON-608 Mpack to install a single-node test clu...

[mattf-horton]

Github user mattf-horton closed the pull request at:

https://github.com/apache/metron/pull/408


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Infrastructure migrated

[Kyle Richardson]

Thanks, Casey!

-Kyle

On Mon, May 15, 2017 at 10:02 AM, Otto Fowler 
wrote:

> Thanks Casey.
>
>
> On May 15, 2017 at 09:51:21, Casey Stella (ceste...@gmail.com) wrote:
>
> It appears that committers who are not PMC members did not have their
> committer status migrated over to the new repo. I have filed an INFRA
> ticket to correct this: https://issues.apache.org/jira/browse/INFRA-14159
>
> I have called out certain people that I remembered. If you are not on the
> list in the JIRA, please let me know and I'll add you.
>
> On Mon, May 15, 2017 at 9:30 AM, Casey Stella  wrote:
>
> > For those who haven't noticed, the git infrastructure has migrated over.
> > If you are a committer, you will need to adjust the URLs for the apache
> > remote repo via:
> >
> > git remote set-url apache https://git-wip-us.apache.org/
> > repos/asf/metron.git
> >
> > Obviously substitute "apache" for "upstream" if you choose that
> > nomenclature, but the general change is the URL of the apache repo
> > changed. I just tried and commits worked. If they do not work for you
> > upon the next commit, let me know and I'll look into it.
> >
> > Best,
> >
> > Casey
> >
>

Re: Infrastructure migrated

[Casey Stella]

It appears that committers who are not PMC members did not have their
committer status migrated over to the new repo.  I have filed an INFRA
ticket to correct this: https://issues.apache.org/jira/browse/INFRA-14159

I have called out certain people that I remembered.  If you are not on the
list in the JIRA, please let me know and I'll add you.

On Mon, May 15, 2017 at 9:30 AM, Casey Stella  wrote:

> For those who haven't noticed, the git infrastructure has migrated over.
> If you are a committer, you will need to adjust the URLs for the apache
> remote repo via:
>
> git remote set-url apache https://git-wip-us.apache.org/
> repos/asf/metron.git
>
> Obviously substitute "apache" for "upstream" if you choose that
> nomenclature, but the general change is the URL of the apache repo
> changed.  I just tried and commits worked.  If they do not work for you
> upon the next commit, let me know and I'll look into it.
>
> Best,
>
> Casey
>

[GitHub] metron issue #586: METRON-508 Expand Elasticsearch templates to support the ...

[JonZeolla]

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/586
  
I updated the instructions to reflect the repo name change; hopefully it 
should work but I won't have a chance to test it out for a couple of days.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Infrastructure migrated

[Casey Stella]

For those who haven't noticed, the git infrastructure has migrated over.
If you are a committer, you will need to adjust the URLs for the apache
remote repo via:

git remote set-url apache https://git-wip-us.apache.org/repos/asf/metron.git

Obviously substitute "apache" for "upstream" if you choose that
nomenclature, but the general change is the URL of the apache repo
changed.  I just tried and commits worked.  If they do not work for you
upon the next commit, let me know and I'll look into it.

Best,

Casey

[GitHub] metron pull request #587: METRON-952 Travis CI Link in README Pointing to Ol...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/587


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #587: METRON-952 Travis CI Link in README Pointing to Old Repo

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/587
  
I'll create a ticket and take care of it.  I have a lot of this already 
done, while I was looking through last night.  I was hoping to get a 
comprehensive ticket out with this change, but I'm not really worried about 
having this separate.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #587: METRON-952 Travis CI Link in README Pointing to Old Repo

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/587
  
agreed; there is more to this than just this PR.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron issue #587: METRON-952 Travis CI Link in README Pointing to Old Repo

[justinleet]

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/587
  
I actually also noticed this last night, and was about to put in a patch 
until I realized there's a good amount of documentation in general that needs 
to point to the newer repos, and our docs and so on shouldn't be referring to 
incubator-metron dir anymore and so on.

I'm fine with doing this separately (and am +1 by inspection, since I did 
this exercise last night), but it seems like we probably should do the rest of 
it also now that we're TLP and everything is setup.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] metron pull request #587: METRON-952 Travis CI Link in README Pointing to Ol...

[nickwallen]

GitHub user nickwallen opened a pull request:

https://github.com/apache/metron/pull/587

METRON-952 Travis CI Link in README Pointing to Old Repo

The Travis CI status link in the README is pointing to the old 
apache/incubator-metron repository.  I updated this point to apache/metron.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/nickwallen/incubator-metron METRON-952

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/587.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #587


commit 753d221e323a4d0ac67f07ac969bddc92fef926c
Author: Nick Allen 
Date:   2017-05-15T13:00:22Z

METRON-952 Travis CI Link in README Pointing to Old Repo




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: integration testing framework

[zeo...@gmail.com]

The standard has been centos6 for installing Metron up to this point.
There are some Ubuntu guides floating around as well.

Jon

On Mon, May 15, 2017, 8:07 AM moshe jarusalem  wrote:

> I would like to ask another question related to this topic.
> If I am going to install metron on a single machine (the same machine as
> development) which operating distribution would be best centos7 or ubuntu
> 16.4 or another?
> And is there any document to follow for such an installation ?
>
> Regards,
>
> On Sat, May 13, 2017 at 1:50 PM, moshe jarusalem  wrote:
>
> > Hi All,
> > I have been looking for an easy way to test metron in the development
> > environment. Using "full-dev-environment" is a bit painful because you
> each
> > time need to copy artifacts etc. I tried to understand integration
> testing
> > framework but it a little complex for a newbie.
> >
> > Would you guys describe how to utilize it? broadly how to quickly write
> > and test? How do you manage?
> >
> > Regards,
> >
>
-- 

Jon

Re: integration testing framework

[moshe jarusalem]

I would like to ask another question related to this topic.
If I am going to install metron on a single machine (the same machine as
development) which operating distribution would be best centos7 or ubuntu
16.4 or another?
And is there any document to follow for such an installation ?

Regards,

On Sat, May 13, 2017 at 1:50 PM, moshe jarusalem  wrote:

> Hi All,
> I have been looking for an easy way to test metron in the development
> environment. Using "full-dev-environment" is a bit painful because you each
> time need to copy artifacts etc. I tried to understand integration testing
> framework but it a little complex for a newbie.
>
> Would you guys describe how to utilize it? broadly how to quickly write
> and test? How do you manage?
>
> Regards,
>