[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/467 There are two things here I *think* 1. The test assumptions and the failure on some platforms ( if this is indeed the problem ) 2. What this means for the real use of the

[GitHub] incubator-metron issue #467: METRON-743: Sort the files when reading results...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/467 What are the performance penalties of doing this? Part of the discussion in [HADOOP-12009](https://issues.apache.org/jira/browse/HADOOP-12009) that lead to the

[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/466 [INFO] [INFO] Reactor Summary: [INFO] [INFO] Metron .

Re: Odd integration-test failures on Fedora/CentOS for RC5

METRON-743 (https://github.com/apache/incubator-metron/pull/467) for reference. On Sat, Feb 25, 2017 at 11:51 PM, Casey Stella wrote: > Hmm, that's a very good catch if it's the issue. I was able to verify > that if you botch the sort order of the files that it fails. > >

[GitHub] incubator-metron pull request #467: METRON-743: Sort the files when reading ...

GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/467 METRON-743: Sort the files when reading results from Pcap The FileSystem.listFiles() call does not return the files in sorted order, which we assume for all FileSystem implementations.

[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/466 It shouldn't because the tests are already run with the code submitted in this PR. This is just about getting git in sync with what's generated. --- If your project is set up for it,

[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/466 Does building or running the tests have anything to do with it? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If

Re: Odd integration-test failures on Fedora/CentOS for RC5

Hmm, that's a very good catch if it's the issue. I was able to verify that if you botch the sort order of the files that it fails. Would you mind sorting the files on PcapJob line 199 by filename? Something like Collections.sort(files, (o1,o2) -> o1.getName().compareTo(o2.getName())); I'm going

[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...

Github user cestella commented on the issue: https://github.com/apache/incubator-metron/pull/466 Yep --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if

[GitHub] incubator-metron issue #466: METRON-742: Generated code for profile window s...

Github user ottobackwards commented on the issue: https://github.com/apache/incubator-metron/pull/466 So - the test is - build without changing anything after clone, and there should not be shown in git status right? --- If your project is set up for it, you can reply to this email

Re: Files modified after building

Well, if you are going to be so quick, how can I make you wait? working on it. On February 25, 2017 at 23:39:15, Casey Stella (ceste...@gmail.com) wrote: Sorry, couldn't wait for monday. Fix is submitted for your review and heckling: https://github.com/apache/incubator-metron/pull/466 :) On

Re: Files modified after building

Sorry, couldn't wait for monday. Fix is submitted for your review and heckling: https://github.com/apache/incubator-metron/pull/466 :) On Sat, Feb 25, 2017 at 11:22 PM, Matt Foley wrote: > Well, the last three are in a path with “/generated/” in it. > > > On 2/25/17, 7:39 PM,

[GitHub] incubator-metron pull request #466: METRON-742: Generated code for profile w...

GitHub user cestella opened a pull request: https://github.com/apache/incubator-metron/pull/466 METRON-742: Generated code for profile window selector DSL did not get committed as part of METRON-690 Lest we be concerned this implies that the testing done on 690 is voided, this was

Re: Odd integration-test failures on Fedora/CentOS for RC5

/** * List the statuses and block locations of the files in the given path. * Does not guarantee to return the iterator that traverses statuses * of the files in a sorted order. * * If the path is a directory, * if recursive is false, returns files in the directory; * if recursive is

Re: Files modified after building

Well, the last three are in a path with “/generated/” in it. On 2/25/17, 7:39 PM, "Casey Stella" wrote: Crap, those are generated by the new profile selector dsl committed Friday. I must've missed them on a commit on the PR. They are generated by the build, so

Re: Odd integration-test failures on Fedora/CentOS for RC5

https://issues.apache.org/jira/browse/HADOOP-12009 makes it seem like there is no order On February 25, 2017 at 23:06:37, Otto Fowler (ottobackwa...@gmail.com) wrote: Maybe Hadoop Local FileSystem returns different things from ListFiles() on different platforms? That would be something to

Re: Odd integration-test failures on Fedora/CentOS for RC5

Maybe Hadoop Local FileSystem returns different things from ListFiles() on different platforms? That would be something to check? Sorry that is all I got right now On February 25, 2017 at 22:57:49, Otto Fowler (ottobackwa...@gmail.com) wrote: There are also some if Log.isDebugEnabled()

Re: Odd integration-test failures on Fedora/CentOS for RC5

There are multiple “tests” within the test, with different parameters. If you look at where this is breaking, it is at { //make sure I get them all. Iterable results = job.query(new Path(outDir.getAbsolutePath()) , new Path(queryDir.getAbsolutePath())

Re: Files modified after building

Crap, those are generated by the new profile selector dsl committed Friday. I must've missed them on a commit on the PR. They are generated by the build, so it was factored into the tests and such for the PR. Sorry for the inconvenience, I'll have to make another PR on Monday to get them in. On

Re: Files modified after building

That's my guess. I noticed the same thing. -Kyle > On Feb 25, 2017, at 8:31 PM, Otto Fowler wrote: > > Changes not staged for commit: > > (use "git add ..." to update what will be committed) > > (use "git checkout -- ..." to discard changes in working directory) >

Re: Odd integration-test failures on Fedora/CentOS for RC5

mvn integration-test Although I have also tried... mvn clean install && mvn integration-test mvn clean package && mvn integration-test mvn install && mvn surefire-test@unit-tests && mvn surefire-test@integration-tests -Kyle > On Feb 25, 2017, at 8:34 PM, Otto Fowler

Re: Odd integration-test failures on Fedora/CentOS for RC5

What command are you using to build? On February 25, 2017 at 17:40:20, Kyle Richardson (kylerichards...@gmail.com) wrote: Tried with Oracle JDK and got the same result. I went as far as trying to run it through the debugger but am not that familiar with this part of the code. The timestamps of

Files modified after building

Changes not staged for commit: (use "git add ..." to update what will be committed) (use "git checkout -- ..." to discard changes in working directory) modified: metron-analytics/metron-profiler-client/src/main/java/Window.tokens modified:

Re: Odd integration-test failures on Fedora/CentOS for RC5

Tried with Oracle JDK and got the same result. I went as far as trying to run it through the debugger but am not that familiar with this part of the code. The timestamps of the packets are definitely not coming back in the expected order, but I'm not sure why. Could it be related to something

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

16GB. It's a MBP from a few years back. I'm getting the same result as most it seems, currently oversubscribed but doable with care, works way better with the Ansible/Ambari integration that I'm doing for METRON-671. -D... On Sat, Feb 25, 2017 at 2:13 PM, Matt Foley wrote: >

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

David, how much RAM do you have in the test system where you run full dev? On 2/25/17, 8:46 AM, "David Lyle" wrote: +1 (binding) Checked signatures Tests passed Full dev ran sucessfully-ish (bit more below, tl;dr- it's ugly, but much better after

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

+1 (binding) Checked signatures Tests passed Full dev ran sucessfully-ish (bit more below, tl;dr- it's ugly, but much better after METRON-671) Data flowed into the index/dashboard/hdfs On full-dev: Monit got into a state where it started killing topologies because the Storm cli was non-responsive

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

I'm running it up now. I don't think it should affect the release, but it's extremely important that full-dev work. It's how quick-dev is created. Any notion that it's been deprecated is incorrect. Fwiw, I've been working with it for the last few weeks as part of METRON-671 and it's been

Re: Travis CI Changes have broken my heart..... or my builds

Of course the same commands build locally On February 25, 2017 at 09:17:50, Otto Fowler (ottobackwa...@gmail.com) wrote: https://s3.amazonaws.com/archive.travis-ci.org/jobs/205276680/log.txt On February 25, 2017 at 08:20:54, Casey Stella (ceste...@gmail.com) wrote: I have not seen those "no

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

I see this with full dev on my laptop pretty much every time. I think it is a resource issue. I see not enough memory errors trying to start things. On February 25, 2017 at 09:15:15, Ryan Merriman (merrim...@gmail.com) wrote: When I go to Ambari to ensure the services are all up, HDFS is

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

When I go to Ambari to ensure the services are all up, HDFS is down. I tried it 4 or 5 times and got the same result each time. I've seen others validate with quick dev so I assumed full dev was not used anymore. I'll spin it up this morning and get a more detailed error. Is anyone else

Odd integration-test failures on Fedora/CentOS for RC5

Ok, I've tried this so many times I may be going crazy, so thought I'd ask the community for a sanity check. I'm trying to verify RC5 and I keep running into the same integration test failures but only on my Fedora (24 and 25) and CentOS 7 systems. It passes fine on my Macbook. It always fails

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

What exactly are the errors that you saw, Ryan? On Sat, Feb 25, 2017 at 07:31 David Lyle wrote: > Is there any reason full dev shouldn't be working? > > On Fri, Feb 24, 2017 at 9:19 PM, Casey Stella wrote: > > > Sounds like a good idea to me; thanks

Re: Travis CI Changes have broken my heart..... or my builds

I have not seen those "no output received for 10m errors before. Can you change the Travis command to not have -q for maven so we can see more context? On Sat, Feb 25, 2017 at 07:56 Otto Fowler wrote: > I have not had a build work on Travis CI ( linked to my fork ) in 4

Re: [VOTE] Releasing Apache Metron (incubating) 0.3.1-RC5

Is there any reason full dev shouldn't be working? On Fri, Feb 24, 2017 at 9:19 PM, Casey Stella wrote: > Sounds like a good idea to me; thanks Ryan! > On Fri, Feb 24, 2017 at 21:11 Ryan Merriman wrote: > > > +1 binding > > > > Verified the signature >