[GitHub] metron pull request #900: METRON-1411: Fix sed command in Upgrading.md

[justinleet]

GitHub user justinleet opened a pull request:

https://github.com/apache/metron/pull/900

METRON-1411: Fix sed command in Upgrading.md

## Contributor Comments
The sed commands in Upgrading.md for the alert field can be problematic on 
some versions of sed, including what's on Vagrant if you ran the instructions 
from there.

The commands will fail with a file not found error as-is (and can be tested 
by running the old version).  When running the new versions, the intended 
result is correct: The wrapping chunk with the template name is removed, and 
the alerts field is added.  Removing the wrapping chunk is to make it possible 
to resubmit to ES (Because it doesn't give an immediately resubmittable 
version), and the alert field is for querying.

This can be tested by spinning up full dev, running the old versions and 
seeing the error, followed by running the new version successfully.

## Pull Request Checklist

Thank you for submitting a contribution to Apache Metron.  
Please refer to our [Development 
Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235)
 for the complete guide to follow for contributions.  
Please refer also to our [Build Verification 
Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview)
 for complete smoke testing guides.  


In order to streamline the review of the contribution we ask you follow 
these guidelines and ask you to double check the following:

### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [x] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [x] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```

 Note:
Please ensure that once the PR is submitted, you check travis-ci for build 
issues and submit an update to your PR as soon as possible.
It is also recommended that [travis-ci](https://travis-ci.org) is set up 
for your personal repository such that your branches are built there before 
submitting a pull request.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/justinleet/metron upgrading_fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/900.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #900


commit 128e15fdad8016a9dc7578c0c1868fbd919c9cf9
Author: justinjleet 
Date:   2018-01-18T22:00:54Z

Removing extraneous single quotes




---

[GitHub] metron issue #891: METRON-1282 creating a new topic using the rest end point...

[merrimanr]

Github user merrimanr commented on the issue:

https://github.com/apache/metron/pull/891
  
I think a separate PR is fine.  Can you at least move it to a separate 
method? 


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/687
  
So, I was wrong when I initially looked at this PR, this adds variable 
updates to the resolver.  Should we perhaps separate parsing assignments (e.g. 
move `StellarAssignment.from` into the language like you've done here) from 
actually updating the variables.  I think I'd like to see Enrichments and the 
REPL refactored to not hand-code the variable update logic when we get around 
to updating the resolver to allow updates.  Thoughts?


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/687
  
@ottobackwards And yet this discussion primarily relates to whether the 
assignment operation as defined in Enrichment should be unified with the 
assignment operation being added to core stellar, presumably due to user demand.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
also @mattf-horton, those other things, to be generally useful and not 
restrictive and assumptive of the host's use cases would have to be generalized 
_so_ much, I think from an api pov would should wait until there is some demand.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
As far as `hosting`.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
So, I would say then that we restrict to ONLY core at the moment.  And then 
we start creating interfaces and other api surfaces ( most likely extracted 
from what we have ) as a follow on.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/687
  
@ottobackwards , that's inarguably true.  We can define the language to be 
just the Stellar Core.  We could also define it to include at least some of 
those very useful other things, which would be a whole lot more useful to 
users.  


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
the details of how we configure transforms and enrichments, and organize 
the stellar statements etc, are not required to implement or host stellar.


---

[GitHub] metron issue #899: METRON-1405: Add Boyer-Moore majority vote algorithm to S...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/899
  
Pulling this PR until I can resolve and underlying issue with the counting 
semantics, however I believe this may by theoretically impossible without a 
2-pass model.


---

[GitHub] metron pull request #899: METRON-1405: Add Boyer-Moore majority vote algorit...

[mmiklavc]

Github user mmiklavc closed the pull request at:

https://github.com/apache/metron/pull/899


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/687
  
> I think we need to keep everything in it's separate pile. We have people 
asking to host in not metron as recently as today.

My interpretation is that, in order to enable hosting Stellar outside of 
Metron, those items which ordinary users think of as "part of Stellar" but we 
have placed in five separate piles need to be united into one external-izable 
pile.  Surely it was Metron's intent to have one DSL, not five, right?  The 
fact the current implementation consists of five piles is an implementation 
detail.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
I vote to get rid of `:=` if we want to make it uniform.

I'll also straighten the scope stuff out etc. the whole nine yards if we 
want to do that.  I think that might make this a feature branch, but that would 
be good as it will be easier for @cestella or whomever wanted to contribute to 
get in on it.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/687
  
If one insists that ':=' in Enrichments is a Metron add-on not Stellar, 
that is equivalent to stating that Enrichment doesn't use Stellar but rather a 
custom DSL derived from Stellar.  The xml stuff wrapped around Stellar by 
Profiler, Enrichments, etc, can reasonably be viewed as external to Stellar, 
but the ':=' token is clearly part of the Enrichment DSL.  

The ':=' token is also part of the Stellar REPL, which again we've wrapped 
with caveats that it's got additions to Stellar that aren't part of Stellar.

Sure quacks like a duck to me, but if it's not a Duck then it's a non-Duck, 
according to Aristotle :-)

Fact is, to any user not immersed in implementation details, it's part of 
Stellar.  Why don't we be nice and make it all uniform?


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
the top level 'scope' being passed in kind of as it is now.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
the scope should be managed by the StellarExpression as a separate stack of 
linked scope nodes and not in the resolvers.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
@mattf-horton 
> As for := vs =, it's not so terrible to have both as exact synonyms. This 
gives you both your preferred aesthetics and backward compatibility.

Not to keep harping on it, but there is a difference between the Language 
and Stellar and the external constructs that metron has built around it.

`:=`  is an external thing that metron does, separate from the language or 
the engine.  It is coincidental that we use that same construct in the shell 
technically.

The way we have done the configurations, and structuring their logical 
meanings is also external to stellar, that is metron.

I think we need to keep everything in it's separate pile.  We have people 
asking to host in `not` metron as recently as today.




---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/687
  
@nickwallen Yes, you are absolutely correct.  Right now we do not have an 
approach to take a list of stellar expressions that are assignments, a variable 
resolver, a function resolver and execute that list of statements in a way that 
variables are updated.  At the time we punted on it due to limitations in the 
VariableResolver to update (also, it's not clear how to do that generally or 
even specifically in all circumstances).  All that being said, it might be nice 
to add a method to do that in `StellarAssignment` that presumes variable 
containers are maps.

Anyway, that aside, this PR seems to not *force* that change since it's 
really just moving variable assignment into the language and not updating the 
resolver explicitly.  I'd suggest we tackle that as a follow-on and make a 
stellar-generic approach to update variable resolvers with new variables.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/687
  
> @cestella: No, both the REPL and enrichments use the same method of 
assignment, the StellarAssignment class. It's just that assignment in stellar 
is available in multiple places.

Ah, yes, I am familiar with `StellarAssignment`.  Ok, but 
`StellarAssignment` just takes an expression as input and returns the variable 
and expression.  It gets us to the red zone, but not the end zone.

To really perform assignment, you still need to (1) execute the expression 
and (2) add the variable to a `VariableResolver`.  Those extra steps are 
performed currently in `AssignmentCommand` for the REPL and they must also be 
performed elsewhere for Enrichment, etc.

I was thinking that we would have all of those steps performed in the same 
place.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
I am not opposed to changing the resolvers, or continuing work as it comes 
out of this discussion


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
@nickwallen pending this discussion yes, it just adds the capability to the 
language and stellar common, but not to _*Metron*_.

As stated in the PR description:

> This PR is to get the concept as a proposal out for review and start the 
discussion.





---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/687
  
@nickwallen No, both the REPL and enrichments use the same method of 
assignment, the `StellarAssignment` class.  It's just that assignment in 
stellar is available in multiple places.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/687
  
> @ottobackwards : This is opt-in for the resolvers. The 
MapVariableResolver does NOT support update, so there is no change to 
Enrichment, Parsers, Profile, and Shell ( beyond what has been mentioned ).

If this PR does not change Parsers, Enrichment, Profiler, and the REPL, 
then what does it change?  Where can I use assignment based on this PR?

Does this just give us the 'capability' to add assignment to each of those? 
 We would then need to change the `VariableResolver`s used in each of those 
areas to use the assignment as you've implemented it here?



---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
OK,

The others can comment, but I think proper scope would be implemented 
within the execution of the call stack, not arbitrarily by the resolvers.  Each 
resolver should be a 'scope' and new scopes should be created.  

A.  Because it is a resolver thing, where would you document it in a way 
that makes sense?  I have been thinking that we should refactor to explicitly 
support scoping in the processor/evaluations and then have the resolvers _used_ 
in scope as scope implementations.  So the nesting of scopes would not be done 
in the resolver.

B.  As noted in the PR description, I was hoping for discussion ( THIS 
discussion as a matter of fact ).  Having had some issues with my other PRs and 
scope, I chose to explicitly NOT go that far until we talked it through.  I did 
not feel all the way back then that I had enough of a grasp of the different 
resolvers to change their behavior and understand the consequences either way.

C.  The LambaExpression actually delegates to the state variable resolver:

```java
  VariableResolver variableResolver = new DefaultVariableResolver(
variable -> lambdaVariables.getOrDefault(variable
, state.variableResolver.resolve(variable)
), variable -> true,
(variable, value) -> {
  if (state.variableResolver.exists(variable)) {
state.variableResolver.update(variable, value);
  }
});
```


D.   
- I think I answered above. 
- I *think* so, maybe @cestella  or @nickwallen  can answer that
- Again, I stopped short until I got some kind of feedback.
- I have changed StellarAssignment, I don't think we need both `=` and 
`:=`, and if there are `:=` floating around they will work, unless we remove 
StellarAssignment


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/687
  
> @cestella: No, that's not correct. Stellar enrichments can contain :=. 
Consider the enrichments here in one of our use-cases. Enrichments can either 
take a map of enrichments 

Thanks for clarifying, but 'ugh'. I'll have to look at how that is 
implemented. I didn't realize we had a separate implementation of assignment.  
Does this PR address that?  Shouldn't we have one implementation of the 
assignment mechanism?

Good commentary from @mattf-horton .  I know there is a reason why I didn't 
want to touch this. :)



---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/687
  
So a bunch more comments came in while I was writing the above.  Nick's 
comment about enrichments using assignment already of course modifies my 
comment about Profilers, etc.  Just making it uniform would give users a lot 
less to wonder about.

As for := vs =, it's not so terrible to have both as exact synonyms.  This 
gives you both your preferred aesthetics and backward compatibility.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[mattf-horton]

Github user mattf-horton commented on the issue:

https://github.com/apache/metron/pull/687
  
As a professed language nut, when you mention the word "variable", I think 
"scope".  And it seems to me that by placing the responsibility for variable 
assignment/update in the VariableResolver, you've actually tied into a very 
natural definition of scope given the way MapVariableResolver already works in 
association with a target dictionary or stack of dictionaries.

A. I think you should explicitly discuss scope in the documentation.  There 
are several scopes already being used in practice, and I think users need to 
understand how it all works together, otherwise they will misunderstand and be 
bewildered.  I am, no doubt, a good example, so dense that I have to ask the 
questions below :-) 

B. I don't understand why you don't go ahead and enable update in 
MapVariableResolver, since IIRC that's the resolver for essentially everything 
interesting that uses variables in Stellar.

C. Do I understand correctly from the patch, that variable assignment IS 
enabled in lambda expressions?  Presumably the scope will be the same instance 
dictionary where the input variables are stored, thus making them local, 
correct?  Can new variables be created as well as input variables be updated?

D. The following scopes come to mind, with associated need for 
clarification:
- Lambdas save their input variables in their ephemeral instance context, 
initialized by call syntax.
  - Comment: questions already asked above
- The REPL used one global dict for scope of its variables, defined by 
':='.  
  - Comment: If I understand correctly, when in the REPL, '=' and ':=' 
assignments will go to the same global context dictionary, is that correct? 
- Profilers, transformations, and several other xml-defined Stellar-using 
features, have a persistent instance context, with variables initialized and 
updated via an xml syntax.
  - Comment: Apparently here we've specifically exclude assignment, even 
though this seems to be where it's needed most.  Why is that?  The instance 
context seems ready-made for local variables.



---

Re: Some more upgrade fallout... Can't restart Metron Indexing

[Otto Fowler]

I assigned METRON-1410 to myself.
I will take a shot at addressing this in the ambari service code.



On January 18, 2018 at 13:03:18, Laurens Vets (laur...@daemon.be) wrote:

On 2018-01-18 09:14, Casey Stella wrote:
> So, the challenge here is that our install script isn't smart enough
> right
> now to skip creating tables that are already created. One thing you
> could
> do is
>
> 1. rename the hbase tables for metron (see
>
>
https://stackoverflow.com/questions/27966072/how-do-you-rename-a-table-in-hbase
> )
> 2. let the install create them anew
> 3. stop metron
> 4. delete the new empty hbase tables
> 5. swap in the old tables
> 6. start metron

This worked, thanks! I'll update
https://issues.apache.org/jira/browse/METRON-1410 as well.

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/687
  
@nickwallen No, that's not correct.  Stellar enrichments can contain `:=`.  
Consider the enrichments 
[here](https://github.com/apache/metron/tree/master/use-cases/geographic_login_outliers#enrich-authentication-events)
 in one of our use-cases.  Enrichments can either take a map of enrichments or 
a list of strings.  If you pass a list of strings (which you would need to do 
to create temporary variables that you then remove before passing through, like 
in the example), you'd use an assignment, which is currently `:=`.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
@cestella  @nickwallen 
I have changed stellar assignment to cover the := already.   I think it 
will continue to work.



---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/687
  
>  @cestella: What about users with existing stellar enrichments using := 
...

Oh, and one clarification.  No users will have production code (like an 
enrichment) that uses `:=`.  That has only ever been available in the REPL.  So 
that should not be a concern, right?  The only concern would be documented uses 
of the REPL itself; like READMEs. 


---

Re: [DISCUSS] Upgrading Solr

[Casey Stella]

+1 to both the feature branch and user@ announcement.

On Thu, Jan 18, 2018 at 2:45 PM, Otto Fowler 
wrote:

> +1 to the feature branch.
>
> Also, there have been some questions about solr support recently, I think
> when the feature branch
> is ready you should announce it on user@ too, we may get some help from
> folks looking for this.
>
>
>
> On January 18, 2018 at 14:26:14, Justin Leet (justinjl...@gmail.com)
> wrote:
>
> Now that we have ES at a modern version, we should consider bringing Solr
> to a modern version as well.
>
> The focus of this work would be to get us in a place where Solr is
> upgraded, along with the related work of building out the Solr
> functionality to parity with Elasticsearch. The goal would not be to add
> net new functionality, just to get Solr and ES in the same place for the
> alerts UI and REST interface. Additionally, it would include the various
> supporting necessities such as ensuring associated DAOs are testable, and
> so on.
>
> Given the testing, reviewing, and iteration involved, I'd like to propose
> doing this work in a feature in a feature branch.
>
> Jiras would be created based on this discussion once it dies down a bit.
>

Re: [DISCUSS] Upgrading Solr

[Otto Fowler]

+1 to the feature branch.

Also, there have been some questions about solr support recently, I think
when the feature branch
is ready you should announce it on user@ too, we may get some help from
folks looking for this.



On January 18, 2018 at 14:26:14, Justin Leet (justinjl...@gmail.com) wrote:

Now that we have ES at a modern version, we should consider bringing Solr
to a modern version as well.

The focus of this work would be to get us in a place where Solr is
upgraded, along with the related work of building out the Solr
functionality to parity with Elasticsearch. The goal would not be to add
net new functionality, just to get Solr and ES in the same place for the
alerts UI and REST interface. Additionally, it would include the various
supporting necessities such as ensuring associated DAOs are testable, and
so on.

Given the testing, reviewing, and iteration involved, I'd like to propose
doing this work in a feature in a feature branch.

Jiras would be created based on this discussion once it dies down a bit.

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/687
  
hah, well @nickwallen, laziness is often a good trait in a developer.  
Honestly, I am ok with *not* migrating here in favor of more intuitive 
approaches.  I chose `:=` because I needed the functionality and worried about 
making the language more complex (there's something different about a language 
that has assignments versus an expression language.  I wanted to keep it an 
expression language at the time.), so I was forced to choose a character 
sequence that didn't exist as a lexable token.  Now, honestly, we're already 
halfway down that road, so probably the right thing to do *is* deprecate it and 
rip the bandaid fast.  I honestly could be convinced either way. 


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/687
  
> @cestella  Migration, sadly sometimes, is also part of user experience. :)

Very true.  That's why I offered up the ["Alternative 
Option"](https://github.com/apache/metron/pull/687#issuecomment-358754026).  

Only my own laziness makes me "mildly against" that  :) .  But based on 
user experience (like you mention), I think we should probably do it.






---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/687
  
Yeah, I'm totally cool with going with `=`, but I will point out one snag.  
What about users with existing stellar enrichments using `:=`, would we provide 
a migration path beyond saying, "Hey, move your stuff over" in the Upgrading 
docs?

Migration, sadly sometimes, is also part of user experience. :)


---

Re: [DISCUSS] Upgrading Solr

[Nick Allen]

+1 to a feature branch for this.  I like that approach Justin.

On Thu, Jan 18, 2018 at 2:26 PM Justin Leet  wrote:

> Now that we have ES at a modern version, we should consider bringing Solr
> to a modern version as well.
>
> The focus of this work would be to get us in a place where Solr is
> upgraded, along with the related work of building out the Solr
> functionality to parity with Elasticsearch. The goal would not be to add
> net new functionality, just to get Solr and ES in the same place for the
> alerts UI and REST interface.  Additionally, it would include the various
> supporting necessities such as ensuring associated DAOs are testable, and
> so on.
>
> Given the testing, reviewing, and iteration involved, I'd like to propose
> doing this work in a feature in a feature branch.
>
> Jiras would be created based on this discussion once it dies down a bit.
>

[DISCUSS] Upgrading Solr

[Justin Leet]

Now that we have ES at a modern version, we should consider bringing Solr
to a modern version as well.

The focus of this work would be to get us in a place where Solr is
upgraded, along with the related work of building out the Solr
functionality to parity with Elasticsearch. The goal would not be to add
net new functionality, just to get Solr and ES in the same place for the
alerts UI and REST interface.  Additionally, it would include the various
supporting necessities such as ensuring associated DAOs are testable, and
so on.

Given the testing, reviewing, and iteration involved, I'd like to propose
doing this work in a feature in a feature branch.

Jiras would be created based on this discussion once it dies down a bit.

[GitHub] metron pull request #894: METRON-1326: Metron deploy with Kerberos fails on ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/894


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[nickwallen]

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/687
  
I have not looked at the code yet, but wanted to comment on the assignment 
discussion.  IMO focusing on user experience is the way to go.  To that end...

* We should use `=` because that is just more obvious.
* We should remove `:=` from the REPL environment (very easy, just delete 
`AssignmentCommand`).
* All references in the docs to `:=` need to be changed to `=`. (Ugh)

All of that can happen in this PR or as multiple PRs (however 
@ottobackwards sees fit), but that is the right end state.

**Alternative Option**: The only other reasonable option IMO is to support 
both `=` and `:=` (for some undetermined period of time.)  The only snafu is 
that both should have the same implementation.  So support for `:=` would have 
to be added to the 'language' and we should NOT rely on the `AssignmentCommand` 
implementation in the REPL.

I am mildly against the "Alternative Option", but wanted to offer it up for 
discussion.


---

[GitHub] metron issue #894: METRON-1326: Metron deploy with Kerberos fails on Ambari ...

[anandsubbu]

Github user anandsubbu commented on the issue:

https://github.com/apache/metron/pull/894
  
+1 thank you @mmiklavc.

Spun up a 12-node Centos 7 cluster with the latest fix and validated the 
following:
* Did 'Stop All Services' -> no issues seen with ES and Kibana services 
stop any longer.
* Kerberized the cluster -> the process completed successfully and all 
services came up fine.

The issues reported in the JIRA ticket now stand resolved.


---

[GitHub] metron issue #894: METRON-1326: Metron deploy with Kerberos fails on Ambari ...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/894
  
I'm +1 on this at this point, but suggest we wait until @anandsubbu gives 
his +1.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
also, we aren't consistent, in some places ( transformations ) we use the 
configuration logical structure to imply assignment, in enrichment it is `:=`, 
but those are configuration mechanisms not stellar mechanisms


---

Re: Some more upgrade fallout... Can't restart Metron Indexing

[Laurens Vets]

On 2018-01-18 09:14, Casey Stella wrote:
So, the challenge here is that our install script isn't smart enough 
right
now to skip creating tables that are already created.  One thing you 
could

do is

   1. rename the hbase tables for metron (see

https://stackoverflow.com/questions/27966072/how-do-you-rename-a-table-in-hbase
   )
   2. let the install create them anew
   3. stop metron
   4. delete the new empty hbase tables
   5. swap in the old tables
   6. start metron


This worked, thanks! I'll update 
https://issues.apache.org/jira/browse/METRON-1410 as well.

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
By punt you mean leave this implementation that doesn't implement update in 
the resolvers?


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/687
  
Agreed, this is a POV issue.  Though from the user perspective they aren't 
distinguishing between "this is in the REPL" vs "this is in the enrichment 
topology", so I think we do need to manage the experience if we're moving 
functionality from the MACHINE to the LANGUAGE to ensure it's consistent.

Beyond that, it's unclear whether we need an resolver `update` function as 
part of THIS PR, but I think we very much *will* need it if we decide to make 
stellar multi-expression.  I suspect we should punt on it until that effort.


---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
POV issue I think.  Stellar has the LANGUAGE ( grammar ), the default 
implementation, and the 'host' execution mechanics.

`:=` is part of the host execution mechanics.

The issue may not be being the same as much as the necessity for `:=` as a 
host construct at all if things are done inside stellar.

Does that make sense?




---

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/687
  
well, `:=` is in the shell and also in the enrichment configuration (see 
[here](https://github.com/apache/metron/tree/master/metron-platform/metron-enrichment#stellar-enrichment-configuration).
  Seems like if we're promoting this to the language, we should make things 
consistent in one way or the other.


---

Re: Some more upgrade fallout... Can't restart Metron Indexing

[Otto Fowler]

JIRAS



On January 18, 2018 at 12:14:11, Casey Stella (ceste...@gmail.com) wrote:

So, the challenge here is that our install script isn't smart enough right
now to skip creating tables that are already created. One thing you could
do is

1. rename the hbase tables for metron (see
https://stackoverflow.com/questions/27966072/how-do-you-rename-a-table-in-hbase
)
2. let the install create them anew
3. stop metron
4. delete the new empty hbase tables
5. swap in the old tables
6. start metron

What we probably should do is not barf if the tables exist, but rather
warn.

On Thu, Jan 18, 2018 at 12:02 PM, Laurens Vets  wrote:

> After upgrading from 0.4.1 to 0.4.2, I can't seem to start or restart
> Metron Indexing. I get the following errors:
>
> stderr: /var/lib/ambari-agent/data/errors-2468.txt
>
> Traceback (most recent call last):
> File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/pa
> ckage/scripts/indexing_master.py", line 160, in 
> Indexing().execute()
> File
"/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",

> line 280, in execute
> method(env)
> File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/pa
> ckage/scripts/indexing_master.py", line 82, in start
> self.configure(env)
> File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/pa
> ckage/scripts/indexing_master.py", line 72, in configure
> commands.create_hbase_tables()
> File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/pa
> ckage/scripts/indexing_commands.py", line 126, in create_hbase_tables
> user=self.__params.hbase_user
> File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
> File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
> File
"/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
> File
"/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",

> line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
> File
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
> File
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
> File
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
> File
"/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'echo
> "create 'metron_update','t'" | hbase shell -n' returned 1. ERROR
> RuntimeError: Table already exists: metron_update!
>
> stdout: /var/lib/ambari-agent/data/output-2468.txt
>
> 2018-01-18 16:54:30,101 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2018-01-18 16:54:30,301 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2018-01-18 16:54:30,302 - Group['metron'] {}
> 2018-01-18 16:54:30,303 - Group['livy'] {}
> 2018-01-18 16:54:30,303 - Group['elasticsearch'] {}
> 2018-01-18 16:54:30,303 - Group['spark'] {}
> 2018-01-18 16:54:30,303 - Group['zeppelin'] {}
> 2018-01-18 16:54:30,304 - Group['hadoop'] {}
> 2018-01-18 16:54:30,304 - Group['kibana'] {}
> 2018-01-18 16:54:30,304 - Group['users'] {}
> 2018-01-18 16:54:30,304 - User['hive'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,305 - User['storm'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,306 - User['zookeeper'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,306 - User['infra-solr'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,307 - User['ams'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,307 - User['tez'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2018-01-18 16:54:30,308 - User['zeppelin'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,309 - User['metron'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,309 - User['livy'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,310 - User['elasticsearch'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,310 - User['spark'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,311 - User['ambari-qa'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2018-01-18 16:54:30,311 - User['flume'] {'gid': 

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
@cestella also:  := is in the shell not the language.  The mechanics of 
what is LANG and what is MACHINE is part of the issue.


---

Re: Some more upgrade fallout... Can't restart Metron Indexing

[Casey Stella]

So, the challenge here is that our install script isn't smart enough right
now to skip creating tables that are already created.  One thing you could
do is

   1. rename the hbase tables for metron (see
   
https://stackoverflow.com/questions/27966072/how-do-you-rename-a-table-in-hbase
   )
   2. let the install create them anew
   3. stop metron
   4. delete the new empty hbase tables
   5. swap in the old tables
   6. start metron

What we probably should do is not barf if the tables exist, but rather warn.

On Thu, Jan 18, 2018 at 12:02 PM, Laurens Vets  wrote:

> After upgrading from 0.4.1 to 0.4.2, I can't seem to start or restart
> Metron Indexing. I get the following errors:
>
> stderr:   /var/lib/ambari-agent/data/errors-2468.txt
>
> Traceback (most recent call last):
>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/pa
> ckage/scripts/indexing_master.py", line 160, in 
> Indexing().execute()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/libraries/script/script.py",
> line 280, in execute
> method(env)
>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/pa
> ckage/scripts/indexing_master.py", line 82, in start
> self.configure(env)
>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/pa
> ckage/scripts/indexing_master.py", line 72, in configure
> commands.create_hbase_tables()
>   File "/var/lib/ambari-agent/cache/common-services/METRON/0.4.2/pa
> ckage/scripts/indexing_commands.py", line 126, in create_hbase_tables
> user=self.__params.hbase_user
>   File "/usr/lib/python2.6/site-packages/resource_management/core/base.py",
> line 155, in __init__
> self.env.run()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 160, in run
> self.run_action(resource, action)
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/environment.py",
> line 124, in run_action
> provider_action()
>   File 
> "/usr/lib/python2.6/site-packages/resource_management/core/providers/system.py",
> line 273, in action_run
> tries=self.resource.tries, try_sleep=self.resource.try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 70, in inner
> result = function(command, **kwargs)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 92, in checked_call
> tries=tries, try_sleep=try_sleep)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 140, in _call_wrapper
> result = _call(command, **kwargs_copy)
>   File "/usr/lib/python2.6/site-packages/resource_management/core/shell.py",
> line 293, in _call
> raise ExecutionFailed(err_msg, code, out, err)
> resource_management.core.exceptions.ExecutionFailed: Execution of 'echo
> "create 'metron_update','t'" | hbase shell -n' returned 1. ERROR
> RuntimeError: Table already exists: metron_update!
>
> stdout:   /var/lib/ambari-agent/data/output-2468.txt
>
> 2018-01-18 16:54:30,101 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2018-01-18 16:54:30,301 - Using hadoop conf dir:
> /usr/hdp/current/hadoop-client/conf
> 2018-01-18 16:54:30,302 - Group['metron'] {}
> 2018-01-18 16:54:30,303 - Group['livy'] {}
> 2018-01-18 16:54:30,303 - Group['elasticsearch'] {}
> 2018-01-18 16:54:30,303 - Group['spark'] {}
> 2018-01-18 16:54:30,303 - Group['zeppelin'] {}
> 2018-01-18 16:54:30,304 - Group['hadoop'] {}
> 2018-01-18 16:54:30,304 - Group['kibana'] {}
> 2018-01-18 16:54:30,304 - Group['users'] {}
> 2018-01-18 16:54:30,304 - User['hive'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,305 - User['storm'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,306 - User['zookeeper'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,306 - User['infra-solr'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,307 - User['ams'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,307 - User['tez'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2018-01-18 16:54:30,308 - User['zeppelin'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,309 - User['metron'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,309 - User['livy'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,310 - User['elasticsearch'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,310 - User['spark'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['hadoop']}
> 2018-01-18 16:54:30,311 - User['ambari-qa'] {'gid': 'hadoop',
> 'fetch_nonlocal_groups': True, 'groups': ['users']}
> 2018-01-18 

Re: Travis for Apache/Metron is in trouble

[Casey Stella]

I made an infra ticket: https://issues.apache.org/jira/browse/INFRA-15865

On Thu, Jan 18, 2018 at 11:42 AM, Otto Fowler 
wrote:

> 24hr long build is blocking up master’s travis build.
> Who can nuke it?
>
> ottO
>

[GitHub] metron issue #687: METRON-1090 Add Assignment to Stellar Language

[ottobackwards]

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/687
  
I just removed the conflicts and changed StellarAssignment to account for 
this


---