[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350575#comment-16350575
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/920


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350434#comment-16350434
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165663581
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

Right now, I never let my IDE reformat for me.  Like you said, if we get 
the code base matching check style and I can load that style into my IDE, then 
I'd gladly let it do most of the work for me.

Maybe I'll open a discuss thread.  I don't know how to handle this kind of 
thing and it happens all the time.

But for this specific scenario in your PR, it really doesn't matter either 
way.  I think you're good to go either way.


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350435#comment-16350435
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/920
  
+1 Thanks @ottobackwards 


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350409#comment-16350409
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165660358
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
 ---
@@ -36,8 +37,8 @@
  *
  */
 public class PausableInput extends InputStream {
-  InputStream in = System.in;
-  boolean paused = false;
+  private InputStream in = System.in;
+  private AtomicBoolean paused = new AtomicBoolean(false);
--- End diff --

Good find!  I'm sure that was frustrating to dig into.


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350422#comment-16350422
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165662017
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

How do you have your formatting preferences set to get the above?


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350419#comment-16350419
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165661676
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

I didn't reformat that like that on purpose.  I had a period where I was 
setting the CONSOLE capability.  When I removed it, it just worked out like 
this.

Even if I select and format in intellij it doesn't change the .build() to 
the next line.

I think what you have above is fine.  I would like it to 'just' happen if I 
format code though, since it is easy for things to slip through.

It is tough right now, because so much of the codebase isn't formatted to 
check style, and I don't think we want every pr to include a lot of formatting 
changes.



> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350404#comment-16350404
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165659854
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
 ---
@@ -36,8 +37,8 @@
  *
  */
 public class PausableInput extends InputStream {
-  InputStream in = System.in;
-  boolean paused = false;
+  private InputStream in = System.in;
+  private AtomicBoolean paused = new AtomicBoolean(false);
--- End diff --

I found in travis and locally, that the PausableInput was hanging, my 
builds where not completing.  PausableInput.INSTANCE.unpause(); was never 
returning.

I couldn't figure out why it was locked.  Reading the class documentation 
and the trying to understand the threading in the shell, I looked at the class 
and saw some inconsistencies with how and when we checked if we where paused in 
the different read() calls, and also that the flag field was not volatile or 
atomic.  I refactored this things to what I *think* would be the correct and 
consistent approaches and  it resolved my issue.


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350380#comment-16350380
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165654991
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/cli/PausableInput.java
 ---
@@ -36,8 +37,8 @@
  *
  */
 public class PausableInput extends InputStream {
-  InputStream in = System.in;
-  boolean paused = false;
+  private InputStream in = System.in;
+  private AtomicBoolean paused = new AtomicBoolean(false);
--- End diff --

What problem were you solving here @ottobackwards ?  Is this bit access by 
multiple threads?


> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16350379#comment-16350379
 ] 

ASF GitHub Bot commented on METRON-1438:


Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/920#discussion_r165653517
  
--- Diff: 
metron-stellar/stellar-common/src/test/java/org/apache/metron/stellar/dsl/functions/ShellFunctionsTest.java
 ---
@@ -40,8 +45,8 @@
   );
 
   Context context = new Context.Builder()
-.with(Context.Capabilities.SHELL_VARIABLES , () -> variables)
-.build();
+.with(Context.Capabilities.SHELL_VARIABLES , () -> 
variables).build();
--- End diff --

I am actually interested in what direction as a project we should be taking 
with these types of fluent, chained statements.  I run across this all the time 
and I want to know the 'right' way that I should be doing it for the project.

IMHO, the way it was (separated by a line break) is more readable.  
Meaning, a long set of chained statements should be separated by line breaks.  
For example...
```
  result = new ProfileMeasurement()
  .withProfileName(profileName)
  .withEntity(entity)
  .withGroups(groups)
  .withPeriod(period)
  .withProfileValue(profileValue)
  .withTriageValues(triageValues)
  .withDefinition(definition);
```

But, of course, in terms of code style my opinion doesn't matter.  It is 
all about our style guidelines. 
 What does the Google code style guidelines say?   

Doesn't 
[this](https://google.github.io/styleguide/javaguide.html#s4.5.1-line-wrapping-where-to-break)
 support what I have said above about line breaks in this case?  




> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1438) STELLAR: Move shell functions to common from metron-management

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/METRON-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16349059#comment-16349059
 ] 

ASF GitHub Bot commented on METRON-1438:


GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/920

METRON-1438 Move SHELL functions from metron-management to stellar-common

Part of making stellar more modular and stand alone is consolidation of 
stellar functionality scattered through metron into stellar common.   That 
should be done if the functionality is generally useful and not metron 
specific. 

Where the functionality *is* metron specific, we should look to factor such 
functionality as to extract generally useful functionality.

In the case of the SHELL- namespace, it is clearly not metron specific.

This pr moves this functionality from the metron-management module to 
stellar common.

## Testing
- build and tests should run as normal
- shell functions such as those in the metron-management readme examples 
should run
- functions should work as before


### For all changes:
- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [x] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && 
dev-utilities/build-utils/verify_licenses.sh 
  ```

- [na] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [na] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [na] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron stellar-move-shell-funcs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/metron/pull/920.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #920


commit fe93fc40d94cc6148667eccc36853a2dc9925546
Author: Otto Fowler 
Date:   2018-02-01T14:28:46Z

move ShellFunctions to stellar-common

commit 38775033240c7c174c99b10ffb5087c39b23672a
Author: Otto Fowler 
Date:   2018-02-01T15:33:42Z

cleanup and refactor for failed tests where PausableInput did not unpause

commit 7af3d347567fe967c9c2ea1191c121a470d2bf34
Author: Otto Fowler 
Date:   2018-02-01T18:28:43Z

fix readme




> STELLAR: Move shell functions to common from metron-management
> --
>
> Key: METRON-1438
> URL: https://issues.apache.org/jira/browse/METRON-1438
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)