date:20171217

[GitHub] metron pull request #871: METRON-1365: Allow PROFILE_GET to return a default...

2017-12-17 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/871#discussion_r157390830
  
--- Diff: 
metron-analytics/metron-profiler-client/src/test/java/org/apache/metron/profiler/client/stellar/GetProfileTest.java
 ---
@@ -310,6 +306,36 @@ public void testOutsideTimeHorizon() {
 Assert.assertEquals(0, result.size());
   }
 
+  /**
+   * Default value should be able to be specified
+   */
+  @Test
+  public void testWithDefaultValue() {
+String expr = "PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(4, 
'HOURS'))";
+@SuppressWarnings("unchecked")
+List result = run(expr, List.class);
+
+// validate - expect to fail to read any values because we didn't 
write any.
+Assert.assertEquals(0, result.size());
+
+// execute - read the profile values - with config_override.
+// first two override values are strings, third is deliberately a 
number.
+testOverride("{'profiler.default.value' : 0}", 0);
+testOverride("{'profiler.default.value' : 'metron'}", "metron");
+testOverride("{'profiler.default.value' : []}", new ArrayList<>());
+  }
+
+  private void testOverride(String overrides, Object defaultVal) {
+  String expr = "PROFILE_GET('profile1', 'entity1', PROFILE_FIXED(4, 
'HOURS'), [], " + overrides + ")"
+  ;
--- End diff --

Small nit... semi on a new line?


---

[GitHub] metron pull request #871: METRON-1365: Allow PROFILE_GET to return a default...

2017-12-17 Thread nickwallen

Github user nickwallen commented on a diff in the pull request:

https://github.com/apache/metron/pull/871#discussion_r157390718
  
--- Diff: 
metron-analytics/metron-profiler-client/src/main/java/org/apache/metron/profiler/client/stellar/ProfilerClientConfig.java
 ---
@@ -56,8 +56,11 @@
   /**
* A global property that defines the salt divisor used to store profile 
data.
*/
-  PROFILER_SALT_DIVISOR("profiler.client.salt.divisor", 1000L, Long.class);
-
+  PROFILER_SALT_DIVISOR("profiler.client.salt.divisor", 1000L, Long.class)
+  /**
+   * The default value to be returned if a profile is not written for a 
given period for a profile and entity.
+   */
+  ,PROFILER_DEFAULT_VALUE("profiler.default.value", null, Object.class);
--- End diff --

Small nits... Add a space above this field and move the comma so that it 
matches the rest of the file


---

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

2017-12-17 Thread JonZeolla

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157385625
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarExecutor.java
 ---
@@ -289,7 +293,25 @@ public Object execute(String expression) {
 VariableResolver variableResolver = new 
MapVariableResolver(Maps.transformValues(variables, result -> 
result.getResult())
, 
Collections.emptyMap());
 StellarProcessor processor = new StellarProcessor();
-return processor.parse(expression, variableResolver, functionResolver, 
context);
+StackWatch watch = new StackWatch("execute");
+watch.startTime(expression);
+context.setWatch(watch);
+try {
+  return processor.parse(expression, variableResolver, 
functionResolver, context);
+} finally {
+  watch.stopTime();
+  final StringBuffer buff = new StringBuffer();
+  watch.visit(((level, node) -> {
+for (int i = 0; i < level; i++) {
+  buff.append("-");
+}
+buff.append("->");
+buff.append(node.getName()).append(" : 
").append(node.getTime()).append("ms : ").
+append(node.getNanoTime()).append("ns").append("\n");
+  }));
+  lastTiming = Optional.of(buff.toString());
+  context.clearWatch();
+}
--- End diff --

That makes sense, just looking to limit the effect here.


---

Re: UPDATE MEETING December Developer Community Meeting

2017-12-17 Thread Nadir Hajiyani

Aah, just noticed it in the middle of the email - 12.30 pm EST - hit send
too early.

Thanks.

On Sun, Dec 17, 2017 at 2:54 PM, Nadir Hajiyani 
wrote:

> Hi,
> What time is this meeting at?
>
> Thanks.
>
> On Sun, Dec 17, 2017 at 1:32 PM, Otto Fowler 
> wrote:
>
>> We will be using this meeting
>>
>> Topic: Community zoom meeting
>> Time: this is a recurring meeting Meet anytime
>>
>> Join from PC, Mac, Linux, iOS or Android:
>> https://hortonworks.zoom.us/j/658498271
>>
>> Or join by phone:
>>
>> +1 669 900 6833  (US Toll) or +1 646 558 8656 > 646
>> 558 8656> (US Toll)
>> +1 877 853 5247  (US Toll Free)
>> +1 877 369 0926  (US Toll Free)
>> Meeting ID: 658 498 271
>> International numbers available:
>> https://hortonworks.zoom.us/zoomconference?m=y7M0gPfv8kRv3Wv
>> XHjXrpc3n3DyNqTMe
>>
>>
>> On December 17, 2017 at 13:05:50, Otto Fowler (ottobackwa...@gmail.com)
>> wrote:
>>
>> The December Community Meeting will be held Monday, December 18th.
>> These are the topics that are up for discussion
>>
>>- Call for reviewers, ideas how to get more involvement, what people
>> can
>>do to help (Otto)
>>- Feature branches : we have two now, what are they and how are we
>> going
>>to work on them (Otto)
>>- Release process WRT formalized upgrade and installation instructions
>>to be
>>included as a part of a release (JZeolla)
>>- Any concerns/questions
>>with the secondary repo for bro. (JZeolla)
>>- ES 5.6 upgrade (michael.miklav...@gmail.com)
>>- Release Status(michael.miklav...@gmail.com)
>>- Short Term Roadmap(michael.miklav...@gmail.com)
>>
>>
>> We may only have 40 minutes, so we’ll try to keep things concise, and
>> follow up with Discuss threads.
>>
>> *NOTE: IF THE ROOM CHANGES I WILL SEND AN UPDATE*
>>
>>
>> Topic: Metron Developer Community Meeting
>> Time: Dec 18, 2017 12:30 PM Eastern Time (US and Canada)
>>
>> Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/4534152036
>>
>> Or iPhone one-tap :
>> US: +16468769923,,4534152036#  or +16699006833,,4534152036#
>> Or Telephone:
>> Dial(for higher quality, dial a number based on your current
>> location)：
>> US: +1 646 876 9923  or +1 669 900 6833
>> Meeting ID: 453 415 2036
>> International numbers available:
>> https://zoom.us/zoomconference?m=iwzFkc-YD_msf1cfRJL21VDYsExP41jo
>>
>>
>> Developer Community Meeting Disclaimers
>>
>>- Developer Community meetings are a means for realtime discussion of
>>development issues
>>- These meetings are not specifically aimed at demonstrations, unless
>>one is required or requested as part of such discussion
>>- These meetings are geared towards Metron development issues, not user
>>issues with deployment or shipped functionality
>>- There are NO decisions made in these meetings. The mailing list is
>> the
>>official communication record of the Apache Metron Project, and as
>> such all
>>public decisions are to be made on the list, as to give the greatest
>>opportunity for community involvement.
>>- There ARE proposals that can be made and discussed in these meetings,
>>that will then be discussed on list for decision.
>>- Notes will be taken of these meetings, and they will be posted to the
>>list
>>- There may also be breakout posts to the list per proposal or topic,
>>for more detailed discussion
>>
>
>
>
> --
> Regards,
> Nadir Hajiyani
>



-- 
Regards,
Nadir Hajiyani

Re: UPDATE MEETING December Developer Community Meeting

2017-12-17 Thread Nadir Hajiyani

Hi,
What time is this meeting at?

Thanks.

On Sun, Dec 17, 2017 at 1:32 PM, Otto Fowler 
wrote:

> We will be using this meeting
>
> Topic: Community zoom meeting
> Time: this is a recurring meeting Meet anytime
>
> Join from PC, Mac, Linux, iOS or Android:
> https://hortonworks.zoom.us/j/658498271
>
> Or join by phone:
>
> +1 669 900 6833  (US Toll) or +1 646 558 8656  558 8656> (US Toll)
> +1 877 853 5247  (US Toll Free)
> +1 877 369 0926  (US Toll Free)
> Meeting ID: 658 498 271
> International numbers available:
> https://hortonworks.zoom.us/zoomconference?m=
> y7M0gPfv8kRv3WvXHjXrpc3n3DyNqTMe
>
>
> On December 17, 2017 at 13:05:50, Otto Fowler (ottobackwa...@gmail.com)
> wrote:
>
> The December Community Meeting will be held Monday, December 18th.
> These are the topics that are up for discussion
>
>- Call for reviewers, ideas how to get more involvement, what people can
>do to help (Otto)
>- Feature branches : we have two now, what are they and how are we going
>to work on them (Otto)
>- Release process WRT formalized upgrade and installation instructions
>to be
>included as a part of a release (JZeolla)
>- Any concerns/questions
>with the secondary repo for bro. (JZeolla)
>- ES 5.6 upgrade (michael.miklav...@gmail.com)
>- Release Status(michael.miklav...@gmail.com)
>- Short Term Roadmap(michael.miklav...@gmail.com)
>
>
> We may only have 40 minutes, so we’ll try to keep things concise, and
> follow up with Discuss threads.
>
> *NOTE: IF THE ROOM CHANGES I WILL SEND AN UPDATE*
>
>
> Topic: Metron Developer Community Meeting
> Time: Dec 18, 2017 12:30 PM Eastern Time (US and Canada)
>
> Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/4534152036
>
> Or iPhone one-tap :
> US: +16468769923,,4534152036#  or +16699006833,,4534152036#
> Or Telephone:
> Dial(for higher quality, dial a number based on your current location)：
> US: +1 646 876 9923  or +1 669 900 6833
> Meeting ID: 453 415 2036
> International numbers available:
> https://zoom.us/zoomconference?m=iwzFkc-YD_msf1cfRJL21VDYsExP41jo
>
>
> Developer Community Meeting Disclaimers
>
>- Developer Community meetings are a means for realtime discussion of
>development issues
>- These meetings are not specifically aimed at demonstrations, unless
>one is required or requested as part of such discussion
>- These meetings are geared towards Metron development issues, not user
>issues with deployment or shipped functionality
>- There are NO decisions made in these meetings. The mailing list is the
>official communication record of the Apache Metron Project, and as such
> all
>public decisions are to be made on the list, as to give the greatest
>opportunity for community involvement.
>- There ARE proposals that can be made and discussed in these meetings,
>that will then be discussed on list for decision.
>- Notes will be taken of these meetings, and they will be posted to the
>list
>- There may also be breakout posts to the list per proposal or topic,
>for more detailed discussion
>



-- 
Regards,
Nadir Hajiyani

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

2017-12-17 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157377882
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -346,6 +348,18 @@ private void handleMagicFunctions(String[] expression) 
{
 writeLine(functions);
   }
 
+  /**
+   * Handle a magice %timing. Returns the results of the last timing 
operation
--- End diff --

Done thanks



---

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

2017-12-17 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157377835
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarExecutor.java
 ---
@@ -289,7 +293,25 @@ public Object execute(String expression) {
 VariableResolver variableResolver = new 
MapVariableResolver(Maps.transformValues(variables, result -> 
result.getResult())
, 
Collections.emptyMap());
 StellarProcessor processor = new StellarProcessor();
-return processor.parse(expression, variableResolver, functionResolver, 
context);
+StackWatch watch = new StackWatch("execute");
+watch.startTime(expression);
+context.setWatch(watch);
+try {
+  return processor.parse(expression, variableResolver, 
functionResolver, context);
+} finally {
+  watch.stopTime();
+  final StringBuffer buff = new StringBuffer();
+  watch.visit(((level, node) -> {
+for (int i = 0; i < level; i++) {
+  buff.append("-");
+}
+buff.append("->");
+buff.append(node.getName()).append(" : 
").append(node.getTime()).append("ms : ").
+append(node.getNanoTime()).append("ns").append("\n");
+  }));
+  lastTiming = Optional.of(buff.toString());
+  context.clearWatch();
+}
--- End diff --

I do not have any metrics, but I believe there is low impact to checking 
the state.context and the thread local's availability.

The runtime ( non-shell ) case, we would be paying the penalty of checking 
that the context watch was there or not, and having it not be there.



---

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

2017-12-17 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157377783
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarExecutor.java
 ---
@@ -289,7 +293,25 @@ public Object execute(String expression) {
 VariableResolver variableResolver = new 
MapVariableResolver(Maps.transformValues(variables, result -> 
result.getResult())
, 
Collections.emptyMap());
 StellarProcessor processor = new StellarProcessor();
-return processor.parse(expression, variableResolver, functionResolver, 
context);
+StackWatch watch = new StackWatch("execute");
+watch.startTime(expression);
+context.setWatch(watch);
+try {
+  return processor.parse(expression, variableResolver, 
functionResolver, context);
+} finally {
+  watch.stopTime();
+  final StringBuffer buff = new StringBuffer();
+  watch.visit(((level, node) -> {
+for (int i = 0; i < level; i++) {
+  buff.append("-");
+}
+buff.append("->");
+buff.append(node.getName()).append(" : 
").append(node.getTime()).append("ms : ").
+append(node.getNanoTime()).append("ns").append("\n");
+  }));
+  lastTiming = Optional.of(buff.toString());
+  context.clearWatch();
+}
--- End diff --

As I hit on in the writeup, stellar expressions are parsed into a stack of 
actions.  Some of those actions put other actions onto the stack, or put values 
etc etc.
So a Stellar Function would not actually wrap the functions inside it the 
way you think.

The scoping doesn't work that way.


---

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

2017-12-17 Thread JonZeolla

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157376617
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarShell.java
 ---
@@ -346,6 +348,18 @@ private void handleMagicFunctions(String[] expression) 
{
 writeLine(functions);
   }
 
+  /**
+   * Handle a magice %timing. Returns the results of the last timing 
operation
--- End diff --

`s/magice/magic/`


---

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

2017-12-17 Thread JonZeolla

Github user JonZeolla commented on a diff in the pull request:

https://github.com/apache/metron/pull/873#discussion_r157376611
  
--- Diff: 
metron-stellar/stellar-common/src/main/java/org/apache/metron/stellar/common/shell/StellarExecutor.java
 ---
@@ -289,7 +293,25 @@ public Object execute(String expression) {
 VariableResolver variableResolver = new 
MapVariableResolver(Maps.transformValues(variables, result -> 
result.getResult())
, 
Collections.emptyMap());
 StellarProcessor processor = new StellarProcessor();
-return processor.parse(expression, variableResolver, functionResolver, 
context);
+StackWatch watch = new StackWatch("execute");
+watch.startTime(expression);
+context.setWatch(watch);
+try {
+  return processor.parse(expression, variableResolver, 
functionResolver, context);
+} finally {
+  watch.stopTime();
+  final StringBuffer buff = new StringBuffer();
+  watch.visit(((level, node) -> {
+for (int i = 0; i < level; i++) {
+  buff.append("-");
+}
+buff.append("->");
+buff.append(node.getName()).append(" : 
").append(node.getTime()).append("ms : ").
+append(node.getNanoTime()).append("ns").append("\n");
+  }));
+  lastTiming = Optional.of(buff.toString());
+  context.clearWatch();
+}
--- End diff --

Do you have any metrics regarding the overhead of this?  Instead of doing 
this for all stellar commands, would it make sense to implement a TIME() 
steller function that could wrap all other commands, similar to how it 
functions on *nix boxes?


---

December Developer Community Meeting

2017-12-17 Thread Otto Fowler

The December Community Meeting will be held Monday, December 18th.
These are the topics that are up for discussion

   - Call for reviewers, ideas how to get more involvement, what people can
   do to help (Otto)
   - Feature branches : we have two now, what are they and how are we going
   to work on them (Otto)
   - Release process WRT formalized upgrade and installation instructions
   to be
   included as a part of a release (JZeolla)
   - Any concerns/questions
   with the secondary repo for bro. (JZeolla)
   - ES 5.6 upgrade (michael.miklav...@gmail.com)
   - Release Status(michael.miklav...@gmail.com)
   - Short Term Roadmap(michael.miklav...@gmail.com)


We may only have 40 minutes, so we’ll try to keep things concise, and
follow up with Discuss threads.

*NOTE: IF THE ROOM CHANGES I WILL SEND AN UPDATE*


Topic: Metron Developer Community Meeting
Time: Dec 18, 2017 12:30 PM Eastern Time (US and Canada)

Join from PC, Mac, Linux, iOS or Android: https://zoom.us/j/4534152036

Or iPhone one-tap :
US: +16468769923,,4534152036#  or +16699006833,,4534152036#
Or Telephone:
Dial(for higher quality, dial a number based on your current location)：
US: +1 646 876 9923  or +1 669 900 6833
Meeting ID: 453 415 2036
International numbers available:
https://zoom.us/zoomconference?m=iwzFkc-YD_msf1cfRJL21VDYsExP41jo


Developer Community Meeting Disclaimers

   - Developer Community meetings are a means for realtime discussion of
   development issues
   - These meetings are not specifically aimed at demonstrations, unless
   one is required or requested as part of such discussion
   - These meetings are geared towards Metron development issues, not user
   issues with deployment or shipped functionality
   - There are NO decisions made in these meetings. The mailing list is the
   official communication record of the Apache Metron Project, and as such all
   public decisions are to be made on the list, as to give the greatest
   opportunity for community involvement.
   - There ARE proposals that can be made and discussed in these meetings,
   that will then be discussed on list for decision.
   - Notes will be taken of these meetings, and they will be posted to the
   list
   - There may also be breakout posts to the list per proposal or topic,
   for more detailed discussion

Re: [DEV COMMUNITY MEETING] Call for Ideas and Schedule

2017-12-17 Thread Otto Fowler

I am still waiting for info on the zoom room.  I will have an email out by
the end of the day with information, one way or another.



On December 15, 2017 at 18:15:21, Otto Fowler (ottobackwa...@gmail.com)
wrote:

Great guys,

I’m going to leave the call OPEN for ideas, but at this point let’s say we
are going to schedule it for that time.
I will send the announce when I hear back form James about the room


On December 15, 2017 at 15:44:59, Michael Miklavcic (
michael.miklav...@gmail.com) wrote:

Sounds good Otto. We probably also want to touch on the ES 5.6 upgrade
along with our current release status and short-term release roadmap that
Nick Allen has been guiding.

On Fri, Dec 15, 2017 at 9:02 AM, Laurens Vets  wrote:

> I'll try to attend :)
>
>
> On 2017-12-14 12:43, Otto Fowler wrote:
>
>> Dev Community Meeting Call
>>
>> I would like to propose a developer community meeting.
>>
>> I propose that we set the meeting early next week, and will throw out
>> Monday, December 18th at 09:30AM PST, 12:30 on the East Coast and 5:30 in
>> London Towne.
>>
>> This meeting will be held over a web-ex, the details of which will be
>> included in the actual meeting notice.
>>
>> Please reply to this with scheduling concerns and topic suggestions.
>> Potential Topics
>>
>>- Call for reviewers, ideas how to get more involvement, what people
>> can
>>do to help
>>- Feature branches : we have two now, what are they and how are we
>> going
>>to work on them
>>- Extension Repository: Default deployment and installation of parsers
>>as it relates to ‘777’
>>- General ‘777’ discussion
>>
>> Developer Community Meeting Disclaimers
>>
>>- Developer Community meetings are a means for realtime discussion of
>>development issues
>>- These meetings are not specifically aimed at demonstrations, unless
>>one is required or requested as part of such discussion
>>- These meetings are geared towards Metron development issues, not user
>>issues with deployment or shipped functionality
>>- There are *NO* decisions made in these meetings. The mailing list is
>>the official communication record of the Apache Metron Project, and as
>> such
>>all public decisions are to be made on the list, as to give the
>> greatest
>>opportunity for community involvement.
>>- There *ARE* proposals that can be made and discussed in these
>>meetings, that will then be discussed on list for decision.
>>- Notes will be taken of these meetings, and they will be posted to the
>>list
>>- There may also be breakout posts to the list per proposal or topic,
>>for more detailed discussion
>>
>

Re: [INTRODUCTIO] Would like to contribute

2017-12-17 Thread Otto Fowler

Hi Pushpitha!

Welcome!

Joining the dev and user mailing lists is a great start.  Also, take a look
through https://metron.apache.org , the community and documentation sites
etc.
I find reading the list archives helpful as well.  You will also find our
free node irc channel there.

As to how to contribute, there are many ways to contribute, depending on
your understanding, skills, and interests.

Reviewing PR’s and Documentation helps even if a person doesn’t code
themselves.  There are newbie jira issues in the Apache jira as well :
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=128=METRON=planning.nodetail=797=visible,
 although I’m not sure that some haven’t be obsoleted…..

There are many ways to get involved, and everyone has probably taken a
different route, but that is ok, there is no ‘one way’ to contribute.





On December 16, 2017 at 00:23:56, Pushpitha Somathilaka (
pushpitha...@cse.mrt.ac.lk) wrote:

Hi all,

I am Pushpitha Dilhan and I am a final year undergraduate majoring Computer
Science. I would like to contribute Apache Metron and hope to have guidance
for a start.

Thank you
Pushpitha

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

2017-12-17 Thread ottobackwards

GitHub user ottobackwards opened a pull request:

https://github.com/apache/metron/pull/873

METRON-1367 Stellar should have some instrumentation of function calls

This PR introduces the ability through a new `magic` command for a stellar 
shell user to view the timing information for the last statement the shell 
executed.  The information provided is the total execution time, and the 
execution time per stellar function.

At this point, only the stellar function execution has been timed, but it 
is possible in the future to add timing to other parts of stellar, for example 
to time how long lambdas take, or matches.

For example:

```bash
Stellar, Go!
Please note that functions are loading lazily in the background and will be 
unavailable until loaded fully.
[Stellar]>>> Functions loaded, you may refer to functions now...

[Stellar]>>> hex:="91IMOR3F41BMUSJCCG=="
[Stellar]>>> if( STARTS_WITH(TO_UPPER(DECODE(hex,'BASE32HEX')), "HELLO")) 
THEN TO_LOWER("OK") ELSE TO_LOWER("")
ok
[Stellar]>>> %timing
->execute : 113ms : 113017132
-->if( STARTS_WITH(TO_UPPER(DECODE(hex,'BASE32HEX')), "HELLO")) THEN 
TO_LOWER("OK") ELSE TO_LOWER("") : 112ms : 112998881ns
--->DECODE : 8ms : 8572968ns
--->TO_UPPER : 0ms : 20018ns
--->STARTS_WITH : 0ms : 30586ns
--->TO_LOWER : 0ms : 19085ns
```

## StackWatch
A set of utility classes under the new package stellar.common.timing have 
been added.  These provide the `StackWatch` functionality.

StackWatch provides an abstraction over the Apache Commons StopWatch class 
that allows callers to 
create multiple named and possibly nested timing operations.

Since Stellar operations are executed in a queue and in isolation ( even a 
Stellar statement that is nested  such as  TO_UPPER(TO_STRING()) is executed as 
 TO_STRING(), TO_UPPER, the benefit of the support for nested timing operations 
is not utilized.  

This class may be more generally useful to this and other projects, but I 
am not sure where it would live since we wouldn't want it in common.

StackWatch uses a combination of `Deque` and a custom Tree implementation 
to create, start and end timing operations.

A `Visitor` pattern is also implemented to allow for retrieving the results 
after the completion of the operation.

See the StackWatch tests for examples of nested function timings.

The StackWatch is optionally available from the Stellar `Context` object.

## Testing

To test:

- Run the Stellar Shell, either from full_dev or using mvn exec ( here run 
from metron-stellar/stellar-common after a mvn package )
```
> % mvn exec:java  
-Dexec.mainClass="org.apache.metron.stellar.common.shell.StellarShell"
```

- run some stellar statements
- run the %timing function
- in full dev verify that data is processed normally through storm, and 
that it shows in kibana



- [x] Is there a JIRA ticket associated with this PR? If not one needs to 
be created at [Metron 
Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel).
 
- [x] Does your PR title start with METRON- where  is the JIRA 
number you are trying to resolve? Pay particular attention to the hyphen "-" 
character.
- [x] Has your PR been rebased against the latest commit within the target 
branch (typically master)?


### For code changes:
- [ ] Have you included steps to reproduce the behavior or problem that is 
being changed or addressed?
- [x] Have you included steps or a guide to how the change may be verified 
and tested manually?
- [x] Have you ensured that the full suite of tests and checks have been 
executed in the root metron folder via:
  ```
  mvn -q clean integration-test install && build_utils/verify_licenses.sh 
  ```

- [x] Have you written or updated unit tests and or integration tests to 
verify your changes?
- [ ] If adding new dependencies to the code, are these dependencies 
licensed in a way that is compatible for inclusion under [ASF 
2.0](http://www.apache.org/legal/resolved.html#category-a)? 
- [ ] Have you verified the basic functionality of the build by building 
and running locally with Vagrant full-dev environment or the equivalent?

### For documentation related changes:
- [x] Have you ensured that format looks appropriate for the output in 
which it is rendered by building and verifying the site-book? If not then run 
the following commands and the verify changes via 
`site-book/target/site/index.html`:

  ```
  cd site-book
  mvn site
  ```


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ottobackwards/metron stellar_timing

Alternatively you can review and apply these changes

[GitHub] metron pull request #871: METRON-1365: Allow PROFILE_GET to return a default...

[GitHub] metron pull request #871: METRON-1365: Allow PROFILE_GET to return a default...

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

Re: UPDATE MEETING December Developer Community Meeting

Re: UPDATE MEETING December Developer Community Meeting

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

December Developer Community Meeting

Re: [DEV COMMUNITY MEETING] Call for Ideas and Schedule

Re: [INTRODUCTIO] Would like to contribute

[GitHub] metron pull request #873: METRON-1367 Stellar should have some instrumentati...

14 matches

Site Navigation

Mail list logo

Footer information