[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/915
  
@mraliagha Nah, they're not a troublemaker.  I'm removing the PR since 
there's interest in them staying turned on by default.


---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[mraliagha]

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/915
  
@cestella Definitely there will be value for having an ability to turn it 
on and off especially since you have already implemented that. However, won't 
be a sort of premature disk optimization? We are storing original_string and 
lots of other things that generally we may not really use them. Are those 
timestamp fields really troublemaker?


---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/915
  
Perhaps we can leverage parts of the performance logging code for this
purpose? The frequency/sampling bit was something I had wanted to add for
some time now.

https://github.com/apache/metron/pull/665


On Wed, Jan 31, 2018 at 7:58 AM, Casey Stella 
wrote:

> @mraliagha  Well, if it's being used
> explicitly and on an on-going manner, then I'll remove this PR. Do you
> think there's some value to being able to slice and dice in the global
> config the ability to turn on and off these by:
>
>- type - threatintel vs enrichment
>- adapter - stellar vs hbase vs geo
>- frequency - kinda like a tracer, have 1 in k messages have them
>
> Thoughts?
>
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> , or 
mute
> the thread
> 

> .
>



---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/915
  
@mraliagha Well, if it's being used explicitly and on an on-going manner, 
then I'll remove this PR.  Do you think there's some value to being able to 
slice and dice in the global config the ability to turn on and off these by:
* type - threatintel vs enrichment
* adapter - stellar vs hbase vs geo
* frequency - kinda like a tracer, have 1 in k messages have them

Thoughts?


---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[mraliagha]

Github user mraliagha commented on the issue:

https://github.com/apache/metron/pull/915
  
@simonellistonball We are using threat intel joiner timestamp currently. I 
am not sure indexing timestamp is added to the latest version or not, but we 
need the ES indexing one as well.


---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[simonellistonball]

Github user simonellistonball commented on the issue:

https://github.com/apache/metron/pull/915
  
Does anyone anticipate using (or even use) these fields to do things like 
monitor the flow and latency of their pipelines? I can imagine people wanting 
to build data quality monitoring reports on these fields, but would like to 
verify that assumption.


---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/915
  
+1, lgtm via inspection.


---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[cestella]

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/915
  
Whoops, missed one. Thanks @mmiklavc !


---

[GitHub] metron issue #915: METRON-1433: Only emit debugging timing fields in enrichm...

[mmiklavc]

Github user mmiklavc commented on the issue:

https://github.com/apache/metron/pull/915
  

https://github.com/apache/metron/blob/master/metron-platform/metron-enrichment/src/main/java/org/apache/metron/enrichment/bolt/EnrichmentJoinBolt.java#L89


---