[jira] [Created] (METRON-2328) [UI] Stale data state inconsistencies in the Alerts UI
Shane Ardell created METRON-2328: Summary: [UI] Stale data state inconsistencies in the Alerts UI Key: METRON-2328 URL: https://issues.apache.org/jira/browse/METRON-2328 Project: Metron Issue Type: Bug Reporter: Shane Ardell Link to original discussion: https://github.com/apache/metron/pull/1527#pullrequestreview-298193000 User scenario #1: User switch to manual query mode Making changes on the filtering Pressing search and receives another set of alerts Then moves back to Query Builder mode This could turn the UI to a "stale data state" bc the list of alerts filtered by the previously shown manual query and the visible filters and the data no longer in sync. User scenario #2: User switch to manual query mode Making changes in the filtering UI should turn to a "stale data state" to keep behavior consistent across ace and manual editors. User scenario #3: User switch to manual query mode Making changes in the filtering UI should turn to a "stale data state" to keep behavior consistent across ace and manual editors. But when User switch back without updating the data based on the manual filter "Stale data state" should be cleared hence the data is in sync with the shown filters again. User scenario #4: User move to manual mode Making changes in the filtering Pressing search and receiving another set of data Then switching to query builder mode UI should be in a "stale data state" as described in scenario #1 If User switch back to manual mode "stale data state" should be cleared hence the data is in sync with the filters again -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [metron] mmiklavc commented on issue #1566: METRON-2322 Add Ambari connection check to upgrade_helper script
mmiklavc commented on issue #1566: METRON-2322 Add Ambari connection check to upgrade_helper script URL: https://github.com/apache/metron/pull/1566#issuecomment-558239580 Had to resolve a conflict with master in the upgrading.md doc. Letting Travis re-run before I merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-2326) Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field
[
https://issues.apache.org/jira/browse/METRON-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic updated METRON-2326:
--
Fix Version/s: Next + 1
> Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field
> --
>
> Key: METRON-2326
> URL: https://issues.apache.org/jira/browse/METRON-2326
> Project: Metron
> Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> A Threat Triage Rule's "reason" field can contain executable Stellar to
> provide an operator context as to why a rule fired during Threat Triage. I
> am unable to call any function that requires a StellarContext during
> initialization, from the 'Reason' field of a Threat Triage Rule. For
> example, I cannot call `ENRICHMENT_GET`.
> h3. Steps to Replicate
> 1. Create a simple file called `user.csv`.
> {code:java}
> [root@node1 ~]# cat user.csv
> jdoe,192.168.138.2
> jane,192.168.66.1
> ciana,192.168.138.158
> danixa,95.163.121.204
> jim,192.168.66.121
> {code}
> 2 . Create a file called `user-extractor.json`.
> {code:java}
> {
> "config": {
> "columns": {
> "user": 0,
> "ip": 1
> },
> "indicator_column": "ip",
> "separator": ",",
> "type": "user"
> },
> "extractor": "CSV"
> }
> {code}
> 3. Import the enrichment data.
> {code:java}
> source /etc/default/metron
> $METRON_HOME/bin/flatfile_loader.sh -i ./user.csv -t enrichment -c t -e
> ./user-extractor.json
> {code}
> 4. Validate that the enrichment loaded successfully.
> {code:java}
> [root@node1 0.7.2]# source /etc/default/metron
> [root@node1 0.7.2]# $METRON_HOME/bin/stellar -z $ZOOKEEPER
>
> [Stellar]>>> ip_dst_addr := "192.168.138.2"
> 192.168.138.2
>
> [Stellar]>>> ENRICHMENT_GET('user', ip_dst_addr, 'enrichment', 't')
> \{ip=192.168.138.2, user=jdoe}
> {code}
> 5. Create a threat triage rule that attempts an ENRICHMENT_GET.
> {code}
> [Stellar]>>> conf := SHELL_EDIT()
> {
> "enrichment": {
> "fieldMap": {
> "stellar": {
> "config": {
> "is_alert": "true"
> }
> }
> },
> "fieldToTypeMap": {},
> "config": {}
> },
> "threatIntel": {
> "fieldMap": {},
> "fieldToTypeMap": {},
> "config": {},
> "triageConfig": {
> "riskLevelRules": [
> {
> "name": "Rule",
> "comment": "This rule does not work when executing the 'reason' field.",
> "rule": "true",
> "reason": "FORMAT('Call to ENRICHMENT_GET=%s', ENRICHMENT_GET('user',
> ip_dst_addr, 'enrichment', 't'))",
> "score": "100"
> }
> ],
> "aggregator": "MAX",
> "aggregationConfig": {}
> }
> },
> "configuration": {}
> }
>
> [Stellar]>>> CONFIG_PUT("ENRICHMENT", conf, "snort")
> {code}
>
> 6. The Storm worker logs for Enrichment show the following error.
> {code:java}
> 2019-11-21 03:54:34.370 o.a.c.f.r.c.TreeCache Curator-TreeCache-4 [ERROR]
> org.apache.metron.jackson.databind.JsonMappingException: Unable to find
> capability GLOBAL_CONFIG; it may not be available in your context.
> at [Source: java.io.ByteArrayInputStream@1f55bdda; line: 24, column: 11]
> (through reference chain:
> org.apache.metron.common.configuration.enrichment.SensorEnrichmentConfig["threatIntel"]->org.apache.metron.common.configuration.enrichment.threatintel.ThreatIntelConfig["triageConfig"]->org.apache.metron.common.configuration.enrichment.threatintel.ThreatTriageConfig["riskLevelRules"])
> at
> org.apache.metron.jackson.databind.JsonMappingException.from(JsonMappingException.java:262)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.SettableBeanProperty._throwAsIOE(SettableBeanProperty.java:537)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.SettableBeanProperty._throwAsIOE(SettableBeanProperty.java:518)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:99)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.SettableBeanProperty.deserialize(SettableBeanProperty.java:490)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.impl.MethodProperty.deserializeAndSet(MethodProperty.java:95)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:260)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:125)
> ~[stormjar.jar:?]
> at
> org.apache.metron.jackson.databind.deser.SettableBeanPr
[GitHub] [metron] asfgit closed pull request #1570: METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field
asfgit closed pull request #1570: METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field URL: https://github.com/apache/metron/pull/1570 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1570: METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field
mmiklavc commented on issue #1570: METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field URL: https://github.com/apache/metron/pull/1570#issuecomment-558235503 +1 by inspection. I like the added docs. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (METRON-2285) Batch Profiler Cannot Persist Data Sketches
[
https://issues.apache.org/jira/browse/METRON-2285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Miklavcic updated METRON-2285:
--
Fix Version/s: Next + 1
> Batch Profiler Cannot Persist Data Sketches
> ---
>
> Key: METRON-2285
> URL: https://issues.apache.org/jira/browse/METRON-2285
> Project: Metron
> Issue Type: Bug
>Affects Versions: 0.7.1
>Reporter: Maxim Dashenko
>Assignee: Nick Allen
>Priority: Major
> Fix For: Next + 1
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Used command:
> {code}
> /usr/hdp/current/spark2-client/bin/spark-submit --class
> org.apache.metron.profiler.spark.cli.BatchProfilerCLI --properties-file
> /usr/hcp/current/metron/config/batch-profiler.properties
> ~/metron-profiler-spark-0.7.1.1.9.1.0-6.jar --config
> /usr/hcp/current/metron/config/batch-profiler.properties --profiles
> ~/profiler.json
> {code}
> cat /usr/hcp/current/metron/config/batch-profiler.properties
> {code}
> profiler.batch.input.path=/tmp/test_data.logs
> profiler.batch.input.format=json
> profiler.period.duration=15
> profiler.period.duration.units=MINUTES
> {code}
>
> cat ~/profiler.json
> {code}
> {
>"profiles":[
> {
> "profile":"batchteststat",
> "onlyif":"source.type == 'testsource' and devicehostname ==
> 'windows9.something.com'",
> "foreach":"devicehostname",
> "update":{
> "s":"STATS_ADD(s, devicehostname)"
> },
> "result":{
> "profile":"s"
> }
> }
>],
>"timestampField":"timestamp"
> }
> {code}
> cat test_data.logs
> {code}
> {"devicehostname": "windows9.something.com", "timestamp": 1567241981000,
> "source.type": "testsource"}
> {code}
> The command raises an exception:
> {code}
> Exception in thread "main" org.apache.spark.SparkException: Job aborted due
> to stage failure: Task 68 in stage 8.0 failed 1 times, most recent failure:
> Lost task 68.0 in stage 8.0 (TID 274, localhost, executor driver):
> com.esotericsoftware.kryo.KryoException: Unable to find class:
> org.apache.metron.statistics.OnlineStatisticsProvider
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readName(DefaultClassResolver.java:156)
> at
> com.esotericsoftware.kryo.util.DefaultClassResolver.readClass(DefaultClassResolver.java:133)
> at com.esotericsoftware.kryo.Kryo.readClass(Kryo.java:670)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:781)
> at
> org.apache.metron.common.utils.SerDeUtils.fromBytes(SerDeUtils.java:262)
> at
> org.apache.metron.profiler.spark.ProfileMeasurementAdapter.toProfileMeasurement(ProfileMeasurementAdapter.java:85)
> at
> org.apache.metron.profiler.spark.function.HBaseWriterFunction.call(HBaseWriterFunction.java:124)
> at org.apache.spark.sql.Dataset$$anonfun$48.apply(Dataset.scala:2266)
> at org.apache.spark.sql.Dataset$$anonfun$48.apply(Dataset.scala:2266)
> at
> org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:196)
> at
> org.apache.spark.sql.execution.MapPartitionsExec$$anonfun$6.apply(objects.scala:193)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
> at
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:827)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at
> org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
> at org.apache.spark.scheduler.Task.run(Task.scala:108)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:338)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException:
> org.apache.metron.statistics.OnlineStatisticsProvider
> at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
> at java.lang
[GitHub] [metron] asfgit closed pull request #1564: METRON-2285 Batch Profiler Cannot Persist Data Sketches
asfgit closed pull request #1564: METRON-2285 Batch Profiler Cannot Persist Data Sketches URL: https://github.com/apache/metron/pull/1564 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] mmiklavc commented on issue #1564: METRON-2285 Batch Profiler Cannot Persist Data Sketches
mmiklavc commented on issue #1564: METRON-2285 Batch Profiler Cannot Persist Data Sketches URL: https://github.com/apache/metron/pull/1564#issuecomment-558233019 +1 by inspection. Thanks for the fix @nickwallen This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] MohanDV commented on issue #1570: METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field
MohanDV commented on issue #1570: METRON-2326 Unable to Call ENRICHMENT_GET from Threat Triage Rule Reason Field URL: https://github.com/apache/metron/pull/1570#issuecomment-558131773 spun up centos 7 full dev environment. Completed the acceptance testing +1 by verification. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] nickwallen commented on issue #1572: METRON-2088 - Support for SOLR time-based arrays
nickwallen commented on issue #1572: METRON-2088 - Support for SOLR time-based arrays URL: https://github.com/apache/metron/pull/1572#issuecomment-558127545 You should create a new JIRA not reuse 2088 please. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] tigerquoll opened a new pull request #1572: METRON-2088 - Support for SOLR time-based arrays
tigerquoll opened a new pull request #1572: METRON-2088 - Support for SOLR time-based arrays URL: https://github.com/apache/metron/pull/1572 ## Contributor Comments Updated schemas and instructions on how to enable SOLR time based arrays. ## Pull Request Checklist Thank you for submitting a contribution to Apache Metron. Please refer to our [Development Guidelines](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=61332235) for the complete guide to follow for contributions. Please refer also to our [Build Verification Guidelines](https://cwiki.apache.org/confluence/display/METRON/Verifying+Builds?show-miniview) for complete smoke testing guides. In order to streamline the review of the contribution we ask you follow these guidelines and ask you to double check the following: ### For all changes: - [x] Is there a JIRA ticket associated with this PR? If not one needs to be created at [Metron Jira](https://issues.apache.org/jira/browse/METRON/?selectedTab=com.atlassian.jira.jira-projects-plugin:summary-panel). - [x] Does your PR title start with METRON- where is the JIRA number you are trying to resolve? Pay particular attention to the hyphen "-" character. - [x] Has your PR been rebased against the latest commit within the target branch (typically master)? ### For documentation related changes: - [x] Have you ensured that format looks appropriate for the output in which it is rendered by building and verifying the site-book? If not then run the following commands and the verify changes via `site-book/target/site/index.html`: ``` cd site-book mvn site ``` - [x] Have you ensured that any documentation diagrams have been updated, along with their source files, using [draw.io](https://www.draw.io/)? See [Metron Development Guidelines](https://cwiki.apache.org/confluence/display/METRON/Development+Guidelines) for instructions. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] MohanDV commented on issue #1566: METRON-2322 Add Ambari connection check to upgrade_helper script
MohanDV commented on issue #1566: METRON-2322 Add Ambari connection check to upgrade_helper script URL: https://github.com/apache/metron/pull/1566#issuecomment-558088321 > Ok @MohanDV give that a shot. I tweaked it a bit, but should still accomplish the same goal. Are you running that command on Centos 7? For Centos 6 I was not able to reproduce the issue, incidentally. @mmiklavc Yes, I ran the script on centos 7. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [metron] MohanDV commented on issue #1566: METRON-2322 Add Ambari connection check to upgrade_helper script
MohanDV commented on issue #1566: METRON-2322 Add Ambari connection check to upgrade_helper script URL: https://github.com/apache/metron/pull/1566#issuecomment-558088428 +1 by verification. @mmiklavc works perfectly !! Thank you This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
