date:20180829

[GitHub] metron pull request #1171: METRON-1740 make parser support CONFIG and SYSTEM...

2018-08-29 Thread liuy-tnz

Github user liuy-tnz commented on a diff in the pull request:

https://github.com/apache/metron/pull/1171#discussion_r213836879
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/README.md
 ---
@@ -0,0 +1,15 @@
+# BasicPaloAltoFirewallParser
--- End diff --

Thank you. I've done it. 


---

[jira] [Commented] (METRON-1751) Storm Profiler dies when consuming null message

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596729#comment-16596729
 ] 

ASF GitHub Bot commented on METRON-1751:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1176


> Storm Profiler dies when consuming null message
> ---
>
> Key: METRON-1751
> URL: https://issues.apache.org/jira/browse/METRON-1751
> Project: Metron
>  Issue Type: Bug
>Reporter: Mohan
>Priority: Major
>
> When You publish a null message to the profiler input kafka topic which is 
> 'indexing' in my case I see the below exception messages on the worker log
> {code:java}
> 2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7] 
> [ERROR] Async loop died! java.lang.RuntimeException: 
> java.lang.NullPointerException at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at 
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: 
> java.lang.NullPointerException at java.lang.String.(String.java:491) 
> ~[?:1.8.0_181] at 
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
>  ~[stormjar.jar:?] at 
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
>  ~[stormjar.jar:?] at 
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] ... 6 more
> {code}
> also the profiler dies , even  if you start up the profiler again and run it 
> through the same topic, it does always die.
> {code:java}
> 2018-08-27 12:46:03.870 o.a.s.util Thread-9-splitterBolt-executor[7 7] 
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at 
> org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at 
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1176: METRON-1751 Storm Profiler dies when consuming nu...

2018-08-29 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1176


---

[GitHub] metron issue #1176: METRON-1751 Storm Profiler dies when consuming null mess...

2018-08-29 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1176
  
Thanks for the review @cestella ! I ran this up in Full Dev and everything 
looked proper.


---

[GitHub] metron pull request #1178: METRON-1757 Storm Profiler Serialization Exceptio...

2018-08-29 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1178


---

[jira] [Commented] (METRON-1757) Storm Profiler Serialization Exception

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596567#comment-16596567
 ] 

ASF GitHub Bot commented on METRON-1757:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1178
  
Yeah that would explain why I wasn't hitting it in full-dev, but I did hit 
it in my test deployment.  


> Storm Profiler Serialization Exception
> --
>
> Key: METRON-1757
> URL: https://issues.apache.org/jira/browse/METRON-1757
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running the Storm Profiler with multiple workers this serialization 
> error can occur.
> Even in an environment with multiple workers It seems to not occur all the 
> time. It may depend on how the individual bolts are executed across the 
> workers in a cluster. This likely occurs only when the HBaseBolt is executed 
> on a separate worker from the ProfileBuilderBolt.
> It will never occur when running with a single worker.
> {code}
> 2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler 
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the 
> request
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing 
> no-arg constructor): 
> org.apache.metron.common.configuration.profiler.ProfileResult
> Serialization trace:
> result (org.apache.metron.common.configuration.profiler.ProfileConfig)
> definition (org.apache.metron.profiler.ProfileMeasurement)
>  at 
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078) 
> ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.received(Server.java:254) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
>

[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception

2018-08-29 Thread JonZeolla

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1178
  
Yeah that would explain why I wasn't hitting it in full-dev, but I did hit 
it in my test deployment.  


---

[jira] [Commented] (METRON-1757) Storm Profiler Serialization Exception

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596486#comment-16596486
 ] 

ASF GitHub Bot commented on METRON-1757:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1178
  
FYI - I tested this on a multi-node cluster.  I was experiencing the 
problem and the topology could not write out any profile measurements. 

I updated the Profiler lib with these changes and restarting the topology.  
After the restart, I was able to successfully write out measurements to HBase. 
I also ensured that the `ProfileBuilderBolt` and `HBaseBolt` were running on 
different workers during this test.


> Storm Profiler Serialization Exception
> --
>
> Key: METRON-1757
> URL: https://issues.apache.org/jira/browse/METRON-1757
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running the Storm Profiler with multiple workers this serialization 
> error can occur.
> Even in an environment with multiple workers It seems to not occur all the 
> time. It may depend on how the individual bolts are executed across the 
> workers in a cluster. This likely occurs only when the HBaseBolt is executed 
> on a separate worker from the ProfileBuilderBolt.
> It will never occur when running with a single worker.
> {code}
> 2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler 
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the 
> request
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing 
> no-arg constructor): 
> org.apache.metron.common.configuration.profiler.ProfileResult
> Serialization trace:
> result (org.apache.metron.common.configuration.profiler.ProfileConfig)
> definition (org.apache.metron.profiler.ProfileMeasurement)
>  at 
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078) 
> ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.received(Server.java:254) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
>

[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception

2018-08-29 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1178
  
FYI - I tested this on a multi-node cluster.  I was experiencing the 
problem and the topology could not write out any profile measurements. 

I updated the Profiler lib with these changes and restarting the topology.  
After the restart, I was able to successfully write out measurements to HBase. 
I also ensured that the `ProfileBuilderBolt` and `HBaseBolt` were running on 
different workers during this test.


---

[jira] [Commented] (METRON-1696) Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596454#comment-16596454
 ] 

ASF GitHub Bot commented on METRON-1696:


Github user MohanDV commented on the issue:

https://github.com/apache/metron/pull/1134
  
 Closed as it is addressed in  METRON-1738


> Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster 
> -
>
> Key: METRON-1696
> URL: https://issues.apache.org/jira/browse/METRON-1696
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> pcap parser fails to write the pcap sequence files to hdfs directory due to 
> insufficient privileges to hdfs folder for 'metron' user 
> {code:java}
> 2018-07-25 10:15:50.035 o.a.m.s.p.HDFSWriterCallback 
> Thread-9-kafkaSpout-executor[3 3] [ERROR] Permission denied: user=metron, 
> access=WRITE, 
> inode="/apps/metron/pcap/pcap_pcap_1532513746365022000_0_pcap-20-1532414055":hdfs:hdfs:drwxr-xr-x
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...

2018-08-29 Thread MohanDV

Github user MohanDV commented on the issue:

https://github.com/apache/metron/pull/1134
  
 Closed as it is addressed in  METRON-1738


---

[jira] [Commented] (METRON-1696) Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596452#comment-16596452
 ] 

ASF GitHub Bot commented on METRON-1696:


Github user MohanDV commented on the issue:

https://github.com/apache/metron/pull/1134
  
closing as it addressed in METRON-1738


> Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster 
> -
>
> Key: METRON-1696
> URL: https://issues.apache.org/jira/browse/METRON-1696
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> pcap parser fails to write the pcap sequence files to hdfs directory due to 
> insufficient privileges to hdfs folder for 'metron' user 
> {code:java}
> 2018-07-25 10:15:50.035 o.a.m.s.p.HDFSWriterCallback 
> Thread-9-kafkaSpout-executor[3 3] [ERROR] Permission denied: user=metron, 
> access=WRITE, 
> inode="/apps/metron/pcap/pcap_pcap_1532513746365022000_0_pcap-20-1532414055":hdfs:hdfs:drwxr-xr-x
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...

2018-08-29 Thread MohanDV

Github user MohanDV commented on the issue:

https://github.com/apache/metron/pull/1134
  
closing as it addressed in METRON-1738


---

[GitHub] metron pull request #1134: METRON-1696: Create the HDFS directory for pcap s...

2018-08-29 Thread MohanDV

Github user MohanDV closed the pull request at:

https://github.com/apache/metron/pull/1134


---

[jira] [Commented] (METRON-1757) Storm Profiler Serialization Exception

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596431#comment-16596431
 ] 

ASF GitHub Bot commented on METRON-1757:


Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1178
  
The work-around to this issue, and some documentation of it to the extent 
you feel necessary should go out to the users list. 


> Storm Profiler Serialization Exception
> --
>
> Key: METRON-1757
> URL: https://issues.apache.org/jira/browse/METRON-1757
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running the Storm Profiler with multiple workers this serialization 
> error can occur.
> Even in an environment with multiple workers It seems to not occur all the 
> time. It may depend on how the individual bolts are executed across the 
> workers in a cluster. This likely occurs only when the HBaseBolt is executed 
> on a separate worker from the ProfileBuilderBolt.
> It will never occur when running with a single worker.
> {code}
> 2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler 
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the 
> request
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing 
> no-arg constructor): 
> org.apache.metron.common.configuration.profiler.ProfileResult
> Serialization trace:
> result (org.apache.metron.common.configuration.profiler.ProfileConfig)
> definition (org.apache.metron.profiler.ProfileMeasurement)
>  at 
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078) 
> ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.received(Server.java:254) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>

[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception

2018-08-29 Thread ottobackwards

Github user ottobackwards commented on the issue:

https://github.com/apache/metron/pull/1178
  
The work-around to this issue, and some documentation of it to the extent 
you feel necessary should go out to the users list. 


---

[jira] [Commented] (METRON-1757) Storm Profiler Serialization Exception

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596426#comment-16596426
 ] 

ASF GitHub Bot commented on METRON-1757:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1178
  
Here is one other detail that I forgot to add to the description.  This may 
be important if you are trying to validate this fix on a multi-node cluster.

* This serialization error will never show itself if you are running with a 
single worker as there is no need for Storm to serialize.  

* Even when running the Profiler in a cluster on multiple workers, it will 
only show-up if Storm runs the `ProfileBuilderBolt` and the downstream bolts 
(`HBaseBolt` or `KafkaBolt`) on a separate worker.  This is why you may not 
always see the problem when running on a cluster (and why it was confusing to 
track down.)


> Storm Profiler Serialization Exception
> --
>
> Key: METRON-1757
> URL: https://issues.apache.org/jira/browse/METRON-1757
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running the Storm Profiler with multiple workers this serialization 
> error can occur.
> Even in an environment with multiple workers It seems to not occur all the 
> time. It may depend on how the individual bolts are executed across the 
> workers in a cluster. This likely occurs only when the HBaseBolt is executed 
> on a separate worker from the ProfileBuilderBolt.
> It will never occur when running with a single worker.
> {code}
> 2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler 
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the 
> request
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing 
> no-arg constructor): 
> org.apache.metron.common.configuration.profiler.ProfileResult
> Serialization trace:
> result (org.apache.metron.common.configuration.profiler.ProfileConfig)
> definition (org.apache.metron.profiler.ProfileMeasurement)
>  at 
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078) 
> ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.received(Server.java:254) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
>

[jira] [Commented] (METRON-1757) Storm Profiler Serialization Exception

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596425#comment-16596425
 ] 

ASF GitHub Bot commented on METRON-1757:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1178
  
This looks good to me +1 thanks @nickwallen 


> Storm Profiler Serialization Exception
> --
>
> Key: METRON-1757
> URL: https://issues.apache.org/jira/browse/METRON-1757
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running the Storm Profiler with multiple workers this serialization 
> error can occur.
> Even in an environment with multiple workers It seems to not occur all the 
> time. It may depend on how the individual bolts are executed across the 
> workers in a cluster. This likely occurs only when the HBaseBolt is executed 
> on a separate worker from the ProfileBuilderBolt.
> It will never occur when running with a single worker.
> {code}
> 2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler 
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the 
> request
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing 
> no-arg constructor): 
> org.apache.metron.common.configuration.profiler.ProfileResult
> Serialization trace:
> result (org.apache.metron.common.configuration.profiler.ProfileConfig)
> definition (org.apache.metron.profiler.ProfileMeasurement)
>  at 
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078) 
> ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.received(Server.java:254) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
>

[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception

2018-08-29 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1178
  
Here is one other detail that I forgot to add to the description.  This may 
be important if you are trying to validate this fix on a multi-node cluster.

* This serialization error will never show itself if you are running with a 
single worker as there is no need for Storm to serialize.  

* Even when running the Profiler in a cluster on multiple workers, it will 
only show-up if Storm runs the `ProfileBuilderBolt` and the downstream bolts 
(`HBaseBolt` or `KafkaBolt`) on a separate worker.  This is why you may not 
always see the problem when running on a cluster (and why it was confusing to 
track down.)


---

[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception

2018-08-29 Thread JonZeolla

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1178
  
This looks good to me +1 thanks @nickwallen 


---

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596422#comment-16596422
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213706134
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  private transient SyslogParser syslogParser;
+
+  @Override
+  public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+  JSONObject jsonObject = new 
JSONObject(syslogParser.parseLine(originalString));
+
+  // be sure to put in the original string, and the timestamp.
+  // we wil just copy over the timestamp from the syslog
+  jsonObject.put("original_string", originalString);
+  jsonObject.put("timestamp", 
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --

Great point, latest commit has the change and the test


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

2018-08-29 Thread ottobackwards

Github user ottobackwards commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213706134
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  private transient SyslogParser syslogParser;
+
+  @Override
+  public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+  JSONObject jsonObject = new 
JSONObject(syslogParser.parseLine(originalString));
+
+  // be sure to put in the original string, and the timestamp.
+  // we wil just copy over the timestamp from the syslog
+  jsonObject.put("original_string", originalString);
+  jsonObject.put("timestamp", 
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --

Great point, latest commit has the change and the test


---

[jira] [Commented] (METRON-1740) Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596419#comment-16596419
 ] 

ASF GitHub Bot commented on METRON-1740:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1171#discussion_r213704549
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/README.md
 ---
@@ -0,0 +1,15 @@
+# BasicPaloAltoFirewallParser
--- End diff --

This needs a license header on line 1:
```

```


> Improve Palo Alto parser to handle CONFIG and SYSTEM syslog messages
> 
>
> Key: METRON-1740
> URL: https://issues.apache.org/jira/browse/METRON-1740
> Project: Metron
>  Issue Type: Improvement
>Reporter: Yi Liu
>Priority: Major
>
> As a Metron's user (security analyst)
> I would like Metron's Palo Alto parser be able to parse CONFIG and SYSTEM 
> PanOS syslog messages
> so that I can know what, when how the system configuration has been changed 
> and how the system has been running. 
>  
> The current PaloAlto parser (BasicPaloAltoFirewallParser) only supports 
> THREAT and TRAFFIC log messages. The task is to extend it to support CONFIG 
> and SYSTEM log messages. The supported PanOS versions are 6.1, 7.0 and 8.0.
> The sample of CONFIG log (PanOS 7.0)
> {code:java}
> 1,2017/08/11 11:23:36,,CONFIG,0,0,2017/08/11 
> 11:23:36,192.168.14.162,,edit,admin,Web,Succeeded, vsys  vsys4 rule X 
> rules  dev-to-dev-ext-http-https,1336,0x0,0,0,0,0,,dev-something200-01
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1171: METRON-1740 make parser support CONFIG and SYSTEM...

2018-08-29 Thread cestella

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1171#discussion_r213704549
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/paloalto/README.md
 ---
@@ -0,0 +1,15 @@
+# BasicPaloAltoFirewallParser
--- End diff --

This needs a license header on line 1:
```

```


---

[jira] [Commented] (METRON-1757) Storm Profiler Serialization Exception

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596413#comment-16596413
 ] 

ASF GitHub Bot commented on METRON-1757:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1178
  
This is really good work, @nickwallen.  I'm +1 by inspection here, pending 
@JonZeolla's +1


> Storm Profiler Serialization Exception
> --
>
> Key: METRON-1757
> URL: https://issues.apache.org/jira/browse/METRON-1757
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running the Storm Profiler with multiple workers this serialization 
> error can occur.
> Even in an environment with multiple workers It seems to not occur all the 
> time. It may depend on how the individual bolts are executed across the 
> workers in a cluster. This likely occurs only when the HBaseBolt is executed 
> on a separate worker from the ProfileBuilderBolt.
> It will never occur when running with a single worker.
> {code}
> 2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler 
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the 
> request
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing 
> no-arg constructor): 
> org.apache.metron.common.configuration.profiler.ProfileResult
> Serialization trace:
> result (org.apache.metron.common.configuration.profiler.ProfileConfig)
> definition (org.apache.metron.profiler.ProfileMeasurement)
>  at 
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078) 
> ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.received(Server.java:254) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
>

[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception

2018-08-29 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1178
  
This is really good work, @nickwallen.  I'm +1 by inspection here, pending 
@JonZeolla's +1


---

[jira] [Commented] (METRON-1751) Storm Profiler dies when consuming null message

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1751?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596408#comment-16596408
 ] 

ASF GitHub Bot commented on METRON-1751:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1176
  
+1 by inspection pending full-dev validation.


> Storm Profiler dies when consuming null message
> ---
>
> Key: METRON-1751
> URL: https://issues.apache.org/jira/browse/METRON-1751
> Project: Metron
>  Issue Type: Bug
>Reporter: Mohan
>Priority: Major
>
> When You publish a null message to the profiler input kafka topic which is 
> 'indexing' in my case I see the below exception messages on the worker log
> {code:java}
> 2018-08-27 12:46:03.825 o.a.s.util Thread-9-splitterBolt-executor[7 7] 
> [ERROR] Async loop died! java.lang.RuntimeException: 
> java.lang.NullPointerException at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:485)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:451)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.daemon.executor$fn__10195$fn__10208$fn__10263.invoke(executor.clj:855)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:484) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?] at 
> java.lang.Thread.run(Thread.java:748) [?:1.8.0_181] Caused by: 
> java.lang.NullPointerException at java.lang.String.(String.java:491) 
> ~[?:1.8.0_181] at 
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.doExecute(ProfileSplitterBolt.java:160)
>  ~[stormjar.jar:?] at 
> org.apache.metron.profiler.bolt.ProfileSplitterBolt.execute(ProfileSplitterBolt.java:145)
>  ~[stormjar.jar:?] at 
> org.apache.storm.daemon.executor$fn__10195$tuple_action_fn__10197.invoke(executor.clj:735)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.daemon.executor$mk_task_receiver$fn__10114.invoke(executor.clj:466)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.disruptor$clojure_handler$reify__4137.onEvent(disruptor.clj:40)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] at 
> org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:472)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292] ... 6 more
> {code}
> also the profiler dies , even  if you start up the profiler again and run it 
> through the same topic, it does always die.
> {code:java}
> 2018-08-27 12:46:03.870 o.a.s.util Thread-9-splitterBolt-executor[7 7] 
> [ERROR] Halting process: ("Worker died")
> java.lang.RuntimeException: ("Worker died")
> at org.apache.storm.util$exit_process_BANG_.doInvoke(util.clj:341) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.RestFn.invoke(RestFn.java:423) [clojure-1.7.0.jar:?]
> at 
> org.apache.storm.daemon.worker$fn__10799$fn__10800.invoke(worker.clj:763) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at 
> org.apache.storm.daemon.executor$mk_executor_data$fn__10011$fn__10012.invoke(executor.clj:276)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at org.apache.storm.util$async_loop$fn__1221.invoke(util.clj:494) 
> [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
> at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
> at java.lang.Thread.run(Thread.java:748) [?:1.8.0_181]
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1176: METRON-1751 Storm Profiler dies when consuming null mess...

2018-08-29 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1176
  
+1 by inspection pending full-dev validation.


---

[jira] [Commented] (METRON-1696) Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596405#comment-16596405
 ] 

ASF GitHub Bot commented on METRON-1696:


Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1134
  
Should this be closed out?


> Pcap parser fails to write pacap sequence file to hdfs on kerberized cluster 
> -
>
> Key: METRON-1696
> URL: https://issues.apache.org/jira/browse/METRON-1696
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Mohan
>Assignee: Mohan
>Priority: Major
>
> pcap parser fails to write the pcap sequence files to hdfs directory due to 
> insufficient privileges to hdfs folder for 'metron' user 
> {code:java}
> 2018-07-25 10:15:50.035 o.a.m.s.p.HDFSWriterCallback 
> Thread-9-kafkaSpout-executor[3 3] [ERROR] Permission denied: user=metron, 
> access=WRITE, 
> inode="/apps/metron/pcap/pcap_pcap_1532513746365022000_0_pcap-20-1532414055":hdfs:hdfs:drwxr-xr-x
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:353)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:325)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:246)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:190)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1950)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1934)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1917)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2767)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2702)
> at 
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:2586)
> at 
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.create(NameNodeRpcServer.java:736)
> at 
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.create(ClientNamenodeProtocolServerSideTranslatorPB.java:409)
> at 
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at 
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:640)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2351)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2347)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at 
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2347)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1134: METRON-1696: Create the HDFS directory for pcap sequence...

2018-08-29 Thread cestella

Github user cestella commented on the issue:

https://github.com/apache/metron/pull/1134
  
Should this be closed out?


---

[jira] [Commented] (METRON-1757) Storm Profiler Serialization Exception

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596394#comment-16596394
 ] 

ASF GitHub Bot commented on METRON-1757:


Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1178
  
Looks good.  I wasn't able to finish my testing last night but I am going 
to take a stab at validating.  Based on your comments above though this was 
great digging, thanks.


> Storm Profiler Serialization Exception
> --
>
> Key: METRON-1757
> URL: https://issues.apache.org/jira/browse/METRON-1757
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running the Storm Profiler with multiple workers this serialization 
> error can occur.
> Even in an environment with multiple workers It seems to not occur all the 
> time. It may depend on how the individual bolts are executed across the 
> workers in a cluster. This likely occurs only when the HBaseBolt is executed 
> on a separate worker from the ProfileBuilderBolt.
> It will never occur when running with a single worker.
> {code}
> 2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler 
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the 
> request
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing 
> no-arg constructor): 
> org.apache.metron.common.configuration.profiler.ProfileResult
> Serialization trace:
> result (org.apache.metron.common.configuration.profiler.ProfileConfig)
> definition (org.apache.metron.profiler.ProfileMeasurement)
>  at 
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078) 
> ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.received(Server.java:254) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>

[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception

2018-08-29 Thread JonZeolla

Github user JonZeolla commented on the issue:

https://github.com/apache/metron/pull/1178
  
Looks good.  I wasn't able to finish my testing last night but I am going 
to take a stab at validating.  Based on your comments above though this was 
great digging, thanks.


---

[jira] [Commented] (METRON-1757) Storm Profiler Serialization Exception

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596384#comment-16596384
 ] 

ASF GitHub Bot commented on METRON-1757:


Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1178
  
Sure. Done.


> Storm Profiler Serialization Exception
> --
>
> Key: METRON-1757
> URL: https://issues.apache.org/jira/browse/METRON-1757
> Project: Metron
>  Issue Type: Bug
>Reporter: Nick Allen
>Assignee: Nick Allen
>Priority: Major
>
> When running the Storm Profiler with multiple workers this serialization 
> error can occur.
> Even in an environment with multiple workers It seems to not occur all the 
> time. It may depend on how the individual bolts are executed across the 
> workers in a cluster. This likely occurs only when the HBaseBolt is executed 
> on a separate worker from the ProfileBuilderBolt.
> It will never occur when running with a single worker.
> {code}
> 2018-08-28 15:24:10.002 o.a.s.m.n.StormServerHandler 
> Netty-server-localhost-6700-worker-1 [ERROR] server errors in handling the 
> request
> com.esotericsoftware.kryo.KryoException: Class cannot be created (missing 
> no-arg constructor): 
> org.apache.metron.common.configuration.profiler.ProfileResult
> Serialization trace:
> result (org.apache.metron.common.configuration.profiler.ProfileConfig)
> definition (org.apache.metron.profiler.ProfileMeasurement)
>  at 
> com.esotericsoftware.kryo.Kryo$DefaultInstantiatorStrategy.newInstantiatorOf(Kryo.java:1272)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstantiator(Kryo.java:1078) 
> ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1087) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:570)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:546)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:711) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:551)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:793) 
> ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:134)
>  ~[kryo-3.0.3.jar:?]
>  at 
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:40)
>  ~[kryo-3.0.3.jar:?]
>  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:689) 
> ~[kryo-3.0.3.jar:?]
>  at 
> org.apache.storm.serialization.KryoValuesDeserializer.deserializeFrom(KryoValuesDeserializer.java:37)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.serialization.KryoTupleDeserializer.deserialize(KryoTupleDeserializer.java:50)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.DeserializingConnectionCallback.recv(DeserializingConnectionCallback.java:56)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.enqueue(Server.java:133) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at org.apache.storm.messaging.netty.Server.received(Server.java:254) 
> ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.messaging.netty.StormServerHandler.messageReceived(StormServerHandler.java:61)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
>  ~[storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
> org.apache.storm.shade.org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
>  [storm-core-1.1.0.2.6.5.0-292.jar:1.1.0.2.6.5.0-292]
>  at 
>

[GitHub] metron issue #1178: METRON-1757 Storm Profiler Serialization Exception

2018-08-29 Thread nickwallen

Github user nickwallen commented on the issue:

https://github.com/apache/metron/pull/1178
  
Sure. Done.


---

[jira] [Commented] (METRON-1743) CEF testPaloAltoCEF test using a confusing variable name

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596343#comment-16596343
 ] 

ASF GitHub Bot commented on METRON-1743:


Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1173


> CEF testPaloAltoCEF test using a confusing variable name
> 
>
> Key: METRON-1743
> URL: https://issues.apache.org/jira/browse/METRON-1743
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Jon Zeolla
>Assignee: Jon Zeolla
>Priority: Trivial
>
> A confusing test URL 
> here



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1173: METRON-1743: CEF test confusing URL

2018-08-29 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/metron/pull/1173


---

[jira] [Commented] (METRON-1476) Update angular

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596327#comment-16596327
 ] 

ASF GitHub Bot commented on METRON-1476:


Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1096
  
@sardell To get this running in full dev, I had to change a line in the 
metron.spec file for the RPMs to build properly.

`%attr(0644,root,root) %{metron_home}/web/alerts-ui/*.bundle.css` to 
`/usr/metron/0.5.1/web/alerts-ui/styles.*.css`. I assume this is a just a 
naming change, but I wanted to confirm that with you.


> Update angular
> --
>
> Key: METRON-1476
> URL: https://issues.apache.org/jira/browse/METRON-1476
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Major
>
> Update angular to speed up development



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron issue #1096: METRON-1476: Update angular

2018-08-29 Thread justinleet

Github user justinleet commented on the issue:

https://github.com/apache/metron/pull/1096
  
@sardell To get this running in full dev, I had to change a line in the 
metron.spec file for the RPMs to build properly.

`%attr(0644,root,root) %{metron_home}/web/alerts-ui/*.bundle.css` to 
`/usr/metron/0.5.1/web/alerts-ui/styles.*.css`. I assume this is a just a 
naming change, but I wanted to confirm that with you.


---

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596310#comment-16596310
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213670531
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  private transient SyslogParser syslogParser;
+
+  @Override
+  public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+  JSONObject jsonObject = new 
JSONObject(syslogParser.parseLine(originalString));
+
+  // be sure to put in the original string, and the timestamp.
+  // we wil just copy over the timestamp from the syslog
+  jsonObject.put("original_string", originalString);
+  jsonObject.put("timestamp", 
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --

Based on looking at the docs for the syslog library, it looks like it's 
possible to not have a timestamp (or to not validly parse a timestamp).  If we 
have a nil for timestamp here, we probably want to default like we do 
elsewhere, which is to current time.  What do you think?


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (METRON-1476) Update angular

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596309#comment-16596309
 ] 

ASF GitHub Bot commented on METRON-1476:


Github user ruffle1986 commented on the issue:

https://github.com/apache/metron/pull/1096
  
It has turned out that Angular's changed the way how it renders templates 
by default in version 6. Earlier, it preserved whitespaces by default but in 
version 6 the `preservedWhitespace` option is false by default when using the 
aot (head of time) compiler. Actually, it's a good thing because it reduces the 
size of the production bundle but during the regression test, we noticed a few 
differences in the layout before and after the version upgrade. So since we've 
been relying on whitespaces, we have to set `preserveWhitespace` true.

https://github.com/sardell/metron/pull/10


> Update angular
> --
>
> Key: METRON-1476
> URL: https://issues.apache.org/jira/browse/METRON-1476
> Project: Metron
>  Issue Type: Improvement
>Reporter: Daniel Toth
>Assignee: Daniel Toth
>Priority: Major
>
> Update angular to speed up development



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

2018-08-29 Thread cestella

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213670531
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  private transient SyslogParser syslogParser;
+
+  @Override
+  public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+  JSONObject jsonObject = new 
JSONObject(syslogParser.parseLine(originalString));
+
+  // be sure to put in the original string, and the timestamp.
+  // we wil just copy over the timestamp from the syslog
+  jsonObject.put("original_string", originalString);
+  jsonObject.put("timestamp", 
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --

Based on looking at the docs for the syslog library, it looks like it's 
possible to not have a timestamp (or to not validly parse a timestamp).  If we 
have a nil for timestamp here, we probably want to default like we do 
elsewhere, which is to current time.  What do you think?


---

[GitHub] metron issue #1096: METRON-1476: Update angular

2018-08-29 Thread ruffle1986

Github user ruffle1986 commented on the issue:

https://github.com/apache/metron/pull/1096
  
It has turned out that Angular's changed the way how it renders templates 
by default in version 6. Earlier, it preserved whitespaces by default but in 
version 6 the `preservedWhitespace` option is false by default when using the 
aot (head of time) compiler. Actually, it's a good thing because it reduces the 
size of the production bundle but during the regression test, we noticed a few 
differences in the layout before and after the version upgrade. So since we've 
been relying on whitespaces, we have to set `preserveWhitespace` true.

https://github.com/sardell/metron/pull/10


---

[GitHub] metron pull request #1175: METRON-1750 Metron Parser for valid RFC 5424 Sysl...

2018-08-29 Thread cestella

Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213669448
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  private transient SyslogParser syslogParser;
+
+  @Override
+  public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+  JSONObject jsonObject = new 
JSONObject(syslogParser.parseLine(originalString));
+
+  // be sure to put in the original string, and the timestamp.
+  // we wil just copy over the timestamp from the syslog
+  jsonObject.put("original_string", originalString);
+  jsonObject.put("timestamp", 
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --

If we aren't able to parse the timestamp here, I presume there will be an 
exception in the parser, right?  I just want to make sure there's no way for 
the parser to fail to return a timestamp.


---

[jira] [Commented] (METRON-1750) Create Parser for Syslog RFC 5424 Messages

2018-08-29 Thread ASF GitHub Bot (JIRA)



[ 
https://issues.apache.org/jira/browse/METRON-1750?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16596304#comment-16596304
 ] 

ASF GitHub Bot commented on METRON-1750:


Github user cestella commented on a diff in the pull request:

https://github.com/apache/metron/pull/1175#discussion_r213669448
  
--- Diff: 
metron-platform/metron-parsers/src/main/java/org/apache/metron/parsers/syslog/Syslog5424Parser.java
 ---
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.metron.parsers.syslog;
+
+import com.github.palindromicity.syslog.NilPolicy;
+import com.github.palindromicity.syslog.SyslogParser;
+import com.github.palindromicity.syslog.SyslogParserBuilder;
+import com.github.palindromicity.syslog.dsl.SyslogFieldKeys;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import org.apache.metron.parsers.BasicParser;
+import org.json.simple.JSONObject;
+
+
+
+/**
+ * Parser for well structured RFC 5424 messages.
+ */
+public class Syslog5424Parser extends BasicParser {
+  public static final String NIL_POLICY_CONFIG = "nilPolicy";
+  private transient SyslogParser syslogParser;
+
+  @Override
+  public void configure(Map config) {
+// Default to OMIT policy for nil fields
+// this means they will not be in the returned field set
+String nilPolicyStr = (String) 
config.getOrDefault(NIL_POLICY_CONFIG,NilPolicy.OMIT.name());
+NilPolicy nilPolicy = NilPolicy.valueOf(nilPolicyStr);
+syslogParser = new 
SyslogParserBuilder().withNilPolicy(nilPolicy).build();
+  }
+
+  @Override
+  public void init() {
+  }
+
+  @Override
+  @SuppressWarnings("unchecked")
+  public List parse(byte[] rawMessage) {
+try {
+  if (rawMessage == null || rawMessage.length == 0) {
+return null;
+  }
+
+  String originalString = new String(rawMessage);
+  JSONObject jsonObject = new 
JSONObject(syslogParser.parseLine(originalString));
+
+  // be sure to put in the original string, and the timestamp.
+  // we wil just copy over the timestamp from the syslog
+  jsonObject.put("original_string", originalString);
+  jsonObject.put("timestamp", 
jsonObject.get(SyslogFieldKeys.HEADER_TIMESTAMP.getField()));
--- End diff --

If we aren't able to parse the timestamp here, I presume there will be an 
exception in the parser, right?  I just want to make sure there's no way for 
the parser to fail to return a timestamp.


> Create Parser for Syslog RFC 5424 Messages
> --
>
> Key: METRON-1750
> URL: https://issues.apache.org/jira/browse/METRON-1750
> Project: Metron
>  Issue Type: Sub-task
>Reporter: Otto Fowler
>Assignee: Otto Fowler
>Priority: Major
>
> Create a Metron parser for working with valid RFC 5424 syslog messages, 
> including support for structured data



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

44 matches

Mail list logo