poorejc commented on issue #243:
URL: 
https://github.com/apache/incubator-flagon-useralejs/issues/243#issuecomment-1107683022

   @EandrewJones Hi! Thanks for the ticket!
   
   A few more questions:
   
   1. You say that the nulls are "breaking" logStash Parsing. Can you clarify a 
bit what you mean there? Are you getting dropped logs? Or are you just getting 
errors like the legacy "grokparsefailure" from Logstash?
   
   2. What version of Logstash are you using? 6.8 or 7+ (note that we'll be 
moving to 7.8 soon, and have a working backend that will be moved over to 
Apache--this is a different build than the previous (@apache-flagon repo) 
build. See [here](https://github.com/UMD-ARLIS/docker-elk/blob/flagon-configs). 
   
   3. Are you using modified your own modified Logstash parsers/plugins?
   
   5. Which version of UserALE.js you are using? 2.2.0 or testing at 2.3.0? If 
2.30, did you notice the same behavior in 2.2.0?
   
   6. Can you tell me a bit more about how you generated that first log. I'm 
guessing this is the view of the message from LogStash... There are some fields 
I'm not familiar with, e.g., `postIds`. `logType`=="raw", yet there are new 
fields added. Is this a custom log of sorts? or is there decoration through 
userale.map API? Any customer AuthHeaders added?
   
   Regarding `null` and LogStash... We are aware that fields with `null` values 
aren't parsed, particularly in fields like `location` and `details`. Examples 
(I used 2.2.0 to generate this, as the `toolVersion` in your logs is 2.2.0):
   
   Here is a `submit` log from the Example page from the example server 
(server.js):
   
   ```
     {
       target: 'form#test_text_input',
       path: [ 'form#test_text_input', 'body', 'html', '#document', 'Window' ],
       pageUrl: 'file:///
       pageTitle: 'UserALE.js - Example Page',
       pageReferrer: '',
       browser: { browser: 'chrome', version: '100.0.4896' },
       clientTime: 1650762410279,
       microTime: 0.3,
       location: { x: null, y: null },
       scrnRes: { width: 1280, height: 577 },
       type: 'submit',
       logType: 'raw',
       userAction: true,
       details: null,
       userId: 'me',
       toolVersion: '2.2.0',
       toolName: 'Apache UserALE.js Example (Custom)',
       useraleVersion: '2.2.0',
       sessionID: 'session_1650636958755'
     }
   ```
   `details` and `location` both include `null` values.
   
   Here is a `submit` log in Elasticsearch (in JSON):
   ```
   {
     "_index": "userale",
     "_type": "_doc",
     "_id": "vGAKWYABe4GJJBS6pZlw",
     "_version": 1,
     "_score": null,
     "fields": {
       "logType": [
         "raw"
       ],
       "userAction": [
         true
       ],
       "microTime": [
         0
       ],
       "pageTitle": [
         "UserALE.js - Example Page"
       ],
       "patch_ver": [
         0
       ],
       "browser.browser": [
         "chrome"
       ],
       "sessionID": [
         "session_1650636958755"
       ],
       "type": [
         "submit"
       ],
       "clientTime": [
         "2022-04-24T00:48:31.457Z"
       ],
       "scrnRes.width": [
         1280
       ],
       "userId": [
         "me"
       ],
       "target": [
         "form#test_text_input"
       ],
       "scrnRes.height": [
         577
       ],
       "minor_ver": [
         2
       ],
       "path": [
         "form#test_text_input",
         "body",
         "html",
         "#document",
         "Window"
       ],
       "toolVersion": [
         "2.2.0"
       ],
       "browser.version": [
         "100.0.4896"
       ],
       "@timestamp": [
         "2022-04-24T00:48:31.495Z"
       ],
       "useraleVersion": [
         "2.2.0"
       ],
       "host": [
         "172.20.0.1"
       ],
       "major_ver": [
         2
       ],
       "pageUrl": [
         "file:///C:/
       ],
       "pageReferrer": [
         ""
       ],
       "toolName": [
         "Apache UserALE.js Example (Custom)"
       ]
     },
     "highlight": {
       "type": [
         "@kibana-highlighted-field@submit@/kibana-highlighted-field@"
       ]
     },
     "sort": [
       1650761311457
     ]
   }
   ```
   Those fields are noticeably absent, although we're still getting `submit` 
logs.
   
   I generated these from the exact same test page... ELK is dropping dropping 
fields with null values (which is an indexing issue--the value null doesn't 
conform to our 
[index](https://github.com/UMD-ARLIS/docker-elk/blob/flagon-configs/logstash/templates/userale.json).
 So far, this has been fine b/c if its null its not really essential to have 
that data (although can create issues in edge cases for back end analyses).
   
   The null values I see in your `logs` output are as expected. Also, I think 
@UncleGedd is right about the empty lines after `submit` events--`submit` 
events trigger a page reload that is likely to blame for that empty line given 
how server.js works. However, I would like to figure out your first example, 
though. I've never seen null keys like that before in a UserALE log--very 
interested in how it was created.
   
   Happy to help out! Please answer those questions above as they'll be useful 
to us in doing forensics. 
   
   Thanks again @EandrewJones !
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@flagon.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to