EandrewJones commented on issue #243:
URL: 
https://github.com/apache/incubator-flagon-useralejs/issues/243#issuecomment-1107941300

   @UncleGedd I am sending the logs directly from UserALE to a logstash 
endpoint. I do have one custom JS script in which I send custom logs via 
userale when triggered by an intersection observer event, but I was 
encountering this issue even prior to adding that.
   
   @poorejc Yes, it's very odd. I've been debugging this on an off since 
December and nothing has worked.
   
   
   > 1. You say that the nulls are "breaking" logStash Parsing. Can you clarify 
a bit what you mean 
[here](https://github.com/UMD-ARLIS/docker-elk/blob/flagon-configs)? Are you 
getting dropped logs? Or are you just getting errors like the legacy 
"grokparsefailure" from Logstash?
   
   The reason I thought/think userale was sending null events was  because I 
did some testing and it didn't appear that any logs were being dropped. The 
message arrays with nulls get stored in ES still. I managed to parse, extract 
and calculate aggregate mouseover times from those in python and it matched 
exactly with my session behavior. But now I'm not sure whether the nulls are 
entirely missing logs.
   
   > 2. What version of Logstash are you using? 6.8 or 7+ (note that we'll be 
moving to 7.8 soon, and have a working backend that will be moved over to 
Apache--this is a different build than the previous (@apache-flagon repo) 
build. See here.
   
   I'm using logstash and ES 6.8. The ES instance is hosted through AWS 
opensearch service and I'm running my own logstash docker image on an ec2 
instance. This problem also happened when I was running both locally in docker 
images. 
   
   > 3. Are you using modified your own modified Logstash parsers/plugins?
   
   No. I'm essentially using the exact logstash.conf file as provided in your 
examples. I recently added another route and port to conditionally send another 
set of logs to a different ES index, but this issue long pre-dates that.
   
   > 4. Which version of UserALE.js you are using? 2.2.0 or testing at 2.3.0? 
If 2.30, did you notice the same behavior in 2.2.0?
   
   2.2.0
   
   > 5. Can you tell me a bit more about how you generated that first log. I'm 
guessing this is the view of the message from LogStash... There are some fields 
I'm not familiar with, e.g., postIds. logType=="raw", yet there are new fields 
added. Is this a custom log of sorts? or is there decoration through 
userale.map API? Any customer AuthHeaders added?
   
   Correct, that was clipped from logstash logs. So I user userale.map API to  
decoraate the logs with two custom fields, `isPost: boolean` and  `postIds: 
string[] | number[]` using another function for extracting specific information 
from an element's datasets if they exist:
   ```
     /** addPostInfo
      * A callback function to be passed to userale.map(...) that adds
      * post metadata to logs for post elements
      *
      * @param {Object} log UserALE log
      * @returns {Object} UserALE log
      */
     function addPostInfo(log) {
       if (typeof log.path !== "undefined") {
         if (log.datasets.length > 0) {
           let postIds = postDetector.postIdExtractor(log.datasets);
           if (postIds.length > 0) {
             return Object.assign({}, log, {
               isPost: true,
               postIds: postIds,
             });
           }
         } else {
           return Object.assign({}, log, {
             isPost: false,
           });
         }
       }
     }
   
   window.userale.map(addPostInfo);
   ```
   
   The parse failure might be related to those tags, but I can say that it is 
not consistently three nulls in the message array. It's usually 1-3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: dev-unsubscr...@flagon.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org

Reply via email to