EandrewJones commented on issue #243: URL: https://github.com/apache/incubator-flagon-useralejs/issues/243#issuecomment-1107941300
@UncleGedd I am sending the logs directly from UserALE to a logstash endpoint. I do have one custom JS script in which I send custom logs via userale when triggered by an intersection observer event, but I was encountering this issue even prior to adding that. @poorejc Yes, it's very odd. I've been debugging this on an off since December and nothing has worked. > 1. You say that the nulls are "breaking" logStash Parsing. Can you clarify a bit what you mean [here](https://github.com/UMD-ARLIS/docker-elk/blob/flagon-configs)? Are you getting dropped logs? Or are you just getting errors like the legacy "grokparsefailure" from Logstash? The reason I thought/think userale was sending null events was because I did some testing and it didn't appear that any logs were being dropped. The message arrays with nulls get stored in ES still. I managed to parse, extract and calculate aggregate mouseover times from those in python and it matched exactly with my session behavior. But now I'm not sure whether the nulls are entirely missing logs. > 2. What version of Logstash are you using? 6.8 or 7+ (note that we'll be moving to 7.8 soon, and have a working backend that will be moved over to Apache--this is a different build than the previous (@apache-flagon repo) build. See here. I'm using logstash and ES 6.8. The ES instance is hosted through AWS opensearch service and I'm running my own logstash docker image on an ec2 instance. This problem also happened when I was running both locally in docker images. > 3. Are you using modified your own modified Logstash parsers/plugins? No. I'm essentially using the exact logstash.conf file as provided in your examples. I recently added another route and port to conditionally send another set of logs to a different ES index, but this issue long pre-dates that. > 4. Which version of UserALE.js you are using? 2.2.0 or testing at 2.3.0? If 2.30, did you notice the same behavior in 2.2.0? 2.2.0 > 5. Can you tell me a bit more about how you generated that first log. I'm guessing this is the view of the message from LogStash... There are some fields I'm not familiar with, e.g., postIds. logType=="raw", yet there are new fields added. Is this a custom log of sorts? or is there decoration through userale.map API? Any customer AuthHeaders added? Correct, that was clipped from logstash logs. So I user userale.map API to decoraate the logs with two custom fields, `isPost: boolean` and `postIds: string[] | number[]` using another function for extracting specific information from an element's datasets if they exist: ``` /** addPostInfo * A callback function to be passed to userale.map(...) that adds * post metadata to logs for post elements * * @param {Object} log UserALE log * @returns {Object} UserALE log */ function addPostInfo(log) { if (typeof log.path !== "undefined") { if (log.datasets.length > 0) { let postIds = postDetector.postIdExtractor(log.datasets); if (postIds.length > 0) { return Object.assign({}, log, { isPost: true, postIds: postIds, }); } } else { return Object.assign({}, log, { isPost: false, }); } } } window.userale.map(addPostInfo); ``` The parse failure might be related to those tags, but I can say that it is not consistently three nulls in the message array. It's usually 1-3. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@flagon.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org