Github user vijoshi commented on the issue:
https://github.com/apache/spark/pull/15410
1. display "Last updated <date-time>" with a generic message, possibly
omitting the count of logs
Ok, while looking at this I notice that the scan date-time the
FsHistoryProvider currently internally captures `lastScanTime` is based on the
filesystem date-time:
`
private def getNewLastScanTime(): Long = {
val fileName = "." + UUID.randomUUID().toString
val path = new Path(logDir, fileName)
val fos = fs.create(path)
try {
fos.close()
fs.getFileStatus(path).getModificationTime
`
This probably made sense when earlier versions of FsHistoryProvider used
this value to determine new/updated logs it needed to replay. This is no longer
the case. In `checkForLogs()` the check is based on file size now. So my
question is that the fact that `getNewLastScanTime` is still present - is there
a reason to rely on or use the filesystem time value? We don't get a timezone
out of the `FileStatus` api. If I were to display an "Updated as of: " value
on the app listing page, I would rather read the value from the JVM and ensure
the Web UI converted it to user's local timezone for display.
2. Display applications that are pending, instead of a count
Though agree that this would be better UX-wise, but we would only have the
filenames (`logInfos`) to go by to build this list until a clean way to
extract app metadata exists without parsing event log file contents. If the UI
just displayed the pending ones is a different table as "names" of pending logs
without asserting that they are necessarily the app-id (though presently the
names may match the app-ids) - would that be good enough ? The message on the
top could link to a new page listing pending logs to avoid cluttering the
current page?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
