[
https://issues.apache.org/jira/browse/SPARK-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15572401#comment-15572401
]
Steve Loughran commented on SPARK-14561:
----------------------------------------
To clarify: it's not changes in existing files that aren't showing up, *it is
new files added to the same destination directory*
If that's the case, something is up with the scanning
#. set the logging of org.apache.spark.deploy.history.FsHistoryProvider to
debug
# have a look at the scan interval. Is it too long?
> History Server does not see new logs in S3
> ------------------------------------------
>
> Key: SPARK-14561
> URL: https://issues.apache.org/jira/browse/SPARK-14561
> Project: Spark
> Issue Type: Bug
> Affects Versions: 1.6.1
> Reporter: Miles Crawford
>
> If you set the Spark history server to use a log directory with an s3a://
> url, everything appears to work fine at first, but new log files written by
> applications are not picked up by the server.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
