[
https://issues.apache.org/jira/browse/SPARK-14561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17346563#comment-17346563
]
Tianbin Jiang commented on SPARK-14561:
---------------------------------------
"If you are going to use s3 as the dest, and you want to see incomplete apps,
then you'll need to configure the spark job to have smaller partition size (64?
128? MB)." can anyone explain more on this? Which configuration maps to
partition size: https://spark.apache.org/docs/2.4.5/configuration.html
> History Server does not see new logs in S3
> ------------------------------------------
>
> Key: SPARK-14561
> URL: https://issues.apache.org/jira/browse/SPARK-14561
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 1.6.1
> Reporter: Miles Crawford
> Priority: Major
> Labels: bulk-closed
>
> If you set the Spark history server to use a log directory with an s3a://
> url, everything appears to work fine at first, but new log files written by
> applications are not picked up by the server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
