[
https://issues.apache.org/jira/browse/SPARK-34107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17264672#comment-17264672
]
Shashank Pedamallu commented on SPARK-34107:
--------------------------------------------
Screenshot of the spark history:
!blank_shs.png!
Also, please find attached the dynamic tracing analysis (using
[btrace|[http://example.com|https://github.com/btraceio/btrace]][^SHS_Profiling_Sorted.csv]
> Spark History not loading when service has to load 300k applications
> initially from S3
> --------------------------------------------------------------------------------------
>
> Key: SPARK-34107
> URL: https://issues.apache.org/jira/browse/SPARK-34107
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
> Affects Versions: 3.0.0
> Reporter: Shashank Pedamallu
> Priority: Major
> Attachments: SHS_Profiling_Sorted.csv, blank_shs.png
>
>
> Spark History Service is having trouble loading when loading initially with
> 300k+ applications from S3. Following are the details and snapshots:
> Number of files in `spark.history.fs.logDirectory`: (Using xxx for anonymity)
> {noformat}
> spedamallu@spedamallu-mbp143 ~/src/spark (spark-bug) $
> | => aws s3 ls s3://xxxx-company/spark-history-fs-logDirectory/ | wc -l
> 305571
> spedamallu@spedamallu-mbp143 ~/src/spark (spark-bug) ${noformat}
> {noformat}
> Logs when starting SparkHistory:
> {noformat}
> root@shs-with-statsd-86d7f54679-t8fqr:/go/src/github.com/xxxx-company/spark-private#
>
> /go/src/github.com/xxxx-company/spark-private/bootstrap/start-history-server.sh
> --properties-file /etc/spark-history-config/shs-default.properties
> 2021/01/14 02:40:28 Spark spark wrapper is disabled
> 2021/01/14 02:40:28 Attempt number 0, Max attempts 0, Left Attempts 0
> 2021/01/14 02:40:28 Statsd disabled
> 2021/01/14 02:40:28 Debug log: /tmp/.log
> 2021/01/14 02:40:28 Job submitted 0 seconds ago, Operator 0, ETL 0, Flyte 0
> Mozart 0
> 2021/01/14 02:40:28 Running command /opt/spark/bin/spark-class.orig with
> arguments [org.apache.spark.deploy.history.HistoryServer --properties-file
> /etc/spark-history-config/shs-default.properties]
> 21/01/14 02:40:29 INFO HistoryServer: Started daemon with process name:
> 2077@shs-with-statsd-86d7f54679-t8fqr
> 21/01/14 02:40:29 INFO SignalUtils: Registered signal handler for TERM
> 21/01/14 02:40:29 INFO SignalUtils: Registered signal handler for HUP
> 21/01/14 02:40:29 INFO SignalUtils: Registered signal handler for INT
> 21/01/14 02:40:30 WARN NativeCodeLoader: Unable to load native-hadoop
> library for your platform... using builtin-java classes where applicable
> 21/01/14 02:40:30 INFO SecurityManager: Changing view acls to: root
> 21/01/14 02:40:30 INFO SecurityManager: Changing modify acls to: root
> 21/01/14 02:40:30 INFO SecurityManager: Changing view acls groups to:
> 21/01/14 02:40:30 INFO SecurityManager: Changing modify acls groups to:
> 21/01/14 02:40:30 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(root); groups
> with view permissions: Set(); users with modify permissions: Set(root);
> groups with modify permissions: Set()
> 21/01/14 02:40:30 INFO FsHistoryProvider: History server ui acls disabled;
> users with admin permissions: ; groups with admin permissions
> 21/01/14 02:40:30 WARN MetricsConfig: Cannot locate configuration: tried
> hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
> 21/01/14 02:40:30 INFO MetricsSystemImpl: Scheduled Metric snapshot period
> at 10 second(s).
> 21/01/14 02:40:30 INFO MetricsSystemImpl: s3a-file-system metrics system
> started
> 21/01/14 02:40:31 INFO log: Logging initialized @1933ms to
> org.sparkproject.jetty.util.log.Slf4jLog
> 21/01/14 02:40:31 INFO Server: jetty-9.4.z-SNAPSHOT; built:
> 2019-04-29T20:42:08.989Z; git: e1bc35120a6617ee3df052294e433f3a25ce7097; jvm
> 1.8.0_242-b08
> 21/01/14 02:40:31 INFO Server: Started @1999ms
> 21/01/14 02:40:31 INFO AbstractConnector: Started ServerConnector@51751e5f
> {HTTP/1.1,[http/1.1]} {0.0.0.0:18080}
> 21/01/14 02:40:31 INFO Utils: Successfully started service on port 18080.
> 21/01/14 02:40:31 INFO ContextHandler: Started
> o.s.j.s.ServletContextHandler@b9dfc5a
> {/,null,AVAILABLE,@Spark}
> 21/01/14 02:40:31 INFO ContextHandler: Started
> o.s.j.s.ServletContextHandler@1bbae752
> {/json,null,AVAILABLE,@Spark}
> 21/01/14 02:40:31 INFO ContextHandler: Started
> o.s.j.s.ServletContextHandler@5cf87cfd
> {/api,null,AVAILABLE,@Spark}
> 21/01/14 02:40:31 INFO ContextHandler: Started
> o.s.j.s.ServletContextHandler@74971ed9
> {/static,null,AVAILABLE,@Spark}
> 21/01/14 02:40:31 INFO ContextHandler: Started
> o.s.j.s.ServletContextHandler@1542af63
> {/history,null,AVAILABLE,@Spark}
> 21/01/14 02:40:31 INFO HistoryServer: Bound HistoryServer to 0.0.0.0, and
> started at http://shs-with-statsd-86d7f54679-t8fqr:18080
> 21/01/14 02:40:31 DEBUG FsHistoryProvider: Scheduling update thread every 10
> seconds
> 21/01/14 02:40:31 DEBUG FsHistoryProvider: Scanning
> s3a://xxxx-company/spark-history-fs-logDirectory/ with
> lastScanTime==-1{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
