Jungtaek Lim created SPARK-24485:
------------------------------------
Summary: Measure and log elapsed time for filesystem operations in
HDFSBackedStateStoreProvider
Key: SPARK-24485
URL: https://issues.apache.org/jira/browse/SPARK-24485
Project: Spark
Issue Type: Improvement
Components: Structured Streaming
Affects Versions: 2.4.0
Reporter: Jungtaek Lim
There're couple of operations which communicate with file system (mostly remote
HDFS in production) in HDFSBackedStateStoreProvider, which contribute huge part
of latency.
It would be better to measure the latency (elapsed time) and log to help
debugging when there's unexpected huge latency on state store.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
