holdenk commented on code in PR #54529:
URL: https://github.com/apache/spark/pull/54529#discussion_r2875354449
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/checkpointing/ChecksumCheckpointFileManager.scala:
##########
@@ -127,12 +128,14 @@ case class ChecksumFile(path: Path) {
* orphan checksum files. If using this, it is
your responsibility
* to clean up the potential orphan checksum
files.
* @param numThreads This is the number of threads to use for the thread pool,
for reading/writing
- * files. To avoid blocking, if the file manager instance is
being used by a
- * single thread, then you can set this to 2 (one thread for
main file, another
- * for checksum file).
- * If file manager is shared by multiple threads, you can
set it to
- * number of threads using file manager * 2.
- * Setting this differently can lead to file operation being
blocked waiting for
+ * files. Must be a non-negative integer.
+ * Setting this to 0 disables the thread pool and runs all
operations
+ * sequentially on the calling thread (no concurrency).
+ * To avoid blocking with a single concurrent caller, set
this to 2 (one thread
+ * for main file, another for checksum file).
+ * If file manager is shared by multiple threads, set it to
+ * number of concurrent callers * 2.
Review Comment:
Should this go over in the doc config? Also how would a user figure out how
many threads are going to be accessing it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
