[GitHub] [spark] gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread

gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread URL: https://github.com/apache/spark/pull/26502#issuecomment-574251955 Yeah, the thread must be turned off not to have flakyness. `IS_TESTING` is something what I would

[GitHub] [spark] gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread

gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread URL: https://github.com/apache/spark/pull/26502#issuecomment-572029841 Started to catch-up which will take some time and going to continue...

[GitHub] [spark] gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread

gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread URL: https://github.com/apache/spark/pull/26502#issuecomment-561152339 > Other than maybe it should be by default just 1 thread. Set it to 1 + moved the `clean`

[GitHub] [spark] gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread

gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread URL: https://github.com/apache/spark/pull/26502#issuecomment-558158189 @zsxwing this is an addition to the source file archival/delete which worth the attention.

[GitHub] [spark] gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread

gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread URL: https://github.com/apache/spark/pull/26502#issuecomment-557134947 "source option" vs "SparkConf" is better I think.

[GitHub] [spark] gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread

gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread URL: https://github.com/apache/spark/pull/26502#issuecomment-556106371 retest this please This is an

[GitHub] [spark] gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread

gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread URL: https://github.com/apache/spark/pull/26502#issuecomment-553445387 cc @vanzin @HeartSaVioR since you know SPARK-20568 well.

[GitHub] [spark] gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread

gaborgsomogyi commented on issue #26502: [SPARK-29876][SS] Delete/archive file source completed files in separate thread URL: https://github.com/apache/spark/pull/26502#issuecomment-553398677 The reason why I've added number of threads configuration possibility is the archiving part.