linhongliu-db commented on code in PR #59890:
URL: https://github.com/apache/spark/pull/59890#discussion_r4472729964
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/CacheManager.scala:
##########
@@ -672,19 +694,21 @@ class CacheManager extends Logging with
AdaptiveSparkPlanHelper {
/**
* Refresh the given [[FileIndex]] if one of its root paths is a
subdirectory of
* `qualifiedPath` and either `includeTimeTravel` is true or the index is
not an immutable
- * time-travel snapshot.
- * @return whether the [[FileIndex]] was refreshed.
+ * time-travel snapshot. Refresh each index at most once per recacheByPath
call.
+ * @return whether the [[FileIndex]] matches, including one already
refreshed during this call.
*/
private def refreshFileIndexIfNecessary(
fileIndex: FileIndex,
fs: FileSystem,
qualifiedPath: Path,
- includeTimeTravel: Boolean): Boolean = {
+ includeTimeTravel: Boolean,
+ refreshedIndexes: JSet[FileIndex]): Boolean = {
val needToRefresh = (includeTimeTravel || !fileIndex.isTimeTravel) &&
fileIndex.rootPaths
.map(_.makeQualified(fs.getUri, fs.getWorkingDirectory))
.exists(isSubDir(qualifiedPath, _))
- if (needToRefresh) fileIndex.refresh()
+ if (needToRefresh && refreshedIndexes.add(fileIndex)) fileIndex.refresh()
Review Comment:
Updated in 3f0b41990ca7: the refreshed-index membership check now runs
before root-path qualification and matching. The comment explains why a prior
match can return true immediately while still selecting every dependent cache
entry for rebuilding. All 11 refreshByPath tests and all 10 time-travel cache
tests pass, along with SQL source and test Scala style checks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
