yyanyy commented on code in PR #57799:
URL: https://github.com/apache/spark/pull/57799#discussion_r3738751217
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/RelationResolution.scala:
##########
@@ -312,12 +320,14 @@ class RelationResolution(
// `Table`.
val sharedRelationCacheMatch = for {
t <- table
- if finalTimeTravelSpec.isEmpty && writePrivileges == null &&
!u.isStreaming
+ if pinnedTable.isEmpty && finalTimeTravelSpec.isEmpty &&
+ writePrivileges == null && !u.isStreaming
cached <- lookupSharedRelationCache(catalog, ident, t)
Review Comment:
Offline conversation for posterity: we were worried that taking all options
into consideration for looking up shared relation cache could cause
non-deterministic behavior when users specify queries against tables in one
single spark session, where some queries have the exact option list matching
shared relation cache, while some others don't; in this case user could observe
different queries using different table versions and cause confusion. One query
referring to the same table with different non-state-options also will be
undeterministic and depend on the order of specifying such tables. (There's
test cases covering scenarios mentioned above.) Also, we were not able to think
of a concrete use case where the non-state options need to participate in the
selection for the table from shared relation cache, as it was used for
obtaining the table without a table reload; the actual reusing of the cache the
data, do consider all options correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
