chamikaramj commented on code in PR #41248:
URL: https://github.com/apache/beam/pull/41248#discussion_r4622481667


##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -520,7 +520,9 @@ public void processElement(
       if (newSpec != null) {
         ReadableState<SerializableTableSpec> existingState = 
cacheState.get(tableId);
         SerializableTableSpec existingSpec = existingState != null ? 
existingState.read() : null;
-        if (existingSpec == null || isNewer(newSpec, existingSpec)) {
+        // Keep the cached spec only if it's newer. A poll of an unchanged 
table still brings the
+        // storage credentials the catalog just vended, and the cached ones 
may be about to expire.
+        if (existingSpec == null || !isNewer(existingSpec, newSpec)) {

Review Comment:
   Probably more robust to add an explicit `getPolledAtMillis` field and use 
that instead of depending on the element order when comparing two specs within 
the same pane.



##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableTableSpec.java:
##########
@@ -267,6 +292,62 @@ public FileIO getFileIO(@Nullable Configuration conf) {
     return local;
   }
 
+  private SharedFileIO acquireSharedFileIO(
+      KV<String, @Nullable List<String>> key, @Nullable Configuration conf) {

Review Comment:
   Seems like this shared `FileIO` will always use the first callers 
`Configuration`.  Can this result in a race when two transforms with different 
`config_properties` produce identical `FileIO` configs ?



##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableTableSpec.java:
##########
@@ -105,6 +112,13 @@ public abstract class SerializableTableSpec implements 
Serializable {
   @SchemaFieldNumber("12")
   public abstract long getLastUpdatedMillis();
 
+  /**
+   * The storage credentials the table's {@link FileIO} holds, such as those a 
REST catalog vends,
+   * as {@link CredentialParser} JSON.
+   */
+  @SchemaFieldNumber("13")
+  public abstract @Nullable List<String> getStorageCredentialJsons();

Review Comment:
   Will any `toString()` calls leak credentials now, for example 
`SideInputTable.toString()` ? If so consider retracting that info from 
`toString()` calls.



##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/TableMetadataDriver.java:
##########
@@ -520,7 +520,9 @@ public void processElement(
       if (newSpec != null) {
         ReadableState<SerializableTableSpec> existingState = 
cacheState.get(tableId);
         SerializableTableSpec existingSpec = existingState != null ? 
existingState.read() : null;
-        if (existingSpec == null || isNewer(newSpec, existingSpec)) {
+        // Keep the cached spec only if it's newer. A poll of an unchanged 
table still brings the
+        // storage credentials the catalog just vended, and the cached ones 
may be about to expire.
+        if (existingSpec == null || !isNewer(existingSpec, newSpec)) {

Review Comment:
   Also, probably missing some tests here that uses more than one snapshot that 
differ only in credentials.



##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/SerializableTableSpec.java:
##########
@@ -245,6 +266,10 @@ public FileIO getFileIO() {
   /**
    * Returns a cached {@link FileIO} instance for this table, configured with 
the provided Hadoop
    * {@link Configuration} if supported.
+   *
+   * <p>Specs with the same {@link #getFileIoJson()} and storage credentials 
share one instance,
+   * because each side input update decodes new copies of every spec. The 
shared instance is closed
+   * once no spec that uses it is reachable.

Review Comment:
   Add more details on what the caller can/should do to keep it reachable ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to