muyihao commented on PR #11914:
URL: https://github.com/apache/hudi/pull/11914#issuecomment-2363683755
> @muyihao Thanks for pinning it out, the `metaClient.getActiveTimeline`
also choose to return the cached timeline first, for one metaClient instantce,
the list would happen only once, so I guess it is not a problem if the
metaClient can be shared for multiple input splits.
>
> BTW, while I reviewing the code, I found a bug for
`FileBasedInternalSchemaStorageManager`, the passed in metaClient does not
always work if the base path of it is not end up with `.schema`, we should fix
it like this:
>
> ```java
> public FileBasedInternalSchemaStorageManager(HoodieTableMetaClient
metaClient) {
> this.baseSchemaPath = new StoragePath(metaClient.getMetaPath(),
SCHEMA_NAME);
> this.storage = metaClient.getStorage();
> this.metaClient =
metaClient.getBasePath().getName().equalsIgnoreCase(SCHEMA_NAME) ? metaClient :
null;
> }
> ```
> @muyihao Thanks for pinning it out, the `metaClient.getActiveTimeline`
also choose to return the cached timeline first, for one metaClient instantce,
the list would happen only once, so I guess it is not a problem if the
metaClient can be shared for multiple input splits.
>
> BTW, while I reviewing the code, I found a bug for
`FileBasedInternalSchemaStorageManager`, the passed in metaClient does not
always work if the base path of it is not end up with `.schema`, we should fix
it like this:
>
> ```java
> public FileBasedInternalSchemaStorageManager(HoodieTableMetaClient
metaClient) {
> this.baseSchemaPath = new StoragePath(metaClient.getMetaPath(),
SCHEMA_NAME);
> this.storage = metaClient.getStorage();
> this.metaClient =
metaClient.getBasePath().getName().equalsIgnoreCase(SCHEMA_NAME) ? metaClient :
null;
> }
> ```
Thank you for your reply. I also tried to serialize the metaClient and put
it into the conf, but found that the StoragePath serialization fails.
<img width="1318" alt="image"
src="https://github.com/user-attachments/assets/627f22a3-fa52-4d8a-ba8d-25c4d21d40ef";>
do you have any better ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
