FANNG1 commented on issue #11027: URL: https://github.com/apache/gravitino/issues/11027#issuecomment-4439483886
> > Proposed fix: Override loadTable() in LanceTableOperations to implement a self-healing mechanism: when lance.create-empty=true is detected, read the actual schema from the Lance dataset on S3, update the Gravitino entity store with the > > correct columns, and clear the lance.create-empty flag. This is a one-time self-heal — subsequent loadTable() calls return the correct metadata via the normal path. > > This may be a workaround for this problem, and I believe this is the design flaw of lance-spark and the lance namespace spec > > * I do not see any deletion operations when `createEmpty` succeeds, and creating a real Lance table fails. In such a case, the orphan table will remain in the Gravitino table. > * The Lance Spark and Lance namespace spec does not design a metadata write-back mechanism to make sure the schema in the namespace implementation and the Lance location are consistent. > > In all, we may try to optimize the API design of Lance space and Lance-spark. If this can't be achieved in a short time, the workaround is also acceptable for me. [@FANNG1](https://github.com/FANNG1) do you have any thought on it? If would be great if lance namespace support write table schema to namespace impls, if not, we may have to query the table schema from underlying storage, maybe leverage cache to improve the performance -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
