Hi All,
In my mind, the proxy option is to encapsulate the implementation and more
likely force the IMetaStoreClient to have understanding about the
RetryingMetaStoreClient or HiveMetaStoreClientWithLocalCache. Which should
not be necessary all the time. IMetaStoreClient should be just an
abstraction of the top of the current one. The concrete implementation of
the interface needs to define if that would like to have retry or cache or
any other special functionality.
Let's imagine the following scenario. Okumin came back and created a purely
memory-based HMS. In this case the caching layer that is enforced by the
design is not necessary for his case. I believe that such capabilities
should be determined by the implemented MetaStoreClient itself. If the
newly created client needs Retry or LocalCache that could be possible if
that would be also implement their interface. Currently the MetaStoreClient
is just thinking about the current HMS and current HMS strategies. Which is
fine, till we do not open it. I think the abstraction should not be
enforced by implementation details like the Cache or Retry classes.
-Attila
On Thu, Oct 19, 2023 at 12:29 PM Stamatis Zampetakis
wrote:
> Hey Okumin,
>
> Thanks for picking up this ticket and driving it forward.
>
> I don't have a strong opinion between the two options.
>
> On the surface the factory option seems simpler and possibly more
> efficient but I am not sure if the changes under the PR are sufficient
> to cover all usages in Hive.
>
> On the other hand, the proxy option looks more cumbersome to configure
> but maybe it is easier to integrate with the existing plumbing of
> RetryingMetaStoreClient in various places.
>
> Best,
> Stamatis
>
>
> On Mon, Oct 16, 2023 at 11:00 AM Attila Turoczy
> wrote:
> >
> > Hi Okumin,
> >
> > I love this initiative. Especially every good platform should be
> pluggable.
> > In my mind the HMS should be just one option that the user can choose
> from.
> > Yes, that will be the default, but the world is way more open now, and we
> > need to provide the choice of freedom. If you or others want to choose a
> > different megastore it should be easy.
> >
> > Both option1 and option2 are acceptable. (Maybe the first one is easier,
> > just need another factory, which are so boring :) )
> >
> > Thank you for your PR and work. I will also check it soon.
> >
> > -Attila
> >
> > On Fri, Oct 13, 2023 at 5:04 PM Okumin wrote:
> >
> > > Hi,
> > >
> > > I'm working on introducing a feature to make IMetaStoreClient
> pluggable.
> > > I'm sending this e-mail to gather opinions in a visible manner because
> it
> > > has controversial points.
> > >
> > > Some Hive users need the feature in order to integrate Hive with a data
> > > catalog other than HMS. Although the original patch was submitted more
> than
> > > 7 years ago and many users have wanted it, it has not been merged yet.
> > > I revived the ticket and PR so that we can maintain or improve it
> within
> > > the community.
> > >
> > > - JIRA: https://issues.apache.org/jira/browse/HIVE-12679
> > > - PR: https://github.com/apache/hive/pull/
> > >
> > > I initially created the above PR based on the original design. That's
> > > because I think it is reasonable enough and I can see some users have
> > > already ported the patch for the past 7 years. But there are also other
> > > opinions to suggest other designs. This is a summary for easy catch-up.
> > >
> > > https://gist.github.com/okumin/30b058b14db1b099ba37ba7dc257fe8e
> > >
> > > If you are interested in this problem and you have any opinions,
> please put
> > > a comment on the Pull Request.
> > >
> > > Regards,
> > > Okumin
> > >
>
