Yeah, my old prototype was catalog metadata only, and written as Java
code. However, I don't think exposing a C++ API would require
significant changes to the existing external data source architecture.
In fact, it should be reasonably straightforward because there is
already a C++ data source scan node class which just calls directly
into Java right now, but that could be made conditional on the data
source extension that was loaded.
On Wed, Feb 7, 2018 at 11:23 PM, Tim Armstrong <tarmstr...@cloudera.com> wrote:
> Kathy's patch was exposing metrics, which is fairly different because it was
> exposing metrics from the backend. The java data source would have just had
> to call back into C++. It also needed to run on every backend, so would have
> required extending the data source API. That might be an argument for a C++
> data source API (or at least a well-defined interface for implementing
> system tables as classes in C++, which it was heading towards).
> IIRC MJ had a very old prototype of using the external data source to expose
> local catalog state (which is all Java).
> On Wed, Feb 7, 2018 at 11:38 AM, Daniel Hecht <dhe...@cloudera.com> wrote:
>> On Wed, Feb 7, 2018 at 9:18 AM, Marcel Kornacker <marc...@gmail.com>
>>> What is gained by removing the code?
>> The motivation was that we should always be looking for opportunities to
>> remove code, since we can't just keep adding or else the complexity of
>> Impala will get out of control over the long term. This was an opportunity
>> to remove ~5k lines, which is a nice benefit if the feature wasn't being
>> Regarding information schema, I definitely agree that is an interesting
>> and useful feature. There was a prototype of parts of that here
>> https://gerrit.cloudera.org/#/c/3863/19 but it did not rely on the external
>> data source. I don't remember whether there was a technical reason why and
>> that's something we can try to understand to see if external datasource
>> needs to evolve or we'd be better starting from scratch.
>> Shant, thanks for chiming in with your take as well and that you have
>> found it useful and are in favor of keeping it. And Matt, thanks for some
>> of the historical context and your opinion. Of course, feel free to