Right, we could use requesting_user for logging, statistics, etc, but it would be problematic to impersonate that user when loading metadata.
It's of course possible that I'm missing something here. On Thu, Jan 3, 2019 at 10:05 AM mhd wrk <mhdwrkoff...@gmail.com> wrote: > Thanks for the link. So the final answer is that even if the libhdfs bug > gets fixed there won't be any changes to Impala to expose requesting_user > in Catalog Service, right? > > On Thu, Jan 3, 2019 at 9:46 AM Tim Armstrong <tarmstr...@cloudera.com> > wrote: > >> > catalog server ignores file system authorization model >> The catalog daemon does this by design - the idea is that the catalog >> server can load and cache metadata on behalf of multiple users. It requires >> that the catalogd user (usually "impala") has permissions to read >> filesystem metadata. >> >> The "user account requirements" section in our docs explains this: >> https://impala.apache.org/docs/build/html/topics/impala_prereqs.html#prereqs >> and >> https://impala.apache.org/docs/build/html/topics/impala_security_files.html >> >> On Wed, Jan 2, 2019 at 5:52 PM mhd wrk <mhdwrkoff...@gmail.com> wrote: >> >>> it's more about enforcing Hadoop file system authorisation. All we have >>> done is implementing a custom Hadoop File System (org.apache.hadoop.fs. >>> FileSystem) and now trying to use Impala to query files hosted on that >>> file system and it fails because catalog server ignores file system >>> authorization model. The same file system works nicely with HDFS commands >>> (e.g. hdfs dfs -ls ...) as well as HiveServer. >>> >>> Our clients expect us to enforce authorization at all levels (HDFS, >>> Accumulo, Hive, Impala and ....) >>> >>> On Wed, Jan 2, 2019 at 4:56 PM Tim Armstrong <tarmstr...@cloudera.com> >>> wrote: >>> >>>> Stepping back for a second, doesn't what you're trying to do assume >>>> that each user will load metadata for each table separately? The whole >>>> point of the catalog server is that we load the metadata once and then >>>> share it between queries and users. >>>> >>>> I don't think we want to have the catalog server load different >>>> versions of a table depending on which user initially loaded the table? >>>> That would cause all sorts of issues. >>>> >>>> On Wed, Jan 2, 2019 at 12:36 PM mhd wrk <mhdwrkoff...@gmail.com> wrote: >>>> >>>>> I see. I was wondering how it works inside hive server. Basically this >>>>> is a HDFS C API issue. Thanks for the elaborate explanation. >>>>> >>>>> On Wed, Jan 2, 2019 at 12:27 PM Shant Hovsepian <sh...@arcadiadata.com> >>>>> wrote: >>>>> >>>>>> Problem is mostly with libhdfs as documented here HADOOP-12953 >>>>>> >>>>>> On a kerberized setup the service principal gets picked up. There are >>>>>> work arounds in the Java HDFS API but the c based one in libhdfs has this >>>>>> issue. Of course caching HDFS will b trickier in impala as well but first >>>>>> his one API in libhdfs needs to be enhanced. >>>>>> >>>>>> Also in general having database authorization at the file level may >>>>>> not be a good idea or clean design and using sentry and extending it's >>>>>> authorization mecuanisms would be cleaner. >>>>>> >>>>>> -Shant >>>>>> >>>>>> On Wed, Jan 2, 2019, 12:21 PM mhd wrk <mhdwrkoff...@gmail.com> wrote: >>>>>> >>>>>>> Thanks for further info. Not sure if our Product Management is OK, >>>>>>> at this point, with us patching Impala server to get our solution >>>>>>> working. >>>>>>> Our product is supposed to work with already installed servers. >>>>>>> >>>>>>> Any plans to address the gap (making requesting_user visible inside >>>>>>> catalog server) in future release? >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Jan 2, 2019 at 11:50 AM Bharath Vissapragada < >>>>>>> bhara...@cloudera.com> wrote: >>>>>>> >>>>>>>> I was poking around in the code and it looks like we have most of >>>>>>>> the code in place >>>>>>>> <https://github.com/apache/impala/blob/27577dd652554dda5a03016e2d1e3ab66fe6b1f5/common/thrift/CatalogService.thrift#L47> >>>>>>>> >>>>>>>> // Common header included in all CatalogService requests. >>>>>>>> // TODO: The CatalogServiceVersion/protocol version should be part >>>>>>>> of the header. >>>>>>>> // This would require changes in BDR and break their compatibility >>>>>>>> story. We should >>>>>>>> // coordinate a joint change somewhere down the line. >>>>>>>> struct TCatalogServiceRequestHeader { >>>>>>>> // The effective user who submitted this request. >>>>>>>> 1: optional string requesting_user >>>>>>>> } >>>>>>>> >>>>>>>> That header is included in all the RPCs. However, that is an >>>>>>>> optional field and may not be in a few places (since we don't actually >>>>>>>> rely >>>>>>>> on that currently). So you could start with making it a "required" >>>>>>>> field >>>>>>>> and see what all breaks. HTH. >>>>>>>> >>>>>>>> On Wed, Jan 2, 2019 at 11:35 AM Bharath Vissapragada < >>>>>>>> bhara...@cloudera.com> wrote: >>>>>>>> >>>>>>>>> I think we expose it via UDF effective_user() (effective user >>>>>>>>> could be different from the connected if delegation/doas is enabled). >>>>>>>>> You >>>>>>>>> can run a query like "select effective_user()" in a session. >>>>>>>>> >>>>>>>>> You can also look it up in the /sessions page on the coordinator >>>>>>>>> web UI (<coordinator>:25000/sessions?json) and you can get a json >>>>>>>>> formatted >>>>>>>>> string containing the connected and delegate user for each session. >>>>>>>>> >>>>>>>>> If you want it on the Catalog side, you probably have to plumb it >>>>>>>>> through the RPC calls (change the thrift spec and pass it along from >>>>>>>>> the >>>>>>>>> coordinator session handling code to the Catalog RPC code). >>>>>>>>> >>>>>>>>> On Wed, Jan 2, 2019 at 11:19 AM mhd wrk <mhdwrkoff...@gmail.com> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Is there any Impala/Sentry specific API we can use inside our >>>>>>>>>> code to figure out who current user is? >>>>>>>>>> >>>>>>>>>> On Wed, Jan 2, 2019 at 11:12 AM Bharath Vissapragada < >>>>>>>>>> bhara...@cloudera.com> wrote: >>>>>>>>>> >>>>>>>>>>> Yes. I think Jeszy is right. Per my understanding too, we don't >>>>>>>>>>> impersonate the client user on the Catalog server. Instead, we >>>>>>>>>>> enforce the >>>>>>>>>>> authorization via Sentry during query planning. >>>>>>>>>>> >>>>>>>>>>> On Wed, Jan 2, 2019 at 7:06 AM mhd wrk <mhdwrkoff...@gmail.com> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> IMPALA-2177 sounds like the correct issue. >>>>>>>>>>>> Here are log messages from authentication.cc for impalad and >>>>>>>>>>>> catalogd respectively: >>>>>>>>>>>> >>>>>>>>>>>> I0102 14:15:06.722666 28195 authentication.cc:478] Successfully >>>>>>>>>>>>> authenticated client user *"ad...@example.com >>>>>>>>>>>>> <ad...@example.com>"* >>>>>>>>>>>>> I0102 03:40:07.972348 27948 authentication.cc:445] >>>>>>>>>>>>> Successfully authenticated principal *"impala/cdh-...@example.com >>>>>>>>>>>>> <cdh-...@example.com>"* on an internal connection >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> As you can see from the messages above, impalad is able to >>>>>>>>>>>> identify the currently connected user correctly. However catalogd >>>>>>>>>>>> always >>>>>>>>>>>> authenticates as impala which causes the problem. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Wed, Jan 2, 2019 at 4:19 AM Jeszy <jes...@gmail.com> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Hey, >>>>>>>>>>>>> >>>>>>>>>>>>> IIUC your question correctly, this is a limitation. >>>>>>>>>>>>> IMPALA-2177 looks >>>>>>>>>>>>> to be the appropriate jira. >>>>>>>>>>>>> Most users use Impala together with Sentry, where the >>>>>>>>>>>>> recommended >>>>>>>>>>>>> approach is to disable impersonation (even in services that >>>>>>>>>>>>> allow it, >>>>>>>>>>>>> like Hive). >>>>>>>>>>>>> >>>>>>>>>>>>> HTH >>>>>>>>>>>>> >>>>>>>>>>>>> On Wed, 2 Jan 2019 at 05:55, Bharath Vissapragada < >>>>>>>>>>>>> bhara...@cloudera.com> wrote: >>>>>>>>>>>>> > >>>>>>>>>>>>> > Hi, >>>>>>>>>>>>> > >>>>>>>>>>>>> > Can you add the stack trace here if possible? It is not >>>>>>>>>>>>> super clear where exactly the problem is. >>>>>>>>>>>>> > >>>>>>>>>>>>> > Thanks, >>>>>>>>>>>>> > Bharath >>>>>>>>>>>>> > >>>>>>>>>>>>> > On Tue, Jan 1, 2019 at 6:34 PM mhd wrk < >>>>>>>>>>>>> mhdwrkoff...@gmail.com> wrote: >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> we have our own implementation of Hadoop FileSystem which >>>>>>>>>>>>> relies on current user in a kerberosied environment to locate >>>>>>>>>>>>> user specific >>>>>>>>>>>>> files in HDFS. This custom file system works fine inside hive to >>>>>>>>>>>>> create >>>>>>>>>>>>> external tables and query them. However trying to access the same >>>>>>>>>>>>> tables >>>>>>>>>>>>> via Impala (jdbc driver) fails. Watching the log messages seems >>>>>>>>>>>>> that when >>>>>>>>>>>>> impalad sends requests to catalogd to get meta data of a given >>>>>>>>>>>>> table the >>>>>>>>>>>>> current user returned by UserGroupInformation is the service >>>>>>>>>>>>> account >>>>>>>>>>>>> running the server (impala/hostn...@example.com) instead of >>>>>>>>>>>>> the currently connected user. >>>>>>>>>>>>> >> >>>>>>>>>>>>> >> Is this a known issue or limitation of Impala? >>>>>>>>>>>>> >>>>>>>>>>>>
