Thanks Stephen for sharing out! I don't have those details as our benchmarking test was done quite a while ago and I wasn't a participant. But using Sentry does add one more service to HDFS critical path which will add overhead more or less especially for reliability. In a long run, we consider completely removing Sentry because we have company-wide generic policy stores, and using Sentry forces us to do the conversion from authoritative to Sentry police store, which introduces some problems.
For your point of "*The ACLs at the HDFS level, if you have a lot of them, will certainly add to NN heap pressure*", that is another reason why I propose this HDFS-15638 to use table-level directory ACLs instead of file-level. It seems GCP/GCS already has the prefix-based permission granting. Check it out here https://cloud.google.com/iam/docs/configuring-resource-based-access . Xinli On Mon, Oct 19, 2020 at 2:14 AM Stephen O'Donnell <sodonn...@cloudera.com.invalid> wrote: > Were you able to trace where the Sentry plugin was causing problems? Was it > during the initial sync of ACLs, or updates of ACLs, or during permission > lookups when accessing files? > > Some time back, I dealt with a ~230M file cluster which was using Sentry. > Accidentally, all the Sentry provided ACLs got copied down to the HDFS > level. At that point, we noted a large increase in NN heap usage. I cannot > recall exactly, but it was in the order of 10's of GB. Heap dumps showed it > was down to the ACLs. Once we removed them at the HDFS level, heap usage > returned to normal. > > The ACLs at the HDFS level, if you have a lot of them, will certainly add > to NN heap pressure. > > On Sun, Oct 18, 2020 at 2:43 PM Xinli shang <sha...@uber.com.invalid> > wrote: > > > We are using Apache Sentry. On the large scale of HDFS, which is our > case, > > we see a performance downgrade when enabling the Sentry plugin in > > NameNode. So we have to disable the plugin in NN and map Sentry policies > > to HDFS ACL. It works great so far. This is the only major issue we see. > > > > On Sun, Oct 18, 2020 at 1:19 AM Stephen O'Donnell > > <sodonn...@cloudera.com.invalid> wrote: > > > > > I agree with Owen on this - I don't think this is a feature we should > add > > > to HDFS. > > > > > > If managing the permissions for Hive tables is becoming a big overhead > > for > > > you, you should look into something like Sentry. It allows you to > manage > > > the permissions of all the files and folders under Hive tables in a > > > centralized place. It is also more memory efficient inside the > namenode, > > as > > > it does not store an ACL object against each file. Sentry also allows > for > > > more than 32 ACLs, which is the normal HDFS limit. At Cloudera, we see > a > > > lot of clusters using Sentry to manage Hive table permissions. > > > > > > Sentry simply uses the existing HDFS Attribute Provider interface, so > it > > > theory, it would be fairly simple to create a plugin of your own to do > > just > > > what you need, but as Sentry exists and is fairly well proven in Hive > > > environments already, it would be simpler to just use it. > > > > > > > > > On Sat, Oct 17, 2020 at 3:23 PM Xinli shang <sha...@uber.com.invalid> > > > wrote: > > > > > > > Hi Vinayakumar, > > > > > > > > The staging tables are dynamic. From the Hadoop security team > > > perspective, > > > > it is unrealistic to force every data writer to do that because they > > are > > > so > > > > many and they write in different ways. > > > > > > > > Rename is just one scenario and there are other scenarios. For > example, > > > > when permission is changed, we need to apply that change to every > file > > > > today. If we can have that flag, we only change the table. or > > > > partition directories. > > > > > > > > Xinli > > > > > > > > > > > > On Sat, Oct 17, 2020 at 12:14 AM Vinayakumar B < > > vinayakum...@apache.org> > > > > wrote: > > > > > > > > > IIUC, hive renames are from hive’s staging directory during write > to > > > > final > > > > > destination within table. > > > > > > > > > > Why not set the default ACLs of staging directory to whatever > > expected, > > > > and > > > > > then continue write remaining files. > > > > > > > > > > In this way even after rename you will have expected ACLs on the > > final > > > > > files. > > > > > > > > > > Setting default ACLs on staging directory can be done using single > > RPC. > > > > > > > > > > -Vinay > > > > > > > > > > On Sat, 17 Oct 2020 at 8:08 AM, Xinli shang > <sha...@uber.com.invalid > > > > > > > > wrote: > > > > > > > > > > > Thanks Owen for your reply! As mentioned in the Jira, default > ACLs > > > > don't > > > > > > apply to rename. Any idea how rename can work without setting > ACLs > > > per > > > > > > file? > > > > > > > > > > > > On Fri, Oct 16, 2020 at 7:25 PM Owen O'Malley < > > > owen.omal...@gmail.com> > > > > > > wrote: > > > > > > > > > > > > > I'm very -1 on adding these semantics. > > > > > > > > > > > > > > When you create the table's directory, set the default ACL. > That > > > will > > > > > > have > > > > > > > exactly the effect that you are looking for without creating > > > > additional > > > > > > > semantics. > > > > > > > > > > > > > > .. Owen > > > > > > > > > > > > > > On Fri, Oct 16, 2020 at 7:02 PM Xinli shang > > > <sha...@uber.com.invalid > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I opened https://issues.apache.org/jira/browse/HDFS-15638 > and > > > want > > > > > to > > > > > > > > collect feedback from the community. I know whenever changing > > the > > > > > > > > permission model that follows POSIX model is never a trivial > > > > change. > > > > > So > > > > > > > > please comment on if you have concerns. For reading > > convenience, > > > > here > > > > > > is > > > > > > > a > > > > > > > > copy of the ticket. > > > > > > > > > > > > > > > > *Problem*: Currently, when a user tries to accesses a file > > he/she > > > > > needs > > > > > > > the > > > > > > > > permissions of it's parent and ancestors and the permission > of > > > that > > > > > > file. > > > > > > > > This is correct generally, but for Hive tables > > directories/files, > > > > all > > > > > > the > > > > > > > > files under a partition or even a table usually have the same > > > > > > permissions > > > > > > > > for the same set of ACL groups. Although the permissions and > > ACL > > > > > groups > > > > > > > are > > > > > > > > the same, the writer still need to call setfacl() for every > > file > > > to > > > > > add > > > > > > > > LDAP groups. This results in a huge amount of RPC calls to > NN. > > > HDFS > > > > > has > > > > > > > > default ACL to solve that but that only applies to create and > > > copy, > > > > > but > > > > > > > not > > > > > > > > apply to rename. However, in Hive ETL, rename is very common. > > > > > > > > > > > > > > > > *Proposal*: Add a 1-bit flag to directory inodes to indicate > > > > whether > > > > > or > > > > > > > not > > > > > > > > it is a Hive table directory. If that flag is set, then all > the > > > > > > > > sub-directory and files under it will just use it's > permission > > > and > > > > > ACL > > > > > > > > groups settings. By doing this way, Hive ETL doesn't need to > > set > > > > > > > > permissions at the file level. If that flag is not set(by > > > default), > > > > > > work > > > > > > > as > > > > > > > > before. To set/unset that flag, it would require admin > > privilege. > > > > > > > > > > > > > > > > -- > > > > > > > > Xinli Shang > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Xinli Shang > > > > > > > > > > > -- > > > > > -Vinay > > > > > > > > > > > > > > > > > > -- > > Xinli Shang > > > -- Xinli Shang
