Were you able to trace where the Sentry plugin was causing problems? Was it during the initial sync of ACLs, or updates of ACLs, or during permission lookups when accessing files?
Some time back, I dealt with a ~230M file cluster which was using Sentry. Accidentally, all the Sentry provided ACLs got copied down to the HDFS level. At that point, we noted a large increase in NN heap usage. I cannot recall exactly, but it was in the order of 10's of GB. Heap dumps showed it was down to the ACLs. Once we removed them at the HDFS level, heap usage returned to normal. The ACLs at the HDFS level, if you have a lot of them, will certainly add to NN heap pressure. On Sun, Oct 18, 2020 at 2:43 PM Xinli shang <sha...@uber.com.invalid> wrote: > We are using Apache Sentry. On the large scale of HDFS, which is our case, > we see a performance downgrade when enabling the Sentry plugin in > NameNode. So we have to disable the plugin in NN and map Sentry policies > to HDFS ACL. It works great so far. This is the only major issue we see. > > On Sun, Oct 18, 2020 at 1:19 AM Stephen O'Donnell > <sodonn...@cloudera.com.invalid> wrote: > > > I agree with Owen on this - I don't think this is a feature we should add > > to HDFS. > > > > If managing the permissions for Hive tables is becoming a big overhead > for > > you, you should look into something like Sentry. It allows you to manage > > the permissions of all the files and folders under Hive tables in a > > centralized place. It is also more memory efficient inside the namenode, > as > > it does not store an ACL object against each file. Sentry also allows for > > more than 32 ACLs, which is the normal HDFS limit. At Cloudera, we see a > > lot of clusters using Sentry to manage Hive table permissions. > > > > Sentry simply uses the existing HDFS Attribute Provider interface, so it > > theory, it would be fairly simple to create a plugin of your own to do > just > > what you need, but as Sentry exists and is fairly well proven in Hive > > environments already, it would be simpler to just use it. > > > > > > On Sat, Oct 17, 2020 at 3:23 PM Xinli shang <sha...@uber.com.invalid> > > wrote: > > > > > Hi Vinayakumar, > > > > > > The staging tables are dynamic. From the Hadoop security team > > perspective, > > > it is unrealistic to force every data writer to do that because they > are > > so > > > many and they write in different ways. > > > > > > Rename is just one scenario and there are other scenarios. For example, > > > when permission is changed, we need to apply that change to every file > > > today. If we can have that flag, we only change the table. or > > > partition directories. > > > > > > Xinli > > > > > > > > > On Sat, Oct 17, 2020 at 12:14 AM Vinayakumar B < > vinayakum...@apache.org> > > > wrote: > > > > > > > IIUC, hive renames are from hive’s staging directory during write to > > > final > > > > destination within table. > > > > > > > > Why not set the default ACLs of staging directory to whatever > expected, > > > and > > > > then continue write remaining files. > > > > > > > > In this way even after rename you will have expected ACLs on the > final > > > > files. > > > > > > > > Setting default ACLs on staging directory can be done using single > RPC. > > > > > > > > -Vinay > > > > > > > > On Sat, 17 Oct 2020 at 8:08 AM, Xinli shang <sha...@uber.com.invalid > > > > > > wrote: > > > > > > > > > Thanks Owen for your reply! As mentioned in the Jira, default ACLs > > > don't > > > > > apply to rename. Any idea how rename can work without setting ACLs > > per > > > > > file? > > > > > > > > > > On Fri, Oct 16, 2020 at 7:25 PM Owen O'Malley < > > owen.omal...@gmail.com> > > > > > wrote: > > > > > > > > > > > I'm very -1 on adding these semantics. > > > > > > > > > > > > When you create the table's directory, set the default ACL. That > > will > > > > > have > > > > > > exactly the effect that you are looking for without creating > > > additional > > > > > > semantics. > > > > > > > > > > > > .. Owen > > > > > > > > > > > > On Fri, Oct 16, 2020 at 7:02 PM Xinli shang > > <sha...@uber.com.invalid > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I opened https://issues.apache.org/jira/browse/HDFS-15638 and > > want > > > > to > > > > > > > collect feedback from the community. I know whenever changing > the > > > > > > > permission model that follows POSIX model is never a trivial > > > change. > > > > So > > > > > > > please comment on if you have concerns. For reading > convenience, > > > here > > > > > is > > > > > > a > > > > > > > copy of the ticket. > > > > > > > > > > > > > > *Problem*: Currently, when a user tries to accesses a file > he/she > > > > needs > > > > > > the > > > > > > > permissions of it's parent and ancestors and the permission of > > that > > > > > file. > > > > > > > This is correct generally, but for Hive tables > directories/files, > > > all > > > > > the > > > > > > > files under a partition or even a table usually have the same > > > > > permissions > > > > > > > for the same set of ACL groups. Although the permissions and > ACL > > > > groups > > > > > > are > > > > > > > the same, the writer still need to call setfacl() for every > file > > to > > > > add > > > > > > > LDAP groups. This results in a huge amount of RPC calls to NN. > > HDFS > > > > has > > > > > > > default ACL to solve that but that only applies to create and > > copy, > > > > but > > > > > > not > > > > > > > apply to rename. However, in Hive ETL, rename is very common. > > > > > > > > > > > > > > *Proposal*: Add a 1-bit flag to directory inodes to indicate > > > whether > > > > or > > > > > > not > > > > > > > it is a Hive table directory. If that flag is set, then all the > > > > > > > sub-directory and files under it will just use it's permission > > and > > > > ACL > > > > > > > groups settings. By doing this way, Hive ETL doesn't need to > set > > > > > > > permissions at the file level. If that flag is not set(by > > default), > > > > > work > > > > > > as > > > > > > > before. To set/unset that flag, it would require admin > privilege. > > > > > > > > > > > > > > -- > > > > > > > Xinli Shang > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > Xinli Shang > > > > > > > > > -- > > > > -Vinay > > > > > > > > > > > > -- > Xinli Shang >
