> On Oct. 19, 2018, 6:46 p.m., Sergio Pena wrote: > > I did a quick look and I feel this logging will be too verbose and may slow > > the fetch process as there is one message per task executed. There can be > > thousands or even millions of tasks (one per databse, one per table, one > > per 100 partitions). > > > > Is there another way to print a progress instead of two messages per task?
Sergio we don't do one per table. We do 1 per 100 tables. If a customer had 100 database, each database had 100 tables (10000) tables and each table had 100 partitions (10000*1000 = 10000000 10 million) we would have a total of 100 + 100 + 100 = 300 tasks. So its not that many. Also we need to know what task is taking how much time. I will post a snippet of log messages - Arjun ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/69076/#review209797 ----------------------------------------------------------- On Oct. 18, 2018, 9:29 p.m., Arjun Mishra wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/69076/ > ----------------------------------------------------------- > > (Updated Oct. 18, 2018, 9:29 p.m.) > > > Review request for sentry, kalyan kumar kalvagadda, Na Li, and Sergio Pena. > > > Repository: sentry > > > Description > ------- > > When sentry is fetching snapshot from HMS, it should log periodically on > where it stands in the snapshot process. This will help person debugging it > and help him understand the progress. > > > > This is important as this process could take magnitude of minutes when the > HMS data is huge. > > > Diffs > ----- > > > sentry-service/sentry-service-server/src/main/java/org/apache/sentry/api/service/thrift/SentryMetrics.java > 214d78c53 > > sentry-service/sentry-service-server/src/main/java/org/apache/sentry/service/thrift/FullUpdateInitializer.java > 3e27d1bbe > > sentry-service/sentry-service-server/src/main/java/org/apache/sentry/service/thrift/SentryHMSClient.java > 4baeb6725 > > > Diff: https://reviews.apache.org/r/69076/diff/2/ > > > Testing > ------- > > > Thanks, > > Arjun Mishra > >
