[
https://issues.apache.org/jira/browse/LENS-910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15093417#comment-15093417
]
Hadoop QA commented on LENS-910:
--------------------------------
Applied patch:
[LENS-910-5.patch|https://issues.apache.org/jira/secure/attachment/12781745/LENS-910-5.patch]
and ran command: mvn clean install -fae. Result: Success. Build Job:
https://builds.apache.org/job/PreCommit-Lens-Build/248/
> Add session config to skip filtering cube related tables from all the tables
> in a database
> ------------------------------------------------------------------------------------------
>
> Key: LENS-910
> URL: https://issues.apache.org/jira/browse/LENS-910
> Project: Apache Lens
> Issue Type: Improvement
> Reporter: Deepak Barr
> Assignee: Deepak Barr
> Priority: Minor
> Attachments: LENS-910-5.patch
>
>
> {code:title=CubeMetastoreServiceImpl.java|borderStyle=solid}
> private List<String> getNativeTablesFromDB(LensSessionHandle sessionid,
> String dbName, boolean prependDbName)
> throws LensException {
> IMetaStoreClient msc = null;
> try {
> msc = getSession(sessionid).getMetaStoreClient();
> List<String> tables = msc.getAllTables(
> dbName);
> List<String> result = new ArrayList<String>();
> if (tables != null && !tables.isEmpty()) {
> List<org.apache.hadoop.hive.metastore.api.Table> tblObjects =
> msc.getTableObjectsByName(dbName, tables);
> Iterator<org.apache.hadoop.hive.metastore.api.Table> it =
> tblObjects.iterator();
> while (it.hasNext()) {
> org.apache.hadoop.hive.metastore.api.Table tbl = it.next();
> if (tbl.getParameters().get(MetastoreConstants.TABLE_TYPE_KEY) ==
> null) {
> if (prependDbName) {
> result.add(dbName + "." + tbl.getTableName());
> } else {
> result.add(tbl.getTableName());
> }
> }
> }
> }
> return result;
> } catch (Exception e) {
> throw new LensException("Error getting native tables from DB", e);
> }
> }
> {code}
> We have approx. 18K tables in one of our hive databases. When fetching native
> tables, the getNativeTablesFromDB() filters out the cube related tables from
> the superset using table property(cube.table.type). This filtering is taking
> a long time for our heavy databases. The call to metastore API
> [msc.getTableObjectsByName()] is taking most of the time. Since, the cube
> related tables may not be present in some/most databases, the filtering is
> not necessary for them. I think we can add a session property, so that a user
> can skip filtering if he wants to. Thoughts ?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
