pan3793 opened a new pull request, #6018: URL: https://github.com/apache/kyuubi/pull/6018
# :mag: Description ## Issue References ๐ <!-- Append the issue number after #. If there is no issue for you to link create one or --> <!-- If there are no issues to link, please provide details here. --> This pull request aims to speed up the GetTables operation for the Spark session catalog. As reported in https://github.com/apache/kyuubi/discussions/4956, https://github.com/apache/kyuubi/discussions/5949, the GetTables operation is quite slow in some cases, and in https://github.com/apache/kyuubi/pull/4444, `kyuubi.operation.getTables.ignoreTableProperties` was introduced to speed up the V2 catalog, but not covers session catalog. ## Describe Your Solution ๐ง Extend the scope of `kyuubi.operation.getTables.ignoreTableProperties` to cover the GetTables operation for the Spark session catalog. Currently, the basic step of GetTables in the Spark engine is ``` val catalog: String = getCatalog(spark, catalogName) val databases: Seq[String] = sessionCatalog.listDatabases(schemaPattern) val identifiers: Seq[TableIdentifier] = catalog.listTables(db, tablePattern, includeLocalTempViews = false) val tableObjects: Seq[CatalogTable] = catalog.getTablesByName(identifiers) ``` then filter `tableObjects` with `tableTypes: Set[String]`. The cost of `catalog.getTablesByName(identifiers)` is quite high when the table number is large, e.g. dozen thousand. For some cases, listing tables only for table name display, it is worth speeding up the operation while ignoring some properties and query criteria(specifically in this case, when `kyuubi.operation.getTables.ignoreTableProperties=true`, criteria `tableTypes` will be ignored, and all tables and views will be treated as TABLE to return.) ## Types of changes :bookmark: <!--- What types of changes does your code introduce? Put an `x` in all the boxes that apply: --> - [ ] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to change) ## Test Plan ๐งช - --- # Checklist ๐ <!--- Go over all the following points, and put an `x` in all the boxes that apply. --> <!--- If you're unsure about any of these, don't hesitate to ask. We're here to help! --> - [x] This patch was not authored or co-authored using [Generative Tooling](https://www.apache.org/legal/generative-tooling.html) **Be nice. Be informative.** -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
