[
https://issues.apache.org/jira/browse/FLINK-30126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17638440#comment-17638440
]
melin commented on FLINK-30126:
-------------------------------
Spark also does not support dynamic catalog, which is created through aop
interception. Can flink do the same? flink does not have a spark.yarn.dist.jars
similar parameter. In the yarn environment, it is convenient to load
aspectjweaver.jar.
{code:java}
@Around("execution(public *
org.apache.spark.sql.connector.catalog.CatalogManager.catalog(..)) &&
args(catalogName)")
public Object aroundCatalog(ProceedingJoinPoint pjp, String catalogName) throws
Throwable {
synchronized (SparkCatalogAspectj.class) {
if (catalogName.equalsIgnoreCase(SESSION_CATALOG_NAME)) {
return pjp.proceed();
}
CatalogPlugin catalog = catalogs.get(catalogName);
if (catalog != null) {
return catalog;
}
DataSourceEntity dataSource =
dataSourceService.queryDataSource(catalogName);
if (dataSource != null) {
LOG.info("init catalog {}", catalogName);
catalog = CatalogUtils.buildJdbcCatalogPlugin(dataSource,
tableService, catalogName);
catalogs.put(catalogName, catalog);
return catalog;
} else {
throw new CatalogNotFoundException("catalog not exists: " +
catalogName);
}
}
} {code}
!image-2022-11-25-09-12-59-477.png!
[https://github.com/melin/spark-jobserver/blob/master/jobserver-admin/src/main/java/io/github/melin/spark/jobserver/deployment/AbstractDriverDeployer.java]
> Delay registration of the catalog, register the catalog as needed
> -----------------------------------------------------------------
>
> Key: FLINK-30126
> URL: https://issues.apache.org/jira/browse/FLINK-30126
> Project: Flink
> Issue Type: New Feature
> Reporter: melin
> Priority: Major
> Attachments: image-2022-11-25-09-12-59-477.png
>
>
> Data platform has registered many relational database data sources such as
> mysql, data source code is used as the catalog name, we are not sure which
> data source needs to register the catalog in flink, we hope that the required
> catalog can be dynamically loaded when sql is executed, flink provides the
> interface. Users can customize the registration catalog
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
