leekeiabstraction commented on code in PR #191:
URL:
https://github.com/apache/flink-connector-aws/pull/191#discussion_r2033639231
##########
flink-catalog-aws-glue/README.md:
##########
@@ -0,0 +1,389 @@
+# Flink AWS Glue Catalog Connector
+
+The Flink AWS Glue Catalog connector provides integration between Apache Flink
and the AWS Glue Data Catalog. This connector enables Flink applications to use
AWS Glue as a metadata catalog for tables, databases, and schemas, allowing
seamless SQL queries against AWS resources.
+
+## Features
+
+- Register AWS Glue as a catalog in Flink applications
+- Access Glue databases and tables through Flink SQL
+- Support for various AWS data sources (S3, Kinesis, MSK)
+- Mapping between Flink and AWS Glue data types
+- Compatibility with Flink's Table API and SQL interface
+
+## Prerequisites
+
+Before getting started, ensure you have the following:
+
+- **Apache Flink 1.20+** installed or running in a managed environment
+- **AWS account** with appropriate permissions for AWS Glue and other required
services
+- **AWS credentials** properly configured
+- **Java 11** or later
+- **Maven 3.8+** for building the project
+
+## Getting Started
+
+### 1. Add Dependency
+
+Add the AWS Glue Catalog connector to your Flink project:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-aws-glue-catalog</artifactId>
+ <version>${flink.version}</version>
+</dependency>
+```
+
+### 2. Configure AWS Credentials
+
+Ensure AWS credentials are configured using one of these methods:
+
+- Environment variables
+- AWS credentials file
+- IAM roles (for applications running on AWS)
+
+### 3. Register the Glue Catalog
+
+You can register the AWS Glue catalog using either the Table API or SQL:
+
+#### Using Table API (Java/Scala)
+
+```java
+// Java/Scala
+import org.apache.flink.table.catalog.glue.GlueCatalog;
+import org.apache.flink.table.catalog.Catalog;
+
+// Create Glue catalog instance
+Catalog glueCatalog = new GlueCatalog(
+ "glue_catalog", // Catalog name
+ "default", // Default database
+ "us-east-1"); // AWS region
+
+
+// Register with table environment
+tableEnv.registerCatalog("glue_catalog", glueCatalog);
+tableEnv.useCatalog("glue_catalog");
+```
+
+#### Using Table API (Python)
+
+```python
+# Python
+from pyflink.table.catalog import GlueCatalog
+
+# Create and register Glue catalog
+glue_catalog = GlueCatalog(
+ "glue_catalog", # Catalog name
+ "default", # Default database
+ "us-east-1") # AWS region
+
+t_env.register_catalog("glue_catalog", glue_catalog)
+t_env.use_catalog("glue_catalog")
+```
+
+#### Using SQL
+
+In the Flink SQL Client, create and use the Glue catalog:
+
+```sql
+-- Create a catalog using Glue
+CREATE CATALOG glue_catalog WITH (
+ 'type' = 'glue',
+ 'catalog-name' = 'glue_catalog',
+ 'default-database' = 'default',
+ 'region' = 'us-east-1'
+);
+
+-- Use the created catalog
+USE CATALOG glue_catalog;
+
+-- Use a specific database
+USE default;
+```
+
+### 4. Create or Reference Glue Tables
+
+Once the catalog is registered, you can create new tables or reference
existing ones:
+
+```sql
+-- Create a new table in Glue
+CREATE TABLE customer_table (
+ id BIGINT,
+ name STRING,
+ region STRING
+) WITH (
+ 'connector' = 'kinesis',
+ 'stream' = 'customer-stream',
+ 'aws.region' = 'us-east-1',
+ 'format' = 'json'
+);
+
+-- Query existing Glue table
+SELECT * FROM glue_catalog.sales_db.order_table;
+```
+
+## Catalog Operations
+
+The AWS Glue Catalog connector supports several catalog operations through
SQL. Here's a list of the operations that are currently implemented:
+
+### Database Operations
+
+```sql
+-- Create a new database
+CREATE DATABASE sales_db;
+
+-- Create a database with comment
+CREATE DATABASE sales_db COMMENT 'Database for sales data';
+
+-- Create a database if it doesn't exist
+CREATE DATABASE IF NOT EXISTS sales_db;
+
+-- Drop a database
+DROP DATABASE sales_db;
+
+-- Drop a database if it exists
+DROP DATABASE IF EXISTS sales_db;
+
+-- Use a specific database
+USE sales_db;
+```
+
+### Table Operations
+
+```sql
+-- Create a table
+CREATE TABLE orders (
+ order_id BIGINT,
+ customer_id BIGINT,
+ order_date TIMESTAMP,
+ amount DECIMAL(10, 2)
+);
+
+-- Create a table with comment and properties
+CREATE TABLE orders (
+ order_id BIGINT,
+ customer_id BIGINT,
+ order_date TIMESTAMP,
+ amount DECIMAL(10, 2),
+ PRIMARY KEY (order_id) NOT ENFORCED
+) COMMENT 'Table storing order information'
+WITH (
+ 'connector' = 's3',
+ 'path' = 's3://bucket/path/to/orders',
+ 'format' = 'parquet'
+);
+
+-- Create table if not exists
+CREATE TABLE IF NOT EXISTS orders (
+ order_id BIGINT,
+ customer_id BIGINT
+);
+
+-- Drop a table
+DROP TABLE orders;
+
+-- Drop a table if it exists
+DROP TABLE IF EXISTS orders;
+
+-- Show table details
+DESCRIBE orders;
+```
+
+### View Operations
+
+```sql
+-- Create a view
+CREATE VIEW order_summary AS
+SELECT customer_id, COUNT(*) as order_count, SUM(amount) as total_amount
+FROM orders
+GROUP BY customer_id;
+
+-- Create a temporary view (only available in current session)
+CREATE TEMPORARY VIEW temp_view AS
+SELECT * FROM orders WHERE amount > 100;
+
+-- Drop a view
+DROP VIEW order_summary;
+
+-- Drop a view if it exists
+DROP VIEW IF EXISTS order_summary;
+```
+
+### Function Operations
+
+```sql
+-- Register a function
+CREATE FUNCTION multiply_func AS 'com.example.functions.MultiplyFunction';
+
+-- Register a temporary function
+CREATE TEMPORARY FUNCTION temp_function AS
'com.example.functions.TempFunction';
+
+-- Drop a function
+DROP FUNCTION multiply_func;
+
+-- Drop a temporary function
+DROP TEMPORARY FUNCTION temp_function;
+```
+
+### Listing Resources
+
+Query available catalogs, databases, and tables:
+
+```sql
+-- List all catalogs
+SHOW CATALOGS;
+
+-- List databases in the current catalog
+SHOW DATABASES;
+
+-- List tables in the current database
+SHOW TABLES;
+
+-- List tables in a specific database
+SHOW TABLES FROM sales_db;
+
+-- List views in the current database
+SHOW VIEWS;
+
+-- List functions
+SHOW FUNCTIONS;
+```
+
+### Cross-Catalog Queries
+
+Access tables from different catalogs in the same query:
+
+```sql
+-- Join tables from different catalogs
Review Comment:
This example joins on customer id match, IIUC, glue shouldn't keep actual
values but rather metadata on schema. Should we have an example that joins on a
more concrete example e.g. field name match?
##########
flink-catalog-aws-glue/dependency-reduced-pom.xml:
##########
@@ -0,0 +1,207 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>flink-data-catalog</artifactId>
+ <version>1.0</version>
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>3.8.1</version>
Review Comment:
Is there a motivation for specifying 3.8.1 here? (Similarly for shade plugin
and flink dependencies)
The pom.xml at repo root already defines build plugins. Should we point to
pom.xml at repo root as parent and rely on convention over configuration?
```
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
</plugin>
```
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/operations/GlueDatabaseOperations.java:
##########
@@ -0,0 +1,145 @@
+package com.amazonaws.services.msf.operations;
+
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.exceptions.*;
Review Comment:
Let's avoid using wildcard imports
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/StreamingJob.java:
##########
@@ -0,0 +1,286 @@
+package com.amazonaws.services.msf;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.catalog.Catalog;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StreamingJob {
+
+ // Logger to log messages for debugging or info
+ private static final Logger LOG =
LoggerFactory.getLogger(StreamingJob.class);
+
+ public static void main(String[] args) throws Exception {
+ // Create a new Flink configuration object
+ Configuration configuration = new Configuration();
+
+ // Create a local streaming execution environment with web UI for
monitoring
+ StreamExecutionEnvironment env =
StreamExecutionEnvironment.createLocalEnvironmentWithWebUI(configuration);
+
+ // Create the TableEnvironment using the provided execution
environment and settings
+ TableEnvironment tEnv = StreamTableEnvironment.create(
+ env, EnvironmentSettings.newInstance().build());
+//
+// // Create a GlueCatalog for connecting Flink with AWS Glue
+// Catalog glueCatalog = new GlueCatalog("glue_catalog", "default",
"us-east-1");
+//
+// // Register the Glue catalog in the TableEnvironment so that it can
be accessed
+// tEnv.registerCatalog("glue_catalog", glueCatalog);
+//
+// // Set the registered Glue catalog as the default catalog
+// tEnv.useCatalog("glue_catalog");
Review Comment:
Let's remove commented codes if they are not needed.
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/operations/GlueDatabaseOperations.java:
##########
@@ -0,0 +1,145 @@
+package com.amazonaws.services.msf.operations;
+
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.exceptions.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.*;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Handles all database-related operations for the Glue catalog.
+ * Provides functionality for listing, retrieving, creating, and deleting
databases in AWS Glue.
+ */
+public class GlueDatabaseOperations extends AbstractGlueOperations {
+
+ /** Logger for logging database operations. */
+ private static final Logger LOG =
LoggerFactory.getLogger(GlueDatabaseOperations.class);
+
+ /**
+ * Constructor for GlueDatabaseOperations.
+ * Initializes the Glue client and catalog name.
+ *
+ * @param glueClient The Glue client to interact with AWS Glue.
+ * @param catalogName The name of the catalog.
+ */
+ public GlueDatabaseOperations(GlueClient glueClient, String catalogName) {
+ super(glueClient, catalogName);
+ }
+
+ /**
+ * Lists all the databases in the Glue catalog.
+ *
+ * @return A list of database names.
+ * @throws CatalogException if there is an error fetching the list of
databases.
+ */
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ LOG.debug("Listing databases from Glue catalog");
+ return
glueClient.getDatabases(GetDatabasesRequest.builder().build())
+ .databaseList()
+ .stream()
+ .map(Database::name)
+ .collect(Collectors.toList());
+ } catch (GlueException e) {
+ LOG.error("Failed to list databases in Glue", e);
+ throw new CatalogException("Failed to list databases: " +
e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Retrieves the specified database from the Glue catalog.
+ *
+ * @param databaseName The name of the database to fetch.
+ * @return The CatalogDatabase object representing the Glue database.
+ * @throws DatabaseNotExistException If the database does not exist in the
Glue catalog.
+ * @throws CatalogException If there is any error retrieving the database.
+ */
+ public CatalogDatabase getDatabase(String databaseName) throws
DatabaseNotExistException, CatalogException {
+ try {
+ GetDatabaseResponse response = glueClient.getDatabase(
+ GetDatabaseRequest.builder()
+ .name(databaseName)
+ .build()
+ );
+
+ Database glueDatabase = response.database();
+ return convertGlueDatabase(glueDatabase);
+ } catch (EntityNotFoundException e) {
+ throw new DatabaseNotExistException(catalogName, databaseName);
+ } catch (Exception e) {
+ throw new CatalogException("Error fetching database: " +
databaseName, e);
+ }
+ }
+
+ /**
+ * Converts the Glue database model to a Flink CatalogDatabase.
+ *
+ * @param glueDatabase The Glue database model.
+ * @return A CatalogDatabase representing the Glue database.
+ */
+ private CatalogDatabase convertGlueDatabase(Database glueDatabase) {
Review Comment:
nit/style: method is only called from `getDatabase`, this can be inlined.
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/operations/GlueFunctionsOperations.java:
##########
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package com.amazonaws.services.msf.operations;
+
+import com.amazonaws.services.msf.util.GlueCatalogConstants;
+import com.amazonaws.services.msf.util.GlueFunctionsUtil;
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.CatalogFunctionImpl;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.FunctionAlreadyExistException;
+import org.apache.flink.table.catalog.exceptions.FunctionNotExistException;
+import org.apache.flink.table.resource.ResourceUri;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.AlreadyExistsException;
+import
software.amazon.awssdk.services.glue.model.CreateUserDefinedFunctionRequest;
+import
software.amazon.awssdk.services.glue.model.CreateUserDefinedFunctionResponse;
+import
software.amazon.awssdk.services.glue.model.DeleteUserDefinedFunctionRequest;
+import
software.amazon.awssdk.services.glue.model.DeleteUserDefinedFunctionResponse;
+import software.amazon.awssdk.services.glue.model.EntityNotFoundException;
+import
software.amazon.awssdk.services.glue.model.GetUserDefinedFunctionRequest;
+import
software.amazon.awssdk.services.glue.model.GetUserDefinedFunctionResponse;
+import
software.amazon.awssdk.services.glue.model.GetUserDefinedFunctionsRequest;
+import
software.amazon.awssdk.services.glue.model.GetUserDefinedFunctionsResponse;
+import software.amazon.awssdk.services.glue.model.GlueException;
+import software.amazon.awssdk.services.glue.model.PrincipalType;
+import
software.amazon.awssdk.services.glue.model.UpdateUserDefinedFunctionRequest;
+import
software.amazon.awssdk.services.glue.model.UpdateUserDefinedFunctionResponse;
+import software.amazon.awssdk.services.glue.model.UserDefinedFunction;
+import software.amazon.awssdk.services.glue.model.UserDefinedFunctionInput;
+
+import java.util.Collection;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Optional;
+import java.util.stream.Collectors;
+
+/** Utilities for Glue catalog Function related operations. */
+@Internal
+public class GlueFunctionsOperations extends AbstractGlueOperations {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(GlueFunctionsOperations.class);
+
+ /**
+ * Constructor to initialize the shared fields.
+ *
+ * @param glueClient The Glue client used for interacting with the AWS
Glue service.
+ * @param catalogName The catalog name associated with the Glue operations.
+ */
+ public GlueFunctionsOperations(GlueClient glueClient, String catalogName) {
+ super(glueClient, catalogName);
+ }
+
+ /**
+ * Create a function. Function name should be handled in a
case-insensitive way.
+ *
+ * @param functionPath path of the function
+ * @param function Flink function to be created
+ * @throws CatalogException in case of any runtime exception
+ */
+ public void createGlueFunction(ObjectPath functionPath, CatalogFunction
function)
+ throws CatalogException, FunctionAlreadyExistException {
+ UserDefinedFunctionInput functionInput =
createFunctionInput(functionPath, function);
+ CreateUserDefinedFunctionRequest.Builder createUDFRequest =
+ CreateUserDefinedFunctionRequest.builder()
+ .databaseName(functionPath.getDatabaseName())
+ .functionInput(functionInput);
+ try {
+ CreateUserDefinedFunctionResponse response =
+
glueClient.createUserDefinedFunction(createUDFRequest.build());
+ LOG.info("Created Function: {}", functionPath.getFullName());
+ } catch (AlreadyExistsException e) {
+ LOG.error(
+ String.format(
+ "%s already Exists. Function language of type: %s.
\n%s",
+ functionPath.getFullName(),
function.getFunctionLanguage(), e));
+ throw new FunctionAlreadyExistException(catalogName, functionPath,
e);
+ } catch (GlueException e) {
+ LOG.error("Error creating glue function: {}\n{}",
functionPath.getFullName(), e);
+ throw new
CatalogException(GlueCatalogConstants.GLUE_EXCEPTION_MSG_IDENTIFIER, e);
+ }
+ }
+
+ /**
+ * Modify an existing function. Function name should be handled in a
case-insensitive way.
+ *
+ * @param functionPath path of function.
+ * @param newFunction modified function.
+ * @throws CatalogException on runtime errors.
+ * @throws FunctionNotExistException if the function doesn't exist.
+ */
+ public void alterGlueFunction(ObjectPath functionPath, CatalogFunction
newFunction)
+ throws CatalogException, FunctionNotExistException {
+
+ UserDefinedFunctionInput functionInput =
createFunctionInput(functionPath, newFunction);
+
+ UpdateUserDefinedFunctionRequest updateUserDefinedFunctionRequest =
+ UpdateUserDefinedFunctionRequest.builder()
+ .functionName(functionPath.getObjectName())
+ .databaseName(functionPath.getDatabaseName())
+ .functionInput(functionInput)
+ .build();
+ try {
+ UpdateUserDefinedFunctionResponse response =
+
glueClient.updateUserDefinedFunction(updateUserDefinedFunctionRequest);
+ LOG.info("Altered Function: {}", functionPath.getFullName());
+ } catch (EntityNotFoundException e) {
+ LOG.error("Function not found: {}", functionPath.getFullName());
+ throw new FunctionNotExistException(catalogName, functionPath, e);
+ } catch (GlueException e) {
+ LOG.error("Error altering glue function: {}\n{}",
functionPath.getFullName(), e);
+ throw new
CatalogException(GlueCatalogConstants.GLUE_EXCEPTION_MSG_IDENTIFIER, e);
+ }
+ }
+
+ /**
+ * Get the user defined function from glue Catalog. Function name should
be handled in a
+ * case-insensitive way.
+ *
+ * @param functionPath path of the function
+ * @return the requested function
+ * @throws CatalogException in case of any runtime exception
+ * @throws FunctionNotExistException if the function doesn't exist
+ */
+ public CatalogFunction getGlueFunction(ObjectPath functionPath) throws
CatalogException, FunctionNotExistException {
+ GetUserDefinedFunctionRequest request =
+ GetUserDefinedFunctionRequest.builder()
+ .databaseName(functionPath.getDatabaseName())
+ .functionName(functionPath.getObjectName())
+ .build();
+ try {
+ GetUserDefinedFunctionResponse response =
glueClient.getUserDefinedFunction(request);
+ UserDefinedFunction udf = response.userDefinedFunction();
+ List<ResourceUri> resourceUris =
+ udf.resourceUris().stream()
+ .map(
+ resourceUri ->
+ new
org.apache.flink.table.resource.ResourceUri(
+
org.apache.flink.table.resource.ResourceType
+
.valueOf(resourceUri.resourceType().name()),
+ resourceUri.uri()))
+ .collect(Collectors.toList());
+ return new CatalogFunctionImpl(
+ GlueFunctionsUtil.getCatalogFunctionClassName(udf),
+ GlueFunctionsUtil.getFunctionalLanguage(udf),
+ resourceUris);
+ } catch (EntityNotFoundException e) {
+ LOG.error("Function not found: {}", functionPath.getFullName());
+ throw new FunctionNotExistException(catalogName, functionPath, e);
+ } catch (GlueException e) {
+ LOG.error("Error fetching function {}: {}",
functionPath.getFullName(), e);
+ throw new CatalogException(
+ String.format("Error getting function %s: %s",
functionPath.getFullName(), e.getMessage()), e);
+ }
+ }
+
+ public List<String> listGlueFunctions(String databaseName) {
+ GetUserDefinedFunctionsRequest.Builder functionsRequest =
+ GetUserDefinedFunctionsRequest.builder()
+ .databaseName(databaseName);
+ List<String> glueFunctions;
+ try {
+ GetUserDefinedFunctionsResponse functionsResponse =
+
glueClient.getUserDefinedFunctions(functionsRequest.build());
+ String token = functionsResponse.nextToken();
+ glueFunctions =
+ functionsResponse.userDefinedFunctions().stream()
+ .map(UserDefinedFunction::functionName)
+ .collect(Collectors.toCollection(LinkedList::new));
+ while (Optional.ofNullable(token).isPresent()) {
+ functionsRequest.nextToken(token);
+ functionsResponse =
glueClient.getUserDefinedFunctions(functionsRequest.build());
+ glueFunctions.addAll(
+ functionsResponse.userDefinedFunctions().stream()
+ .map(UserDefinedFunction::functionName)
+
.collect(Collectors.toCollection(LinkedList::new)));
+ token = functionsResponse.nextToken();
+ }
+ } catch (GlueException e) {
+ throw new
CatalogException(GlueCatalogConstants.GLUE_EXCEPTION_MSG_IDENTIFIER, e);
+ }
+ return glueFunctions;
+ }
+
+ public boolean glueFunctionExists(ObjectPath functionPath) {
+ GetUserDefinedFunctionRequest request =
+ GetUserDefinedFunctionRequest.builder()
+ .functionName(functionPath.getObjectName())
+ .databaseName(functionPath.getDatabaseName())
+ .build();
+
+ try {
+ GetUserDefinedFunctionResponse response =
glueClient.getUserDefinedFunction(request);
+ return response.userDefinedFunction() != null;
+ } catch (EntityNotFoundException e) {
+ return false;
+ } catch (GlueException e) {
+ LOG.error(GlueCatalogConstants.GLUE_EXCEPTION_MSG_IDENTIFIER, e);
+ throw new
CatalogException(GlueCatalogConstants.GLUE_EXCEPTION_MSG_IDENTIFIER, e);
+ }
+ }
+
+ /**
+ * Drop / Delete UserDefinedFunction from glue data catalog.
+ *
+ * @param functionPath fully qualified function path
+ * @throws CatalogException In case of Unexpected errors.
+ */
+ public void dropGlueFunction(ObjectPath functionPath) throws
CatalogException {
Review Comment:
It seems like we do not throw CatalogException within the method, do we need
to catch, wrap and rethrow?
##########
flink-catalog-aws-glue/pom.xml:
##########
@@ -0,0 +1,225 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xmlns="http://maven.apache.org/POM/4.0.0";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>com.amazonaws</groupId>
+ <artifactId>flink-data-catalog</artifactId>
+ <version>1.0</version>
+ <packaging>jar</packaging>
+
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <target.java.version>11</target.java.version>
+ <maven.compiler.source>${target.java.version}</maven.compiler.source>
+ <maven.compiler.target>${target.java.version}</maven.compiler.target>
+ <flink.version>1.20.0</flink.version>
+ <scala.binary.version>2.12</scala.binary.version>
+ <kda.runtime.version>1.2.0</kda.runtime.version>
+ <log4j.version>2.17.2</log4j.version>
+ <slf4j.version>1.7.32</slf4j.version>
+ </properties>
+
+ <dependencies>
+ <!-- Flink Core dependencies -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-clients</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+
<artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-common</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-json</artifactId>
+ <version>1.18.0</version>
Review Comment:
Similar comments over the rest of this pom.xml on flink packages, jackson
databind, connector etc.
##########
flink-catalog-aws-glue/pom.xml:
##########
@@ -0,0 +1,225 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xmlns="http://maven.apache.org/POM/4.0.0";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>com.amazonaws</groupId>
+ <artifactId>flink-data-catalog</artifactId>
+ <version>1.0</version>
+ <packaging>jar</packaging>
+
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <target.java.version>11</target.java.version>
+ <maven.compiler.source>${target.java.version}</maven.compiler.source>
+ <maven.compiler.target>${target.java.version}</maven.compiler.target>
+ <flink.version>1.20.0</flink.version>
+ <scala.binary.version>2.12</scala.binary.version>
+ <kda.runtime.version>1.2.0</kda.runtime.version>
+ <log4j.version>2.17.2</log4j.version>
+ <slf4j.version>1.7.32</slf4j.version>
+ </properties>
+
+ <dependencies>
+ <!-- Flink Core dependencies -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-clients</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+
<artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-common</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-json</artifactId>
+ <version>1.18.0</version>
Review Comment:
Why do we use 1.18 here over `${flink.version}`?
##########
flink-catalog-aws-glue/pom.xml:
##########
@@ -0,0 +1,225 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xmlns="http://maven.apache.org/POM/4.0.0";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <groupId>com.amazonaws</groupId>
+ <artifactId>flink-data-catalog</artifactId>
+ <version>1.0</version>
+ <packaging>jar</packaging>
+
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <target.java.version>11</target.java.version>
+ <maven.compiler.source>${target.java.version}</maven.compiler.source>
+ <maven.compiler.target>${target.java.version}</maven.compiler.target>
+ <flink.version>1.20.0</flink.version>
+ <scala.binary.version>2.12</scala.binary.version>
+ <kda.runtime.version>1.2.0</kda.runtime.version>
+ <log4j.version>2.17.2</log4j.version>
+ <slf4j.version>1.7.32</slf4j.version>
+ </properties>
+
+ <dependencies>
+ <!-- Flink Core dependencies -->
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-streaming-java</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-clients</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+
<artifactId>flink-table-planner_${scala.binary.version}</artifactId>
+ <version>${flink.version}</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-api-java-bridge</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-common</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-json</artifactId>
+ <version>1.18.0</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>glue</artifactId>
+ <version>2.30.2</version> <!-- Use the latest version -->
+ </dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>sts</artifactId>
+ <version>2.30.2</version> <!-- For authentication if needed -->
+ </dependency>
+ <dependency>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>aws-kinesisanalytics-runtime</artifactId>
+ <version>${kda.runtime.version}</version>
+ </dependency>
+
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-base</artifactId>
+ <version>${flink.version}</version>
+ </dependency>
+ <dependency>
+ <groupId>software.amazon.awssdk</groupId>
+ <artifactId>bedrockruntime</artifactId>
Review Comment:
Can you elaborate why bedrockruntime is needed?
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/GlueCatalogFactory.java:
##########
@@ -0,0 +1,66 @@
+package com.amazonaws.services.msf;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.catalog.Catalog;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.factories.CatalogFactory;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.glue.GlueClient;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class GlueCatalogFactory implements CatalogFactory {
+
+ // Define configuration options that users must provide
+ public static final ConfigOption<String> REGION =
+ ConfigOptions.key("region")
Review Comment:
Should we have qualifier/prefix so that it's clear that the config is for
GlueCatalogConnector?
For example:
```
catalog.glue.region
catalog.glue.default-database
```
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/GlueCatalogFactory.java:
##########
@@ -0,0 +1,66 @@
+package com.amazonaws.services.msf;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.table.catalog.Catalog;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.factories.CatalogFactory;
+import software.amazon.awssdk.regions.Region;
+import software.amazon.awssdk.services.glue.GlueClient;
+
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+public class GlueCatalogFactory implements CatalogFactory {
+
+ // Define configuration options that users must provide
+ public static final ConfigOption<String> REGION =
+ ConfigOptions.key("region")
+ .stringType()
+ .noDefaultValue()
+ .withDescription("AWS region for the Glue catalog");
+
+ public static final ConfigOption<String> DEFAULT_DATABASE =
+ ConfigOptions.key("default-database")
+ .stringType()
+ .defaultValue("default")
+ .withDescription("Default database to use in Glue
catalog");
+
+ @Override
+ public String factoryIdentifier() {
+ return "glue"; // This allows users to use `type = 'glue'` in SQL
+ }
+
+ @Override
+ public Set<ConfigOption<?>> requiredOptions() {
+ Set<ConfigOption<?>> options = new HashSet<>();
+ options.add(REGION);
+ return options;
+ }
+
+ @Override
+ public Set<ConfigOption<?>> optionalOptions() {
+ Set<ConfigOption<?>> options = new HashSet<>();
+ options.add(DEFAULT_DATABASE);
+ return options;
+ }
+
+ @Override
+ public Catalog createCatalog(Context context) {
+ // Read the configuration options
+ Map<String,String> config = context.getOptions();
+ String name = context.getName();
+ String region = config.get("region");
+ String defaultDatabase = config.get("default-database");
Review Comment:
Instead of declaring string here, can we use the REGION/DEFAULT_DATABASE
ConfigOptions so that there's a single place to update the config key in case
we ever need to?
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/StreamingJob.java:
##########
@@ -0,0 +1,286 @@
+package com.amazonaws.services.msf;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+import org.apache.flink.table.api.EnvironmentSettings;
+import org.apache.flink.table.api.TableEnvironment;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.catalog.Catalog;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+public class StreamingJob {
Review Comment:
Does this class serve as an example or is it a sandbox job for you to test?
Do we need this committed?
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/operations/GlueDatabaseOperations.java:
##########
@@ -0,0 +1,145 @@
+package com.amazonaws.services.msf.operations;
+
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.exceptions.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.*;
Review Comment:
Let's avoid using wildcard imports
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/operations/GlueDatabaseOperations.java:
##########
@@ -0,0 +1,145 @@
+package com.amazonaws.services.msf.operations;
+
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.exceptions.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.*;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Handles all database-related operations for the Glue catalog.
+ * Provides functionality for listing, retrieving, creating, and deleting
databases in AWS Glue.
+ */
+public class GlueDatabaseOperations extends AbstractGlueOperations {
+
+ /** Logger for logging database operations. */
+ private static final Logger LOG =
LoggerFactory.getLogger(GlueDatabaseOperations.class);
+
+ /**
+ * Constructor for GlueDatabaseOperations.
+ * Initializes the Glue client and catalog name.
+ *
+ * @param glueClient The Glue client to interact with AWS Glue.
+ * @param catalogName The name of the catalog.
+ */
+ public GlueDatabaseOperations(GlueClient glueClient, String catalogName) {
+ super(glueClient, catalogName);
+ }
+
+ /**
+ * Lists all the databases in the Glue catalog.
+ *
+ * @return A list of database names.
+ * @throws CatalogException if there is an error fetching the list of
databases.
+ */
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ LOG.debug("Listing databases from Glue catalog");
+ return
glueClient.getDatabases(GetDatabasesRequest.builder().build())
+ .databaseList()
+ .stream()
+ .map(Database::name)
+ .collect(Collectors.toList());
+ } catch (GlueException e) {
+ LOG.error("Failed to list databases in Glue", e);
+ throw new CatalogException("Failed to list databases: " +
e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Retrieves the specified database from the Glue catalog.
+ *
+ * @param databaseName The name of the database to fetch.
+ * @return The CatalogDatabase object representing the Glue database.
+ * @throws DatabaseNotExistException If the database does not exist in the
Glue catalog.
+ * @throws CatalogException If there is any error retrieving the database.
+ */
+ public CatalogDatabase getDatabase(String databaseName) throws
DatabaseNotExistException, CatalogException {
+ try {
+ GetDatabaseResponse response = glueClient.getDatabase(
+ GetDatabaseRequest.builder()
+ .name(databaseName)
+ .build()
+ );
+
+ Database glueDatabase = response.database();
+ return convertGlueDatabase(glueDatabase);
+ } catch (EntityNotFoundException e) {
+ throw new DatabaseNotExistException(catalogName, databaseName);
+ } catch (Exception e) {
Review Comment:
Should we catch Glue or SdkException here instead of non-specific Exception?
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/util/GlueFunctionsUtil.java:
##########
@@ -0,0 +1,52 @@
+package com.amazonaws.services.msf.util;
+
+import org.apache.flink.table.catalog.CatalogFunction;
+import org.apache.flink.table.catalog.FunctionLanguage;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import software.amazon.awssdk.services.glue.model.UserDefinedFunction;
+
+import java.util.Arrays;
+import java.util.stream.Collectors;
+
+public class GlueFunctionsUtil {
+
+ public static String getCatalogFunctionClassName(final UserDefinedFunction
udf) {
+ String[] splitName =
udf.className().split(GlueCatalogConstants.DEFAULT_SEPARATOR);
+ return splitName[splitName.length - 1];
+ }
+ public static FunctionLanguage getFunctionalLanguage(final
UserDefinedFunction glueFunction) {
+ if
(glueFunction.className().startsWith(GlueCatalogConstants.FLINK_JAVA_FUNCTION_PREFIX))
{
+ return FunctionLanguage.JAVA;
+ } else if (glueFunction
+ .className()
+
.startsWith(GlueCatalogConstants.FLINK_PYTHON_FUNCTION_PREFIX)) {
+ return FunctionLanguage.PYTHON;
+ } else if (glueFunction
+ .className()
+ .startsWith(GlueCatalogConstants.FLINK_SCALA_FUNCTION_PREFIX))
{
+ return FunctionLanguage.SCALA;
+ } else {
+ throw new CatalogException(
+ "Invalid Functional Language for className: " +
glueFunction.className());
+ }
+ }
+
+ public static String getGlueFunctionClassName(CatalogFunction function) {
+ switch (function.getFunctionLanguage()) {
+ case JAVA:
+ return GlueCatalogConstants.FLINK_JAVA_FUNCTION_PREFIX +
function.getClassName();
+ case SCALA:
+ return GlueCatalogConstants.FLINK_SCALA_FUNCTION_PREFIX +
function.getClassName();
+ case PYTHON:
+ return GlueCatalogConstants.FLINK_PYTHON_FUNCTION_PREFIX +
function.getClassName();
Review Comment:
It seems like we are using a self defined pattern for class names
```
public static final String FLINK_SCALA_FUNCTION_PREFIX = "flink:scala:";
public static final String FLINK_PYTHON_FUNCTION_PREFIX =
"flink:python:";
public static final String FLINK_JAVA_FUNCTION_PREFIX = "flink:java:";
```
Documentation on glue's UserDefinedFunction actually says `The Java class
that contains the function code. `. Should we use Java namespace + class name
format instead?
https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/services/glue/model/UserDefinedFunction.html#className()
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/operations/GlueDatabaseOperations.java:
##########
@@ -0,0 +1,145 @@
+package com.amazonaws.services.msf.operations;
+
+import org.apache.flink.table.catalog.CatalogDatabase;
+import org.apache.flink.table.catalog.CatalogDatabaseImpl;
+import org.apache.flink.table.catalog.exceptions.*;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.*;
+
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+
+/**
+ * Handles all database-related operations for the Glue catalog.
+ * Provides functionality for listing, retrieving, creating, and deleting
databases in AWS Glue.
+ */
+public class GlueDatabaseOperations extends AbstractGlueOperations {
+
+ /** Logger for logging database operations. */
+ private static final Logger LOG =
LoggerFactory.getLogger(GlueDatabaseOperations.class);
+
+ /**
+ * Constructor for GlueDatabaseOperations.
+ * Initializes the Glue client and catalog name.
+ *
+ * @param glueClient The Glue client to interact with AWS Glue.
+ * @param catalogName The name of the catalog.
+ */
+ public GlueDatabaseOperations(GlueClient glueClient, String catalogName) {
+ super(glueClient, catalogName);
+ }
+
+ /**
+ * Lists all the databases in the Glue catalog.
+ *
+ * @return A list of database names.
+ * @throws CatalogException if there is an error fetching the list of
databases.
+ */
+ public List<String> listDatabases() throws CatalogException {
+ try {
+ LOG.debug("Listing databases from Glue catalog");
+ return
glueClient.getDatabases(GetDatabasesRequest.builder().build())
+ .databaseList()
+ .stream()
+ .map(Database::name)
+ .collect(Collectors.toList());
+ } catch (GlueException e) {
+ LOG.error("Failed to list databases in Glue", e);
+ throw new CatalogException("Failed to list databases: " +
e.getMessage(), e);
+ }
+ }
+
+ /**
+ * Retrieves the specified database from the Glue catalog.
+ *
+ * @param databaseName The name of the database to fetch.
+ * @return The CatalogDatabase object representing the Glue database.
+ * @throws DatabaseNotExistException If the database does not exist in the
Glue catalog.
+ * @throws CatalogException If there is any error retrieving the database.
+ */
+ public CatalogDatabase getDatabase(String databaseName) throws
DatabaseNotExistException, CatalogException {
+ try {
+ GetDatabaseResponse response = glueClient.getDatabase(
+ GetDatabaseRequest.builder()
+ .name(databaseName)
+ .build()
+ );
+
+ Database glueDatabase = response.database();
+ return convertGlueDatabase(glueDatabase);
+ } catch (EntityNotFoundException e) {
+ throw new DatabaseNotExistException(catalogName, databaseName);
+ } catch (Exception e) {
+ throw new CatalogException("Error fetching database: " +
databaseName, e);
+ }
+ }
+
+ /**
+ * Converts the Glue database model to a Flink CatalogDatabase.
+ *
+ * @param glueDatabase The Glue database model.
+ * @return A CatalogDatabase representing the Glue database.
+ */
+ private CatalogDatabase convertGlueDatabase(Database glueDatabase) {
+ Map<String, String> properties = new
HashMap<>(glueDatabase.parameters());
+ return new CatalogDatabaseImpl(
+ properties,
+ glueDatabase.description()
+ );
+ }
+
+ /**
+ * Checks whether a database exists in Glue.
+ *
+ * @param databaseName The name of the database to check.
+ * @return true if the database exists, false otherwise.
+ */
+ public boolean glueDatabaseExists(String databaseName) {
+ try {
+ glueClient.getDatabase(builder -> builder.name(databaseName));
+ return true;
+ } catch (EntityNotFoundException e) {
+ return false;
+ } catch (Exception e) {
Review Comment:
Let's use more specific exception here
##########
flink-catalog-aws-glue/dependency-reduced-pom.xml:
##########
@@ -0,0 +1,207 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>flink-data-catalog</artifactId>
+ <version>1.0</version>
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>3.8.1</version>
Review Comment:
Also applies to flink-catalog-aws-glue/pom.xml
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/util/ConnectorRegistry.java:
##########
@@ -0,0 +1,53 @@
+package com.amazonaws.services.msf.util;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.HashMap;
+import java.util.Map;
+
+/**
+ * This class is responsible for storing and retrieving location-specific keys
for different connectors.
+ * It maps connector types to their corresponding location keys (e.g.,
Kinesis, Kafka).
+ */
+public class ConnectorRegistry {
+
+ /** Logger for logging connector registry actions. */
+ private static final Logger LOG =
LoggerFactory.getLogger(ConnectorRegistry.class);
+
+ /** Map to store connector types and their corresponding location-specific
keys. */
+ private static final Map<String, String> connectorLocationKeys = new
HashMap<>();
+
+ // Static block to initialize the connector keys mapping.
+ static {
+ connectorLocationKeys.put("kinesis", "stream.arn");
+ connectorLocationKeys.put("kafka", "properties.bootstrap.servers");
+ connectorLocationKeys.put("jdbc", "url");
+ connectorLocationKeys.put("filesystem", "path");
+ connectorLocationKeys.put("elasticsearch", "hosts");
+ connectorLocationKeys.put("opensearch", "hosts");
+ connectorLocationKeys.put("hbase", "zookeeper.quorum");
+ connectorLocationKeys.put("dynamodb", "table.name");
+ connectorLocationKeys.put("mongodb", "uri");
+ connectorLocationKeys.put("hive", "hive-conf-dir");
Review Comment:
Should we have qualifier/prefix for the location keys to avoid clashes and
also so that it's easier to decipher without any further context?
##########
flink-catalog-aws-glue/src/main/java/com/amazonaws/services/msf/operations/GlueTableOperations.java:
##########
@@ -0,0 +1,157 @@
+package com.amazonaws.services.msf.operations;
+
+import org.apache.flink.table.catalog.CatalogTable;
+import org.apache.flink.table.catalog.ObjectPath;
+import org.apache.flink.table.catalog.exceptions.CatalogException;
+import org.apache.flink.table.catalog.exceptions.TableNotExistException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import software.amazon.awssdk.services.glue.GlueClient;
+import software.amazon.awssdk.services.glue.model.*;
+
+import java.util.*;
+import java.util.stream.Collectors;
+
+/**
+ * Handles all table-related operations for the Glue catalog.
+ * Provides functionality for checking existence, listing, creating, getting,
and dropping tables in AWS Glue.
+ */
+public class GlueTableOperations extends AbstractGlueOperations {
+
+ /**
+ * Logger for logging table operations.
+ */
+ private static final Logger LOG =
LoggerFactory.getLogger(GlueTableOperations.class);
+
+ /**
+ * Constructor for GlueTableOperations.
+ * Initializes the Glue client and catalog name.
+ *
+ * @param glueClient The Glue client to interact with AWS Glue.
+ * @param catalogName The name of the catalog.
+ */
+ public GlueTableOperations(GlueClient glueClient, String catalogName) {
+ super(glueClient, catalogName);
+ }
+
+ /**
+ * Checks whether a table exists in the Glue catalog.
+ *
+ * @param databaseName The name of the database where the table should
exist.
+ * @param tableName The name of the table to check.
+ * @return true if the table exists, false otherwise.
+ */
+ public boolean tableExists(String databaseName, String tableName) {
+ try {
+ glueClient.getTable(builder ->
builder.databaseName(databaseName).name(tableName));
+ return true;
+ } catch (EntityNotFoundException e) {
+ return false;
+ } catch (Exception e) {
Review Comment:
Let's catch more specific exception, similarly on 69, 84, 109, 153
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
