hlteoh37 commented on code in PR #191:
URL:
https://github.com/apache/flink-connector-aws/pull/191#discussion_r2044486912
##########
flink-catalog-aws-glue/README.md:
##########
@@ -0,0 +1,377 @@
+# Flink AWS Glue Catalog Connector
+
+The Flink AWS Glue Catalog connector provides integration between Apache Flink
and the AWS Glue Data Catalog. This connector enables Flink applications to use
AWS Glue as a metadata catalog for tables, databases, and schemas, allowing
seamless SQL queries against AWS resources.
+
+## Features
+
+- Register AWS Glue as a catalog in Flink applications
+- Access Glue databases and tables through Flink SQL
+- Support for various AWS data sources (S3, Kinesis, MSK)
+- Mapping between Flink and AWS Glue data types
+- Compatibility with Flink's Table API and SQL interface
+
+## Prerequisites
+
+Before getting started, ensure you have the following:
+
+- **Apache Flink 1.20+** installed or running in a managed environment
+- **AWS account** with appropriate permissions for AWS Glue and other required
services
+- **AWS credentials** properly configured
+- **Java 11** or later
+- **Maven 3.8+** for building the project
+
+## Getting Started
+
+### 1. Add Dependency
+
+Add the AWS Glue Catalog connector to your Flink project:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-aws-glue-catalog</artifactId>
+ <version>${flink.version}</version>
Review Comment:
Hmm I think this is not correct. Maybe we should link directly to the stable
docs instead for more up-to-date documentation. We can update after we do the
first release.
##########
flink-catalog-aws-glue/README.md:
##########
@@ -0,0 +1,377 @@
+# Flink AWS Glue Catalog Connector
+
+The Flink AWS Glue Catalog connector provides integration between Apache Flink
and the AWS Glue Data Catalog. This connector enables Flink applications to use
AWS Glue as a metadata catalog for tables, databases, and schemas, allowing
seamless SQL queries against AWS resources.
+
+## Features
+
+- Register AWS Glue as a catalog in Flink applications
+- Access Glue databases and tables through Flink SQL
+- Support for various AWS data sources (S3, Kinesis, MSK)
+- Mapping between Flink and AWS Glue data types
+- Compatibility with Flink's Table API and SQL interface
Review Comment:
Awesome!
##########
flink-catalog-aws-glue/dependency-reduced-pom.xml:
##########
@@ -0,0 +1,207 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>flink-data-catalog</artifactId>
+ <version>1.0</version>
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>3.8.1</version>
+ <configuration>
+ <source>${target.java.version}</source>
+ <target>${target.java.version}</target>
+ </configuration>
+ </plugin>
+ <plugin>
+ <groupId>org.jacoco</groupId>
+ <artifactId>jacoco-maven-plugin</artifactId>
+ <version>0.8.12</version>
+ <executions>
+ <execution>
+ <id>prepare-agent</id>
+ <goals>
+ <goal>prepare-agent</goal>
+ </goals>
+ </execution>
+ <execution>
+ <id>report</id>
+ <phase>verify</phase>
+ <goals>
+ <goal>report</goal>
+ </goals>
+ </execution>
+ </executions>
+ <configuration>
+ <excludes>
+ <exclude>org.apache.flink.table.catalog.glue.StreamingJob</exclude>
+ </excludes>
+ </configuration>
+ </plugin>
+ <plugin>
+ <artifactId>maven-shade-plugin</artifactId>
+ <version>3.2.1</version>
+ <executions>
+ <execution>
+ <phase>package</phase>
+ <goals>
+ <goal>shade</goal>
+ </goals>
+ <configuration>
+ <artifactSet>
+ <excludes>
+ <exclude>org.apache.flink:force-shading</exclude>
+ <exclude>com.google.code.findbugs:jsr305</exclude>
+ <exclude>org.slf4j:*</exclude>
+ <exclude>log4j:*</exclude>
+ </excludes>
+ </artifactSet>
+ <filters>
+ <filter>
+ <artifact>*:*</artifact>
+ <excludes>
+ <exclude>META-INF/*.SF</exclude>
+ <exclude>META-INF/*.DSA</exclude>
+ <exclude>META-INF/*.RSA</exclude>
+ </excludes>
+ </filter>
+ </filters>
+ <transformers>
+ <transformer />
+ <transformer>
+
<mainClass>org.apache.flink.table.catalog.glue.StreamingJob</mainClass>
+ </transformer>
+ </transformers>
+ </configuration>
+ </execution>
+ </executions>
+ </plugin>
+ </plugins>
+ </build>
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-clients</artifactId>
+ <version>1.20.0</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <artifactId>flink-optimizer</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-datastream</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-table-planner_2.12</artifactId>
+ <version>1.20.0</version>
+ <scope>provided</scope>
+ <exclusions>
+ <exclusion>
+ <artifactId>value</artifactId>
+ <groupId>org.immutables</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>value-annotations</artifactId>
+ <groupId>org.immutables</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>commons-compiler</artifactId>
+ <groupId>org.codehaus.janino</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>janino</artifactId>
+ <groupId>org.codehaus.janino</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-scala_2.12</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-table-runtime</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-json</artifactId>
+ <version>1.18.0</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-files</artifactId>
+ <version>1.20.0</version>
+ <scope>provided</scope>
+ </dependency>
+ <dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-test-utils</artifactId>
+ <version>1.20.0</version>
+ <scope>test</scope>
+ <exclusions>
+ <exclusion>
+ <artifactId>flink-test-utils-junit</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-runtime</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-core</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-rpc-akka-loader</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>junit-jupiter</artifactId>
+ <groupId>org.junit.jupiter</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>assertj-core</artifactId>
+ <groupId>org.assertj</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-streaming-java</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-statebackend-rocksdb</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-statebackend-changelog</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>flink-dstl-dfs</artifactId>
+ <groupId>org.apache.flink</groupId>
+ </exclusion>
+ <exclusion>
+ <artifactId>curator-test</artifactId>
+ <groupId>org.apache.curator</groupId>
+ </exclusion>
+ </exclusions>
+ </dependency>
+ </dependencies>
+ <properties>
+ <scala.binary.version>2.12</scala.binary.version>
+ <maven.compiler.target>${target.java.version}</maven.compiler.target>
+ <slf4j.version>1.7.32</slf4j.version>
+ <maven.compiler.source>${target.java.version}</maven.compiler.source>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <log4j.version>2.17.2</log4j.version>
+ <target.java.version>11</target.java.version>
+ <flink.version>1.20.0</flink.version>
+ <kda.runtime.version>1.2.0</kda.runtime.version>
+ </properties>
+</project>
Review Comment:
We don't need to commit `dependency-reduced-pom.xml`. Let's remove it.
##########
flink-catalog-aws-glue/README.md:
##########
@@ -0,0 +1,377 @@
+# Flink AWS Glue Catalog Connector
+
+The Flink AWS Glue Catalog connector provides integration between Apache Flink
and the AWS Glue Data Catalog. This connector enables Flink applications to use
AWS Glue as a metadata catalog for tables, databases, and schemas, allowing
seamless SQL queries against AWS resources.
+
+## Features
+
+- Register AWS Glue as a catalog in Flink applications
+- Access Glue databases and tables through Flink SQL
+- Support for various AWS data sources (S3, Kinesis, MSK)
+- Mapping between Flink and AWS Glue data types
+- Compatibility with Flink's Table API and SQL interface
+
+## Prerequisites
+
+Before getting started, ensure you have the following:
+
+- **Apache Flink 1.20+** installed or running in a managed environment
+- **AWS account** with appropriate permissions for AWS Glue and other required
services
+- **AWS credentials** properly configured
+- **Java 11** or later
+- **Maven 3.8+** for building the project
+
+## Getting Started
+
+### 1. Add Dependency
+
+Add the AWS Glue Catalog connector to your Flink project:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-aws-glue-catalog</artifactId>
+ <version>${flink.version}</version>
+</dependency>
+```
+
+### 2. Configure AWS Credentials
+
+Ensure AWS credentials are configured using one of these methods:
+
+- Environment variables
+- AWS credentials file
+- IAM roles (for applications running on AWS)
+
+### 3. Register the Glue Catalog
+
+You can register the AWS Glue catalog using either the Table API or SQL:
+
+#### Using Table API (Java/Scala)
+
+```java
+// Java/Scala
+import org.apache.flink.table.catalog.glue.GlueCatalog;
+import org.apache.flink.table.catalog.Catalog;
+
+// Create Glue catalog instance
+Catalog glueCatalog = new GlueCatalog(
+ "glue_catalog", // Catalog name
+ "default", // Default database
+ "us-east-1"); // AWS region
+
+
+// Register with table environment
+tableEnv.registerCatalog("glue_catalog", glueCatalog);
+tableEnv.useCatalog("glue_catalog");
+```
+
+#### Using Table API (Python)
+
+```python
+# Python
+from pyflink.table.catalog import GlueCatalog
+
+# Create and register Glue catalog
+glue_catalog = GlueCatalog(
+ "glue_catalog", # Catalog name
+ "default", # Default database
+ "us-east-1") # AWS region
+
+t_env.register_catalog("glue_catalog", glue_catalog)
+t_env.use_catalog("glue_catalog")
+```
+
+#### Using SQL
+
+In the Flink SQL Client, create and use the Glue catalog:
+
+```sql
+-- Create a catalog using Glue
+CREATE CATALOG glue_catalog WITH (
+ 'type' = 'glue',
+ 'catalog-name' = 'glue_catalog',
+ 'default-database' = 'default',
+ 'region' = 'us-east-1'
+);
+
+-- Use the created catalog
+USE CATALOG glue_catalog;
+
+-- Use a specific database
+USE default;
+```
+
+### 4. Create or Reference Glue Tables
+
+Once the catalog is registered, you can create new tables or reference
existing ones:
+
+```sql
+-- Create a new table in Glue
+CREATE TABLE customer_table (
+ id BIGINT,
+ name STRING,
+ region STRING
+) WITH (
+ 'connector' = 'kinesis',
+ 'stream' = 'customer-stream',
+ 'aws.region' = 'us-east-1',
+ 'format' = 'json'
+);
+
+-- Query existing Glue table
+SELECT * FROM glue_catalog.sales_db.order_table;
+```
+
+## Catalog Operations
+
+The AWS Glue Catalog connector supports several catalog operations through
SQL. Here's a list of the operations that are currently implemented:
+
+### Database Operations
+
+```sql
+-- Create a new database
+CREATE DATABASE sales_db;
+
+-- Create a database with comment
+CREATE DATABASE sales_db COMMENT 'Database for sales data';
+
+-- Create a database if it doesn't exist
+CREATE DATABASE IF NOT EXISTS sales_db;
+
+-- Drop a database
+DROP DATABASE sales_db;
+
+-- Drop a database if it exists
+DROP DATABASE IF EXISTS sales_db;
+
+-- Use a specific database
+USE sales_db;
+```
+
+### Table Operations
+
+```sql
+-- Create a table
+CREATE TABLE orders (
+ order_id BIGINT,
+ customer_id BIGINT,
+ order_date TIMESTAMP,
+ amount DECIMAL(10, 2)
+);
+
+-- Create a table with comment and properties
+CREATE TABLE orders (
+ order_id BIGINT,
+ customer_id BIGINT,
+ order_date TIMESTAMP,
+ amount DECIMAL(10, 2),
+ PRIMARY KEY (order_id) NOT ENFORCED
+) COMMENT 'Table storing order information'
+WITH (
+ 'connector' = 's3',
+ 'path' = 's3://bucket/path/to/orders',
+ 'format' = 'parquet'
+);
+
+-- Create table if not exists
+CREATE TABLE IF NOT EXISTS orders (
+ order_id BIGINT,
+ customer_id BIGINT
+);
+
+-- Drop a table
+DROP TABLE orders;
+
+-- Drop a table if it exists
+DROP TABLE IF EXISTS orders;
+
+-- Show table details
+DESCRIBE orders;
+```
+
+### View Operations
+
+```sql
+-- Create a view
+CREATE VIEW order_summary AS
+SELECT customer_id, COUNT(*) as order_count, SUM(amount) as total_amount
+FROM orders
+GROUP BY customer_id;
+
+-- Create a temporary view (only available in current session)
+CREATE TEMPORARY VIEW temp_view AS
+SELECT * FROM orders WHERE amount > 100;
+
+-- Drop a view
+DROP VIEW order_summary;
+
+-- Drop a view if it exists
+DROP VIEW IF EXISTS order_summary;
+```
+
+### Function Operations
+
+```sql
+-- Register a function
+CREATE FUNCTION multiply_func AS 'com.example.functions.MultiplyFunction';
+
+-- Register a temporary function
+CREATE TEMPORARY FUNCTION temp_function AS
'com.example.functions.TempFunction';
+
+-- Drop a function
+DROP FUNCTION multiply_func;
+
+-- Drop a temporary function
+DROP TEMPORARY FUNCTION temp_function;
+```
+
+### Listing Resources
+
+Query available catalogs, databases, and tables:
+
+```sql
+-- List all catalogs
+SHOW CATALOGS;
+
+-- List databases in the current catalog
+SHOW DATABASES;
+
+-- List tables in the current database
+SHOW TABLES;
+
+-- List tables in a specific database
+SHOW TABLES FROM sales_db;
+
+-- List views in the current database
+SHOW VIEWS;
+
+-- List functions
+SHOW FUNCTIONS;
+```
+
+## Case Sensitivity in AWS Glue
+
+### Understanding Case Handling
+
+AWS Glue handles case sensitivity in a specific way:
+
+1. **Top-level column names** are automatically lowercased in Glue (e.g.,
`UserProfile` becomes `userprofile`)
+2. **Nested struct field names** preserve their original case in Glue (e.g.,
inside a struct, `FirstName` stays as `FirstName`)
+
+However, when writing queries in Flink SQL, you should use the **original
column names** as defined in your `CREATE TABLE` statement, not how they are
stored in Glue.
+
+### Example with Nested Fields
+
+Consider this table definition:
+
+```sql
+CREATE TABLE nested_json_test (
+ `Id` INT,
+ `UserProfile` ROW<
+ `FirstName` VARCHAR(255),
+ `lastName` VARCHAR(255)
+ >,
+ `event_data` ROW<
+ `EventType` VARCHAR(50),
+ `eventTimestamp` TIMESTAMP(3)
+ >,
+ `metadata` MAP<VARCHAR(100), VARCHAR(255)>
+)
+```
+
+When stored in Glue, the schema looks like:
+
+```json
+{
+ "userprofile": { // Note: lowercased
+ "FirstName": "string", // Note: original case preserved
+ "lastName": "string" // Note: original case preserved
+ }
+}
+```
+
+### Querying Nested Fields
+
+When querying, always use the original column names as defined in your `CREATE
TABLE` statement:
+
+```sql
+-- CORRECT: Use the original column names from CREATE TABLE
+SELECT UserProfile.FirstName FROM nested_json_test;
+
+-- INCORRECT: This doesn't match your schema definition
+SELECT `userprofile`.`FirstName` FROM nested_json_test;
+
+-- For nested fields within nested fields, also use original case
+SELECT event_data.EventType, event_data.eventTimestamp FROM nested_json_test;
+
+-- Accessing map fields
+SELECT metadata['source_system'] FROM nested_json_test;
+```
+
+### Important Notes on Case Sensitivity
+
+1. Always use the original column names as defined in your `CREATE TABLE`
statement
+2. Use backticks (`) when column names contain special characters or spaces
+3. Remember that regardless of how Glue stores the data internally, your
queries should match your schema definition
+4. When creating tables, defining the schema with backticks is recommended for
clarity
+
+## Data Type Mapping
+
+The connector handles mapping between Flink data types and AWS Glue data types
automatically. The following table shows the basic type mappings:
+
+| Flink Type | AWS Glue Type |
+|------------|---------------|
+| CHAR | string |
+| VARCHAR | string |
+| BOOLEAN | boolean |
+| BINARY | binary |
+| VARBINARY | binary |
+| DECIMAL | decimal |
+| TINYINT | byte |
+| SMALLINT | short |
+| INTEGER | int |
+| BIGINT | long |
+| FLOAT | float |
+| DOUBLE | double |
+| DATE | date |
+| TIME | string |
+| TIMESTAMP | timestamp |
+| ROW | struct |
+| ARRAY | array |
+| MAP | map |
+
+## Limitations and Considerations
+
+1. **Case Sensitivity**: As detailed above, always use the original column
names from your schema definition when querying.
+3. **AWS Service Limits**: Be aware of AWS Glue service limits that may affect
your application.
+4. **Authentication**: Ensure proper AWS credentials with appropriate
permissions are available.
+5. **Region Selection**: The Glue catalog must be registered with the correct
AWS region where your Glue resources exist.
+6. **Unsupported Operations**: The following operations are not currently
supported:
+ - ALTER DATABASE (modifying database properties)
+ - ALTER TABLE (modifying table properties or schema)
+ - RENAME TABLE
+ - Partition management operations (ADD/DROP PARTITION)
+
+## Troubleshooting
+
+### Common Issues
+
+1. **"Table not found"**: Verify the table exists in the specified Glue
database and catalog.
+2. **Authentication errors**: Check AWS credentials and permissions.
+3. **Case sensitivity errors**: Ensure you're using the original column names
as defined in your schema.
+4. **Type conversion errors**: Verify that data types are compatible between
Flink and Glue.
+
+### Logging
+
+Enable detailed logging for troubleshooting:
+
+```java
+// Set to DEBUG level for more detailed logs
+LOG.setLevel(Level.DEBUG);
+```
+
Review Comment:
Seems like this is not really related to Glue catalog. We can consider
removing.
##########
flink-catalog-aws-glue/README.md:
##########
@@ -0,0 +1,377 @@
+# Flink AWS Glue Catalog Connector
+
+The Flink AWS Glue Catalog connector provides integration between Apache Flink
and the AWS Glue Data Catalog. This connector enables Flink applications to use
AWS Glue as a metadata catalog for tables, databases, and schemas, allowing
seamless SQL queries against AWS resources.
+
+## Features
+
+- Register AWS Glue as a catalog in Flink applications
+- Access Glue databases and tables through Flink SQL
+- Support for various AWS data sources (S3, Kinesis, MSK)
+- Mapping between Flink and AWS Glue data types
+- Compatibility with Flink's Table API and SQL interface
+
+## Prerequisites
+
+Before getting started, ensure you have the following:
+
+- **Apache Flink 1.20+** installed or running in a managed environment
+- **AWS account** with appropriate permissions for AWS Glue and other required
services
+- **AWS credentials** properly configured
+- **Java 11** or later
+- **Maven 3.8+** for building the project
+
+## Getting Started
+
+### 1. Add Dependency
+
+Add the AWS Glue Catalog connector to your Flink project:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-aws-glue-catalog</artifactId>
+ <version>${flink.version}</version>
+</dependency>
+```
+
+### 2. Configure AWS Credentials
+
+Ensure AWS credentials are configured using one of these methods:
+
+- Environment variables
+- AWS credentials file
+- IAM roles (for applications running on AWS)
+
+### 3. Register the Glue Catalog
+
+You can register the AWS Glue catalog using either the Table API or SQL:
+
+#### Using Table API (Java/Scala)
+
+```java
+// Java/Scala
+import org.apache.flink.table.catalog.glue.GlueCatalog;
+import org.apache.flink.table.catalog.Catalog;
+
+// Create Glue catalog instance
+Catalog glueCatalog = new GlueCatalog(
+ "glue_catalog", // Catalog name
+ "default", // Default database
+ "us-east-1"); // AWS region
+
+
+// Register with table environment
+tableEnv.registerCatalog("glue_catalog", glueCatalog);
+tableEnv.useCatalog("glue_catalog");
+```
+
Review Comment:
Simple and easy to read. Nice API :)
##########
flink-catalog-aws-glue/README.md:
##########
@@ -0,0 +1,377 @@
+# Flink AWS Glue Catalog Connector
+
+The Flink AWS Glue Catalog connector provides integration between Apache Flink
and the AWS Glue Data Catalog. This connector enables Flink applications to use
AWS Glue as a metadata catalog for tables, databases, and schemas, allowing
seamless SQL queries against AWS resources.
+
+## Features
+
+- Register AWS Glue as a catalog in Flink applications
+- Access Glue databases and tables through Flink SQL
+- Support for various AWS data sources (S3, Kinesis, MSK)
+- Mapping between Flink and AWS Glue data types
+- Compatibility with Flink's Table API and SQL interface
+
+## Prerequisites
+
+Before getting started, ensure you have the following:
+
+- **Apache Flink 1.20+** installed or running in a managed environment
+- **AWS account** with appropriate permissions for AWS Glue and other required
services
+- **AWS credentials** properly configured
+- **Java 11** or later
+- **Maven 3.8+** for building the project
+
+## Getting Started
+
+### 1. Add Dependency
+
+Add the AWS Glue Catalog connector to your Flink project:
+
+```xml
+<dependency>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connector-aws-glue-catalog</artifactId>
+ <version>${flink.version}</version>
+</dependency>
+```
+
+### 2. Configure AWS Credentials
+
Review Comment:
Hmm... Seems like maybe we should make this more generic, rather than
specific to table catalog. We can discuss and fine tune after.
##########
flink-catalog-aws-glue/README.md:
##########
@@ -0,0 +1,377 @@
+# Flink AWS Glue Catalog Connector
+
+The Flink AWS Glue Catalog connector provides integration between Apache Flink
and the AWS Glue Data Catalog. This connector enables Flink applications to use
AWS Glue as a metadata catalog for tables, databases, and schemas, allowing
seamless SQL queries against AWS resources.
+
+## Features
+
+- Register AWS Glue as a catalog in Flink applications
+- Access Glue databases and tables through Flink SQL
+- Support for various AWS data sources (S3, Kinesis, MSK)
+- Mapping between Flink and AWS Glue data types
+- Compatibility with Flink's Table API and SQL interface
+
+## Prerequisites
+
+Before getting started, ensure you have the following:
+
+- **Apache Flink 1.20+** installed or running in a managed environment
+- **AWS account** with appropriate permissions for AWS Glue and other required
services
+- **AWS credentials** properly configured
+- **Java 11** or later
+- **Maven 3.8+** for building the project
+
Review Comment:
Flink version, java and maven version are controlled by the
`Flink-connector-aws` project. Let's either remove these, or link the `pom.xml`
for the project to maintain the source of truth in one place!
##########
flink-catalog-aws-glue/pom.xml:
##########
@@ -0,0 +1,200 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xmlns="http://maven.apache.org/POM/4.0.0";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connectors</artifactId>
+ <version>1.20.0</version>
+ </parent>
+
+ <artifactId>flink-catalog-aws-glue</artifactId>
+ <name>Flink : Catalogs : AWS Glue</name>
+ <packaging>jar</packaging>
+
+ <properties>
+ <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
+ <target.java.version>11</target.java.version>
+ <maven.compiler.source>${target.java.version}</maven.compiler.source>
+ <maven.compiler.target>${target.java.version}</maven.compiler.target>
+ <flink.version>1.20.0</flink.version>
+ <scala.binary.version>2.12</scala.binary.version>
+ <kda.runtime.version>1.2.0</kda.runtime.version>
+ <log4j.version>2.17.2</log4j.version>
+ <slf4j.version>1.7.32</slf4j.version>
+ <aws.sdk.version>2.31.17</aws.sdk.version>
+ </properties>
+
+ <dependencies>
Review Comment:
Can we remove `kda.runtime.version` please?
Also we should defer to the main `pom.xml` in `flink-connector-aws` when
possible (all of these)
##########
flink-catalog-aws-glue/dependency-reduced-pom.xml:
##########
@@ -0,0 +1,207 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<project xmlns="http://maven.apache.org/POM/4.0.0";
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+ <groupId>com.amazonaws</groupId>
+ <artifactId>flink-data-catalog</artifactId>
+ <version>1.0</version>
+ <build>
+ <plugins>
+ <plugin>
+ <artifactId>maven-compiler-plugin</artifactId>
+ <version>3.8.1</version>
Review Comment:
+1 - let's delegate this to the project setup on `flink-connector-aws`
##########
flink-catalog-aws-glue/pom.xml:
##########
@@ -0,0 +1,200 @@
+<?xml version="1.0" encoding="UTF-8"?>
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+-->
+<project xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
+ xmlns="http://maven.apache.org/POM/4.0.0";
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
+ <modelVersion>4.0.0</modelVersion>
+
+ <parent>
+ <groupId>org.apache.flink</groupId>
+ <artifactId>flink-connectors</artifactId>
+ <version>1.20.0</version>
Review Comment:
This versioning is not correct - can we use `flink-connector-aws` as the
parent instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
