zhilinli123 commented on PR #5427:
URL: https://github.com/apache/seatunnel/pull/5427#issuecomment-1715168843
## Hive Source table
```
CREATE TABLE `default.my_table`(
`id` int,
`name` string)
PARTITIONED BY (
`year` int,
`month` int)
ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
STORED AS INPUTFORMAT
'org.apache.hadoop.mapred.TextInputFormat'
OUTPUTFORMAT
'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'
LOCATION
'hdfs://localhost:9000/user/hive/warehouse/my_table'
TBLPROPERTIES (
'bucketing_version'='2',
'transient_lastDdlTime'='1693883163');
```
## Mysql Sink table
```
CREATE TABLE `my_table_hive` (
`id` int NOT NULL,
`name` varchar(25) DEFAULT NULL,
`year` int DEFAULT NULL,
`month` int DEFAULT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_0900_ai_ci
```
## Spark Demo Conf
```
env {
# You can set spark configuration here
# see available properties defined by spark:
https://spark.apache.org/docs/latest/configuration.html#available-properties
#job.mode = BATCH
job.name = "SeaTunnel"
spark.executor.instances = 1
spark.executor.cores = 1
spark.executor.memory = "1g"
spark.master = local
}
source {
Hive {
table_name = "default.my_table"
metastore_uri = "thrift://localhost:9083"
}
}
transform {
}
sink {
jdbc {
url = "jdbc:mysql://localhost:3306/test"
driver = "com.mysql.cj.jdbc.Driver"
user = "root"
password = "12345678"
query = "insert into my_table_hive (id, name, year,month) values
(?,?,?,?)"
}
}
```
```
## Hdfs Dir
drwxr-xr-x - mac supergroup 0 2023-09-05 11:32
hdfs://localhost:9000/user/hive/warehouse/my_table/year=2023/month=7
drwxr-xr-x - mac supergroup 0 2023-09-05 11:07
hdfs://localhost:9000/user/hive/warehouse/my_table/year=2023/month=8
drwxr-xr-x - mac supergroup 0 2023-09-05 11:06
hdfs://localhost:9000/user/hive/warehouse/my_table/year=2023/month=9
mac@zhilinliMac ~ % hadoop fs -ls
hdfs://localhost:9000/user/hive/warehouse/my_table/year=2023/month=7
2023-09-12 15:35:09,997 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
mac@zhilinliMac ~ % hadoop fs -ls
hdfs://localhost:9000/user/hive/warehouse/my_table/year=2023/month=8
2023-09-12 15:36:42,740 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
hadoop fs -ls
hdfs://localhost:9000/user/hive/warehouse/my_table/year=2023/month=9
2023-09-12 15:35:40,602 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java classes where
applicable
Found 1 items
-rw-r--r-- 1 mac supergroup 14 2023-09-05 11:06
hdfs://localhost:9000/user/hive/warehouse/my_table/year=2023/month=9/000000_0
## hive query Data
select * from default.my_table_hive;
1 Alice 2023 9
2 Bob 2023 9
```
<img width="685" alt="image"
src="https://github.com/apache/seatunnel/assets/76689593/93063844-3fd8-42b3-bfac-214e9da43526";>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
