chenchurong opened a new issue, #4452: URL: https://github.com/apache/incubator-seatunnel/issues/4452
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-seatunnel/issues?q=is%3Aissue+label%3A%22bug%22) and found no similar issues. ### What happened two table from diff dataSource (source-sqlserver & sink-hive) , ddl sql just as below,but i cant appoint hive-partition HiveDDl： CREATE TABLE IF NOT EXISTS signal_r.ods_jdbc_jy_cfi_guarantee_1( ID string COMMENT '' , CompanyCode string COMMENT '' , EnterpriseCode string COMMENT '', InfoPublDate string COMMENT '' , InfoSource string COMMENT '' , InfoSourceCode string COMMENT '', EndDate string COMMENT '', GuarantBusiType string COMMENT '' , CurrencyUnit string COMMENT '' , IndicatorCode string COMMENT '' , DataValue string COMMENT '' , TotalMark string COMMENT '' , GUARdesc string COMMENT '' , Remark string COMMENT '' , AID string COMMENT '' , InsertTime string COMMENT '' , UpdateTime string COMMENT '' , JSID string COMMENT '' , BeginDate string COMMENT '' , FiscalYear string COMMENT '' , PeriodMark string COMMENT '' , kp_synctime string COMMENT '' )PARTITIONED BY (day_id string) ROW FORMAT DELIMITED FIELDS TERMINATED BY '|' STORED AS TEXTFILE; SqlServer DDl： CREATE TABLE [dbo].[CFI_GuaranteeTotal] ( [ID] bigint NOT NULL, [CompanyCode] int NOT NULL, [EnterpriseCode] varchar(12) COLLATE Chinese_PRC_CI_AS NULL, [InfoPublDate] datetime NOT NULL, [InfoSource] varchar(200) COLLATE Chinese_PRC_CI_AS NULL, [InfoSourceCode] varchar(12) COLLATE Chinese_PRC_CI_AS NOT NULL, [EndDate] datetime NOT NULL, [GuarantBusiType] varchar(12) COLLATE Chinese_PRC_CI_AS NOT NULL, [CurrencyUnit] varchar(12) COLLATE Chinese_PRC_CI_AS NOT NULL, [IndicatorCode] varchar(12) COLLATE Chinese_PRC_CI_AS NOT NULL, [DataValue] decimal(19,4) NULL, [TotalMark] varchar(12) COLLATE Chinese_PRC_CI_AS NOT NULL, [GUARdesc] varchar(max) COLLATE Chinese_PRC_CI_AS NULL, [Remark] varchar(max) COLLATE Chinese_PRC_CI_AS NULL, [AID] varchar(50) COLLATE Chinese_PRC_CI_AS NULL, [InsertTime] datetime NOT NULL, [UpdateTime] datetime NOT NULL, [JSID] bigint NOT NULL, [BeginDate] datetime NULL, [FiscalYear] datetime NULL, [PeriodMark] varchar(12) COLLATE Chinese_PRC_CI_AS NULL, [kp_synctime] datetime NULL ) ### SeaTunnel Version 2.3.1 ### SeaTunnel Config ```conf env { job.mode = "BATCH" } source { jdbc { password = "*****" driver = "com.microsoft.sqlserver.jdbc.SQLServerDriver" user = "*****" url = "jdbc:sqlserver://IP:1113;DatabaseName=jydb;encrypt=false" query = "select * from dbo.CFI_GuaranteeTotal" } } transform { } sink { Hive { table_name = "signal_r.ods_jdbc_jy_cfi_guarantee_1" metastore_uri = "thrift://IP:9083" } } ``` ### Running Command ```shell ./seatunnel.sh --config ../config/sqlserver2hive2.conf -e local ``` ### Error Exception ```log 2023-03-30 09:43:54,359 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSource Plugin from /data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel 2023-03-30 09:43:54,370 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: jdbc at: file:/data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-jdbc-2.3.1.jar 2023-03-30 09:43:54,375 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='source', pluginName='jdbc'} from path: file:/data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-jdbc-2.3.1.jar use classloader: org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader 2023-03-30 09:43:54,802 INFO org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSource - The partition_column parameter is not configured, and the source parallelism is set to 1 2023-03-30 09:43:54,808 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load SeaTunnelSink Plugin from /data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel 2023-03-30 09:43:54,809 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Discovery plugin jar: Hive at: file:/data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar 2023-03-30 09:43:54,814 INFO org.apache.seatunnel.plugin.discovery.AbstractPluginDiscovery - Load plugin: PluginIdentifier{engineType='seatunnel', pluginType='sink', pluginName='Hive'} from path: file:/data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar use classloader: org.apache.seatunnel.engine.common.loader.SeaTunnelChildFirstClassLoader 2023-03-30 09:43:54,848 INFO org.apache.hadoop.hive.conf.HiveConf - Found configuration file file:/data/hadoop/apache-hive-2.3.9-bin/conf/hive-site.xml 2023-03-30 09:43:55,242 INFO hive.metastore - Trying to connect to metastore with URI thrift://10.119.169.84:9083 2023-03-30 09:43:55,261 INFO hive.metastore - Opened a connection to metastore, current connections: 1 2023-03-30 09:43:55,288 WARN org.apache.hadoop.util.NativeCodeLoader - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-30 09:43:55,353 INFO hive.metastore - Connected to metastore. 2023-03-30 09:43:55,576 INFO hive.metastore - Closed a connection to metastore, current connections: 0 2023-03-30 09:43:55,645 INFO org.apache.seatunnel.engine.client.job.ClientJobProxy - start submit job, job id: 693636522976477185, with plugin jar [file:/data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar, file:/data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-jdbc-2.3.1.jar] 2023-03-30 09:43:55,651 WARN org.apache.seatunnel.engine.server.SeaTunnelServer - This is master node, waiting the coordinator service init finished 2023-03-30 09:43:55,717 INFO org.apache.seatunnel.engine.server.CoordinatorService - [localhost]:5801 [seatunnel-8145] [5.1] This node become a new active master node, begin init coordinator service 2023-03-30 09:43:56,157 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - Init ResourceManager 2023-03-30 09:43:56,157 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker... 2023-03-30 09:43:56,158 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - initWorker live nodes: [[localhost]:5801] 2023-03-30 09:43:56,159 INFO org.apache.seatunnel.engine.server.resourcemanager.AbstractResourceManager - registerWorker: {[localhost]:5801=WorkerProfile(address=[localhost]:5801, profile=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=477626368}}, unassignedResource=ResourceProfile{cpu=CPU{core=0}, heapMemory=Memory{bytes=477626368}}, assignedSlots=[], unassignedSlots=[])} 2023-03-30 09:43:56,178 INFO org.apache.seatunnel.engine.server.master.JobMaster - Init JobMaster for Job SeaTunnel_Job (693636522976477185) 2023-03-30 09:43:56,178 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (693636522976477185) needed jar urls [file:/data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-hive-2.3.1.jar, file:/data/hadoop/apache-seatunnel-incubating-2.3.1/connectors/seatunnel/connector-jdbc-2.3.1.jar] 2023-03-30 09:43:56,556 INFO org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage - Path /tmp/seatunnel/checkpoint_snapshot/693636522976477185 is not a directory 2023-03-30 09:43:56,556 INFO org.apache.seatunnel.engine.checkpoint.storage.hdfs.HdfsStorage - No checkpoint found for job, job id is: 693636522976477185 2023-03-30 09:43:56,559 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc-default]-SourceTask (1/1)] is in state CREATED when init state future 2023-03-30 09:43:56,560 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc-default]-SplitEnumerator (1/1)] is in state CREATED when init state future 2023-03-30 09:43:56,561 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - The task Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-Hive-default]-AggregatedCommitterTask (1/1)] is in state CREATED when init state future 2023-03-30 09:43:56,565 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job Job SeaTunnel_Job (693636522976477185) waiting for scheduler finished 2023-03-30 09:43:56,569 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - Job SeaTunnel_Job (693636522976477185) turn from state CREATED to SCHEDULED. 2023-03-30 09:43:56,584 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)] turn from state CREATED to SCHEDULED. 2023-03-30 09:43:56,599 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc-default]-SplitEnumerator (1/1)] turn from state CREATED to SCHEDULED. 2023-03-30 09:43:56,599 WARN org.apache.seatunnel.engine.client.job.JobMetricsRunner - Failed to get job metrics summary, it maybe first-run 2023-03-30 09:43:56,607 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-Hive-default]-AggregatedCommitterTask (1/1)] turn from state CREATED to SCHEDULED. 2023-03-30 09:43:56,611 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc-default]-SourceTask (1/1)] turn from state CREATED to SCHEDULED. 2023-03-30 09:43:56,624 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)] turn from state SCHEDULED to DEPLOYING. 2023-03-30 09:43:56,628 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc-default]-SplitEnumerator (1/1)] turn from state SCHEDULED to DEPLOYING. 2023-03-30 09:43:56,631 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-Hive-default]-AggregatedCommitterTask (1/1)] turn from state SCHEDULED to DEPLOYING. 2023-03-30 09:43:56,634 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc-default]-SourceTask (1/1)] turn from state SCHEDULED to DEPLOYING. 2023-03-30 09:43:56,642 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-8145] [5.1] received deploying task executionId [693636533076361217] 2023-03-30 09:43:56,649 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-8145] [5.1] received deploying task executionId [693636533076426753] 2023-03-30 09:43:56,659 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-8145] [5.1] received deploying task executionId [693636533076492289] 2023-03-30 09:43:56,681 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - starting seatunnel source split enumerator task, source name: pipeline-1 [Source[0]-Jdbc-default] 2023-03-30 09:43:56,686 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc-default]-SplitEnumerator (1/1)] turn from state DEPLOYING to RUNNING. 2023-03-30 09:43:56,759 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Source[0]-Jdbc-default]-SourceTask (1/1)] turn from state DEPLOYING to RUNNING. 2023-03-30 09:43:56,760 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalVertex - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)], task: [pipeline-1 [Sink[0]-Hive-default]-AggregatedCommitterTask (1/1)] turn from state DEPLOYING to RUNNING. 2023-03-30 09:43:56,765 INFO org.apache.seatunnel.engine.server.dag.physical.SubPlan - Job SeaTunnel_Job (693636522976477185), Pipeline: [(1/1)] turn from state DEPLOYING to RUNNING. 2023-03-30 09:43:56,769 INFO org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask - starting seatunnel transform task, index 0 2023-03-30 09:43:56,773 INFO org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask - starting seatunnel source task, index 0 2023-03-30 09:43:56,783 INFO org.apache.seatunnel.engine.server.dag.physical.PhysicalPlan - Job SeaTunnel_Job (693636522976477185) turn from state SCHEDULED to RUNNING. 2023-03-30 09:43:56,791 INFO org.apache.seatunnel.engine.server.master.JobMaster - Job SeaTunnel_Job (693636522976477185) scheduler finished 2023-03-30 09:43:56,802 INFO hive.metastore - Trying to connect to metastore with URI thrift://10.119.169.84:9083 2023-03-30 09:43:56,802 INFO hive.metastore - Opened a connection to metastore, current connections: 1 2023-03-30 09:43:56,811 INFO hive.metastore - Connected to metastore. 2023-03-30 09:43:56,813 INFO hive.metastore - Closed a connection to metastore, current connections: 0 2023-03-30 09:43:56,827 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received reader register, readerID: TaskLocation{taskGroupLocation=TaskGroupLocation{jobId=693636522976477185, pipelineId=1, taskGroupId=50000}, taskID=60000, index=0} 2023-03-30 09:43:56,833 INFO org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask - received enough reader, starting enumerator... 2023-03-30 09:43:56,834 INFO org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceSplitEnumerator - Starting to calculate splits. 2023-03-30 09:43:56,834 INFO org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceSplitEnumerator - Assigning JdbcSourceSplit(parameterValues=null, splitId=0) to 0 reader. 2023-03-30 09:43:56,834 INFO org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceSplitEnumerator - Assign splits [JdbcSourceSplit(parameterValues=null, splitId=0)] to reader 0 2023-03-30 09:43:56,871 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-8145] [5.1] Exception in org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask@592ba930 java.lang.RuntimeException: java.lang.NullPointerException at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:206) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:59) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:39) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.task.SeaTunnelTransformCollector.collect(SeaTunnelTransformCollector.java:27) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.handleRecord(IntermediateBlockingQueue.java:76) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.task.group.queue.IntermediateBlockingQueue.collect(IntermediateBlockingQueue.java:51) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.task.flow.IntermediateQueueFlowLifeCycle.collect(IntermediateQueueFlowLifeCycle.java:52) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.collect(TransformSeaTunnelTask.java:73) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.task.SeaTunnelTask.stateProcess(SeaTunnelTask.java:161) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.task.TransformSeaTunnelTask.call(TransformSeaTunnelTask.java:78) ~[seatunnel-starter.jar:2.3.1] at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:526) [seatunnel-starter.jar:2.3.1] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_362] at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_362] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_362] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_362] at java.lang.Thread.run(Thread.java:750) [?:1.8.0_362] Caused by: java.lang.NullPointerException at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.AbstractWriteStrategy.generatorPartitionDir(AbstractWriteStrategy.java:203) ~[connector-hive-2.3.1.jar:2.3.1] at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.AbstractWriteStrategy.getOrCreateFilePathBeingWritten(AbstractWriteStrategy.java:367) ~[connector-hive-2.3.1.jar:2.3.1] at org.apache.seatunnel.connectors.seatunnel.file.sink.writer.TextWriteStrategy.write(TextWriteStrategy.java:79) ~[connector-hive-2.3.1.jar:2.3.1] at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:126) ~[connector-hive-2.3.1.jar:2.3.1] at org.apache.seatunnel.connectors.seatunnel.file.sink.BaseFileSinkWriter.write(BaseFileSinkWriter.java:43) ~[connector-hive-2.3.1.jar:2.3.1] at org.apache.seatunnel.engine.server.task.flow.SinkFlowLifeCycle.received(SinkFlowLifeCycle.java:201) ~[seatunnel-starter.jar:2.3.1] ... 15 more 2023-03-30 09:43:56,878 INFO org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-8145] [5.1] taskDone, taskId = 70000, taskGroup = TaskGroupLocation{jobId=693636522976477185, pipelineId=1, taskGroupId=50000} 2023-03-30 09:43:56,879 WARN org.apache.seatunnel.engine.server.TaskExecutionService - [localhost]:5801 [seatunnel-8145] [5.1] Exception in org.apache.seatunnel.engine.server.task.SourceSeaTunnelTask@126bdcdb ``` ### Flink or Spark Version no ### Java or Scala Version 1.8 ### Screenshots _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
