[
https://issues.apache.org/jira/browse/CARBONDATA-3925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
yutao updated CARBONDATA-3925:
--
Description:
in CarbonWriter.java code ,you can find this;
public abstract class *{color:#FF}CarbonWriter{color}* extends
ProxyFileWriter {
private static final Logger LOGGER =
LogServiceFactory.getLogService({color:#FF}CarbonS3Writer{color}.class.getName());}
always wo can find logfile print like ;
2020-07-27 14:19:25,107 DEBUG org.apache.carbon.flink.CarbonS3Writer
this is puzzledï¼›
2020-08-19 13:14:13,329 INFO
org.apache.carbondata.hadoop.api.CarbonTableOutputFormat - Closed writer task
attempt_f229b922-1f77-426f-a4bc-42e49aa53df7__m_1968253873_-1049302646
2020-08-19 13:14:13,329 DEBUG org.apache.carbon.flink.CarbonLocalWriter -
Commit write. org.apache.carbon.flink.CarbonLocalWriter@41f5c4a9
2020-08-19 13:14:13,329 DEBUG org.apache.carbon.flink.CarbonS3Writer - Upload
file[/home/hadoop/yutest/d963e9836ccb4318aa8fc953af983d07/part-0-a132f98547584dcabae6c43090626baf_batchno0-0-null-1597814047953.snappy.carbondata]
to [hdfs://beh/user/dc_cbss/warehouse/yutest/tf_b_trade/stage_data] start.
2020-08-19 13:14:13,329 INFO org.apache.carbondata.core.util.CarbonUtil -
Copying
/home/hadoop/yutest/d963e9836ccb4318aa8fc953af983d07/part-0-a132f98547584dcabae6c43090626baf_batchno0-0-null-1597814047953.snappy.carbondata
to hdfs://beh/user/dc_cbss/warehouse/yutest/tf_b_trade/stage_data, operation
id 1597814053329
2020-08-19 13:14:13,331 DEBUG org.apache.carbondata.core.util.CarbonUtil
-{color:#FF} The configured block size is 1 KB, the actual carbon file size
is 277 KB, choose the max value 277 KB as the block size on HDFS{color}
2020-08-19 13:14:13,331 DEBUG org.apache.carbondata.core.util.CarbonUtil - HDFS
file block size for file:
hdfs://beh/user/dc_cbss/warehouse/yutest/tf_b_trade/stage_data/part-0-a132f98547584dcabae6c43090626baf_batchno0-0-null-1597814047953.snappy.carbondata
is 284160 (bytes
2020-08-19 13:14:13,332 INFO
org.apache.carbondata.processing.util.CarbonLoaderUtil - Deleted the local
store location:
/tmp/f97548ae6efc43d2ba269c9d35295bb9_attempt_f229b922-1f77-426f-a4bc-42e49aa53df7__m_1968253873_-1049302646
: Time taken: 2
2020-08-19 13:14:13,358 ERROR org.apache.carbon.flink.CarbonS3Writer - Problem
while copying file from local store to carbon store
org.apache.carbondata.core.datastore.exception.CarbonDataWriterException:
Problem while copying file from local store to carbon store
at
org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2694)
at
org.apache.carbon.flink.CarbonWriter.uploadSegmentDataFiles(CarbonWriter.java:90)
at org.apache.carbon.flink.CarbonLocalWriter.commit(CarbonLocalWriter.java:155)
at org.apache.carbon.flink.CarbonLocalWriter.flush(CarbonLocalWriter.java:129)
at
org.apache.flink.streaming.api.functions.sink.filesystem.BulkPartWriter.closeForCommit(BulkPartWriter.java:61)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.closePartFile(Bucket.java:239)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.prepareBucketForCheckpointing(Bucket.java:280)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.onReceptionOfCheckpoint(Bucket.java:253)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.snapshotActiveBuckets(Buckets.java:250)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.snapshotState(Buckets.java:241)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSink.snapshotState(StreamingFileSink.java:447)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:118)
at
org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:99)
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:90)
at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:402)
at
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.checkpointStreamOperator(StreamTask.java:1403)
at
org.apache.flink.streaming.runtime.tasks.StreamTask$CheckpointingOperation.executeCheckpointing(StreamTask.java:1337)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.checkpointState(StreamTask.java:974)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$5(StreamTask.java:870)
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:94)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:843)
at
