[
https://issues.apache.org/jira/browse/CARBONDATA-3490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ajantha Bhat updated CARBONDATA-3490:
-------------------------------------
Fix Version/s: (was: 1.6.0)
1.6.1
> Concurrent data load failure with carbondata FileNotFound exception
> -------------------------------------------------------------------
>
> Key: CARBONDATA-3490
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3490
> Project: CarbonData
> Issue Type: Bug
> Reporter: Ajantha Bhat
> Priority: Major
> Fix For: 1.6.1
>
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> Caused by:
> org.apache.carbondata.core.datastore.exception.CarbonDataWriterException:
> Problem while copying file from local store to carbon store
> at
> org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2750)
> at
> org.apache.carbondata.processing.store.writer.AbstractFactDataWriter.commitCurrentFile(AbstractFactDataWriter.java:283)
> at
> org.apache.carbondata.processing.store.writer.v3.CarbonFactDataWriterImplV3.closeWriter(CarbonFactDataWriterImplV3.java:393)
> ... 11 more
> Caused by: java.io.FileNotFoundException:
> /tmp/carbon865982118689228_1/Fact/Part0/Segment_6/1/part-0-1_batchno0-0-6-1565329654844.carbondata
> (No such file or directory)
> at java.io.FileInputStream.open0(Native Method)
> at java.io.FileInputStream.open(FileInputStream.java:195)
> at java.io.FileInputStream.<init>(FileInputStream.java:138)
> at java.io.FileInputStream.<init>(FileInputStream.java:93)
> at
> org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataInputStream(LocalCarbonFile.java:309)
> at
> org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataInputStream(LocalCarbonFile.java:299)
> at
> org.apache.carbondata.core.datastore.impl.FileFactory.getDataInputStream(FileFactory.java:179)
> at
> org.apache.carbondata.core.datastore.impl.FileFactory.getDataInputStream(FileFactory.java:175)
> at
> org.apache.carbondata.core.util.CarbonUtil.copyLocalFileToCarbonStore(CarbonUtil.java:2781)
> at
> org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2746)
> ... 13 more
> problem: When two load is happening concurrently, one load is cleaning the
> temp directory of the concurrent load
> cause: temp directory to store the carbon files is created using system.get
> nano time, due to this two load have same store location. when one load is
> completed, it cleaned the temp directory. causing dataload failure for other
> load.
> solution:
> use UUID instead of nano time while creating the temp directory to have each
> load a unique directory.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
