[jira] [Created] (CARBONDATA-3861) Support show stage information
Zhi Liu created CARBONDATA-3861: --- Summary: Support show stage information Key: CARBONDATA-3861 URL: https://issues.apache.org/jira/browse/CARBONDATA-3861 Project: CarbonData Issue Type: New Feature Reporter: Zhi Liu Sometimes, user need to know information about table stages, so that they can make stage load plan better. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3821) Cache database metadata for mv
Zhi Liu created CARBONDATA-3821: --- Summary: Cache database metadata for mv Key: CARBONDATA-3821 URL: https://issues.apache.org/jira/browse/CARBONDATA-3821 Project: CarbonData Issue Type: Improvement Reporter: Zhi Liu Assignee: Zhi Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3804) Provide end-to-end flink integration guide
Zhi Liu created CARBONDATA-3804: --- Summary: Provide end-to-end flink integration guide Key: CARBONDATA-3804 URL: https://issues.apache.org/jira/browse/CARBONDATA-3804 Project: CarbonData Issue Type: Bug Reporter: Zhi Liu Assignee: Zhi Liu Provide an end-to-end guide to help user understand and use flink integration module.. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CARBONDATA-3798) Fix mv not refresh after insert into fact table from stage
[ https://issues.apache.org/jira/browse/CARBONDATA-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu closed CARBONDATA-3798. --- Resolution: Duplicate > Fix mv not refresh after insert into fact table from stage > -- > > Key: CARBONDATA-3798 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3798 > Project: CarbonData > Issue Type: Bug >Reporter: Zhi Liu >Assignee: Zhi Liu >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > After insert into non-partition table from stage, does not trigger all > materialized views which related the fact table auto refresh. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3798) Fix mv not refresh after insert into fact table from stage
[ https://issues.apache.org/jira/browse/CARBONDATA-3798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu updated CARBONDATA-3798: Description: After insert into non-partition table from stage, does not trigger all materialized views which related the fact table auto refresh. > Fix mv not refresh after insert into fact table from stage > -- > > Key: CARBONDATA-3798 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3798 > Project: CarbonData > Issue Type: Bug >Reporter: Zhi Liu >Assignee: Zhi Liu >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > After insert into non-partition table from stage, does not trigger all > materialized views which related the fact table auto refresh. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3798) Fix mv not refresh after insert into fact table from stage
Zhi Liu created CARBONDATA-3798: --- Summary: Fix mv not refresh after insert into fact table from stage Key: CARBONDATA-3798 URL: https://issues.apache.org/jira/browse/CARBONDATA-3798 Project: CarbonData Issue Type: Bug Reporter: Zhi Liu Assignee: Zhi Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3776) Clean old materialized view implementation
[ https://issues.apache.org/jira/browse/CARBONDATA-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu resolved CARBONDATA-3776. - Resolution: Fixed > Clean old materialized view implementation > -- > > Key: CARBONDATA-3776 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3776 > Project: CarbonData > Issue Type: Task >Reporter: Zhi Liu >Assignee: Zhi Liu >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > After the materialized view is separated from the DataMap module, the old > materialized view implementation in DataMap module is not used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3775) Update materialized view document
[ https://issues.apache.org/jira/browse/CARBONDATA-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu resolved CARBONDATA-3775. - Resolution: Fixed > Update materialized view document > - > > Key: CARBONDATA-3775 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3775 > Project: CarbonData > Issue Type: Improvement >Reporter: Zhi Liu >Assignee: Zhi Liu >Priority: Major > Time Spent: 8h 10m > Remaining Estimate: 0h > > Update materialized view document synchronously, after the materialized view > is separated from the DataMap module. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3778) Change the way of mv module get database location
Zhi Liu created CARBONDATA-3778: --- Summary: Change the way of mv module get database location Key: CARBONDATA-3778 URL: https://issues.apache.org/jira/browse/CARBONDATA-3778 Project: CarbonData Issue Type: Bug Reporter: Zhi Liu Assignee: Zhi Liu Change the way of mv module get database location to recommend way. Currently , the recommend way is using CarbonEvn.getDatabaseLocation，not using carbon.storelocation variable. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CARBONDATA-3776) Clean old materialized view implementation
[ https://issues.apache.org/jira/browse/CARBONDATA-3776?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu reassigned CARBONDATA-3776: --- Assignee: Zhi Liu > Clean old materialized view implementation > -- > > Key: CARBONDATA-3776 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3776 > Project: CarbonData > Issue Type: Task >Reporter: Zhi Liu >Assignee: Zhi Liu >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > After the materialized view is separated from the DataMap module, the old > materialized view implementation in DataMap module is not used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3776) Clean old materialized view implementation
Zhi Liu created CARBONDATA-3776: --- Summary: Clean old materialized view implementation Key: CARBONDATA-3776 URL: https://issues.apache.org/jira/browse/CARBONDATA-3776 Project: CarbonData Issue Type: Task Reporter: Zhi Liu After the materialized view is separated from the DataMap module, the old materialized view implementation in DataMap module is not used. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (CARBONDATA-3775) Update materialized view document
[ https://issues.apache.org/jira/browse/CARBONDATA-3775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu reassigned CARBONDATA-3775: --- Assignee: Zhi Liu > Update materialized view document > - > > Key: CARBONDATA-3775 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3775 > Project: CarbonData > Issue Type: Improvement >Reporter: Zhi Liu >Assignee: Zhi Liu >Priority: Major > > Update materialized view document synchronously, after the materialized view > is separated from the DataMap module. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3775) Update materialized view document
Zhi Liu created CARBONDATA-3775: --- Summary: Update materialized view document Key: CARBONDATA-3775 URL: https://issues.apache.org/jira/browse/CARBONDATA-3775 Project: CarbonData Issue Type: Improvement Reporter: Zhi Liu Update materialized view document synchronously, after the materialized view is separated from the DataMap module. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3714) Support specify order type when list stage files
[ https://issues.apache.org/jira/browse/CARBONDATA-3714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu updated CARBONDATA-3714: Description: Sometimes, user want load the lastest data to table first. > Support specify order type when list stage files > > > Key: CARBONDATA-3714 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3714 > Project: CarbonData > Issue Type: New Feature >Reporter: Zhi Liu >Priority: Major > > Sometimes, user want load the lastest data to table first. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3714) Support specify order type when list stage files
Zhi Liu created CARBONDATA-3714: --- Summary: Support specify order type when list stage files Key: CARBONDATA-3714 URL: https://issues.apache.org/jira/browse/CARBONDATA-3714 Project: CarbonData Issue Type: New Feature Reporter: Zhi Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3709) Move stage_data directory to the $tablePath/ from $tablePath/Metadata/
[ https://issues.apache.org/jira/browse/CARBONDATA-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu resolved CARBONDATA-3709. - Resolution: Fixed > Move stage_data directory to the $tablePath/ from $tablePath/Metadata/ > -- > > Key: CARBONDATA-3709 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3709 > Project: CarbonData > Issue Type: New Feature >Reporter: Zhi Liu >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Move stage_data directory to the $tablePath/ from $tablePath/Metadata/, and > keep the $tablePath/Metadata/ directory not include any data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3709) Move stage_data directory to the $tablePath/ from $tablePath/Metadata/
[ https://issues.apache.org/jira/browse/CARBONDATA-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu updated CARBONDATA-3709: Description: Move stage_data directory to the $tablePath/ from $tablePath/Metadata/, and keep the $tablePath/Metadata/ directory clean. > Move stage_data directory to the $tablePath/ from $tablePath/Metadata/ > -- > > Key: CARBONDATA-3709 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3709 > Project: CarbonData > Issue Type: New Feature >Reporter: Zhi Liu >Priority: Major > > Move stage_data directory to the $tablePath/ from $tablePath/Metadata/, and > keep the $tablePath/Metadata/ directory clean. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3709) Move stage_data directory to the $tablePath/ from $tablePath/Metadata/
[ https://issues.apache.org/jira/browse/CARBONDATA-3709?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu updated CARBONDATA-3709: Description: Move stage_data directory to the $tablePath/ from $tablePath/Metadata/, and keep the $tablePath/Metadata/ directory not include any data. (was: Move stage_data directory to the $tablePath/ from $tablePath/Metadata/, and keep the $tablePath/Metadata/ directory clean.) > Move stage_data directory to the $tablePath/ from $tablePath/Metadata/ > -- > > Key: CARBONDATA-3709 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3709 > Project: CarbonData > Issue Type: New Feature >Reporter: Zhi Liu >Priority: Major > > Move stage_data directory to the $tablePath/ from $tablePath/Metadata/, and > keep the $tablePath/Metadata/ directory not include any data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3709) Move stage_data directory to the $tablePath/ from $tablePath/Metadata/
Zhi Liu created CARBONDATA-3709: --- Summary: Move stage_data directory to the $tablePath/ from $tablePath/Metadata/ Key: CARBONDATA-3709 URL: https://issues.apache.org/jira/browse/CARBONDATA-3709 Project: CarbonData Issue Type: New Feature Reporter: Zhi Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3676) Support clean carbon data files of stages
[ https://issues.apache.org/jira/browse/CARBONDATA-3676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu updated CARBONDATA-3676: Description: At the end of the CarbonInsertFromStageCommand, the stage files will be cleared, but the data files which referenced by stage files will be not cleared. This could lead to a large backlog of data files。 > Support clean carbon data files of stages > - > > Key: CARBONDATA-3676 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3676 > Project: CarbonData > Issue Type: New Feature > Components: spark-integration >Reporter: Zhi Liu >Priority: Major > > At the end of the CarbonInsertFromStageCommand, the stage files will be > cleared, but the data files which referenced by stage files will be not > cleared. This could lead to a large backlog of data files。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3676) Support clean carbon data files of stages
Zhi Liu created CARBONDATA-3676: --- Summary: Support clean carbon data files of stages Key: CARBONDATA-3676 URL: https://issues.apache.org/jira/browse/CARBONDATA-3676 Project: CarbonData Issue Type: New Feature Components: spark-integration Reporter: Zhi Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3663) Support loading stage files in batches
Zhi Liu created CARBONDATA-3663: --- Summary: Support loading stage files in batches Key: CARBONDATA-3663 URL: https://issues.apache.org/jira/browse/CARBONDATA-3663 Project: CarbonData Issue Type: New Feature Reporter: Zhi Liu When there are a lots of stage files in the stage directory, if load all of them in once time, the loading time will can not be control. There need a way for users to specify the number of stage files per processing, to control the execution time of commands. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3661) Fix target file size check fail when upload local file to carbon store
[ https://issues.apache.org/jira/browse/CARBONDATA-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu resolved CARBONDATA-3661. - Resolution: Fixed > Fix target file size check fail when upload local file to carbon store > -- > > Key: CARBONDATA-3661 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3661 > Project: CarbonData > Issue Type: Bug >Reporter: Zhi Liu >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Multi flink tasks write carbon data may use the same carbon data file name, > it will cause target file size check fail when upload local file to carbon > store. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3660) Throw FileNotFoundException when concurrent loading
[ https://issues.apache.org/jira/browse/CARBONDATA-3660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu resolved CARBONDATA-3660. - Resolution: Fixed > Throw FileNotFoundException when concurrent loading > --- > > Key: CARBONDATA-3660 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3660 > Project: CarbonData > Issue Type: Bug >Reporter: Zhi Liu >Priority: Major > Time Spent: 10m > Remaining Estimate: 0h > > 2020-01-09 14:42:47 ERROR CarbonFactDataWriterImplV3:390 - Problem while > writing the index file2020-01-09 14:42:47 ERROR > CarbonFactDataWriterImplV3:390 - Problem while writing the index > fileorg.apache.carbondata.core.datastore.exception.CarbonDataWriterException: > Problem while copying file from local store to carbon store at > org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2772) > at > org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2721) > at > org.apache.carbondata.processing.store.writer.AbstractFactDataWriter.commitCurrentFile(AbstractFactDataWriter.java:277) > at > org.apache.carbondata.processing.store.writer.v3.CarbonFactDataWriterImplV3.closeWriter(CarbonFactDataWriterImplV3.java:387) > at > org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.closeHandler(CarbonFactDataHandlerColumnar.java:508) > at > org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.processingComplete(CarbonRowDataWriterProcessorStepImpl.java:233) > at > org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.finish(CarbonRowDataWriterProcessorStepImpl.java:211) > at > org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.doExecute(CarbonRowDataWriterProcessorStepImpl.java:175) > at > org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.execute(CarbonRowDataWriterProcessorStepImpl.java:129) > at > org.apache.carbondata.processing.loading.DataLoadExecutor.execute(DataLoadExecutor.java:52) > at > org.apache.carbondata.hadoop.api.CarbonTableOutputFormat$1.run(CarbonTableOutputFormat.java:278) > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > at java.lang.Thread.run(Thread.java:748)Caused by: > java.io.FileNotFoundException: > D:\Workspace\Carbon-Master\integration\flink\target\test-classes\data\temp\8df45d4dc38449c69147083cdbe79e4d\part-0-156_batchno0-0-null-1578552167465.carbondata > (系统找不到指定的路径。) at java.io.FileOutputStream.open0(Native Method) at > java.io.FileOutputStream.open(FileOutputStream.java:270) at > java.io.FileOutputStream.(FileOutputStream.java:213) at > java.io.FileOutputStream.(FileOutputStream.java:101) at > org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataOutputStream(LocalCarbonFile.java:371) > at > org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataOutputStream(LocalCarbonFile.java:365) > at > org.apache.carbondata.core.datastore.impl.FileFactory.getDataOutputStream(FileFactory.java:231) > at > org.apache.carbondata.core.util.CarbonUtil.copyLocalFileToCarbonStore(CarbonUtil.java:2799) > at > org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2756) > ... 15 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (CARBONDATA-3655) Support set base64 string as struct field value.
[ https://issues.apache.org/jira/browse/CARBONDATA-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu resolved CARBONDATA-3655. - Resolution: Fixed > Support set base64 string as struct field value. > > > Key: CARBONDATA-3655 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3655 > Project: CarbonData > Issue Type: New Feature >Reporter: Zhi Liu >Priority: Major > Time Spent: 6h 10m > Remaining Estimate: 0h > > Currently, only support set string with delimiter as struct field value, > sometime it doesn't work very well on struct field. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3661) Fix target file size check fail when upload local file to carbon store
Zhi Liu created CARBONDATA-3661: --- Summary: Fix target file size check fail when upload local file to carbon store Key: CARBONDATA-3661 URL: https://issues.apache.org/jira/browse/CARBONDATA-3661 Project: CarbonData Issue Type: Bug Reporter: Zhi Liu Multi flink tasks write carbon data may use the same carbon data file name, it will cause target file size check fail when upload local file to carbon store. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3660) Throw FileNotFoundException when concurrent loading
Zhi Liu created CARBONDATA-3660: --- Summary: Throw FileNotFoundException when concurrent loading Key: CARBONDATA-3660 URL: https://issues.apache.org/jira/browse/CARBONDATA-3660 Project: CarbonData Issue Type: Bug Reporter: Zhi Liu 2020-01-09 14:42:47 ERROR CarbonFactDataWriterImplV3:390 - Problem while writing the index file2020-01-09 14:42:47 ERROR CarbonFactDataWriterImplV3:390 - Problem while writing the index fileorg.apache.carbondata.core.datastore.exception.CarbonDataWriterException: Problem while copying file from local store to carbon store at org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2772) at org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2721) at org.apache.carbondata.processing.store.writer.AbstractFactDataWriter.commitCurrentFile(AbstractFactDataWriter.java:277) at org.apache.carbondata.processing.store.writer.v3.CarbonFactDataWriterImplV3.closeWriter(CarbonFactDataWriterImplV3.java:387) at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.closeHandler(CarbonFactDataHandlerColumnar.java:508) at org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.processingComplete(CarbonRowDataWriterProcessorStepImpl.java:233) at org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.finish(CarbonRowDataWriterProcessorStepImpl.java:211) at org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.doExecute(CarbonRowDataWriterProcessorStepImpl.java:175) at org.apache.carbondata.processing.loading.steps.CarbonRowDataWriterProcessorStepImpl.execute(CarbonRowDataWriterProcessorStepImpl.java:129) at org.apache.carbondata.processing.loading.DataLoadExecutor.execute(DataLoadExecutor.java:52) at org.apache.carbondata.hadoop.api.CarbonTableOutputFormat$1.run(CarbonTableOutputFormat.java:278) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.io.FileNotFoundException: D:\Workspace\Carbon-Master\integration\flink\target\test-classes\data\temp\8df45d4dc38449c69147083cdbe79e4d\part-0-156_batchno0-0-null-1578552167465.carbondata (系统找不到指定的路径。) at java.io.FileOutputStream.open0(Native Method) at java.io.FileOutputStream.open(FileOutputStream.java:270) at java.io.FileOutputStream.(FileOutputStream.java:213) at java.io.FileOutputStream.(FileOutputStream.java:101) at org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataOutputStream(LocalCarbonFile.java:371) at org.apache.carbondata.core.datastore.filesystem.LocalCarbonFile.getDataOutputStream(LocalCarbonFile.java:365) at org.apache.carbondata.core.datastore.impl.FileFactory.getDataOutputStream(FileFactory.java:231) at org.apache.carbondata.core.util.CarbonUtil.copyLocalFileToCarbonStore(CarbonUtil.java:2799) at org.apache.carbondata.core.util.CarbonUtil.copyCarbonDataFileToCarbonStorePath(CarbonUtil.java:2756) ... 15 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3655) Support set base64 string as struct field value.
[ https://issues.apache.org/jira/browse/CARBONDATA-3655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu updated CARBONDATA-3655: Description: Currently, only support set string with delimiter as struct field value, sometime it doesn't work very well on struct field. > Support set base64 string as struct field value. > > > Key: CARBONDATA-3655 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3655 > Project: CarbonData > Issue Type: New Feature >Reporter: Zhi Liu >Priority: Major > > Currently, only support set string with delimiter as struct field value, > sometime it doesn't work very well on struct field. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3655) Support set base64 string as struct field value.
Zhi Liu created CARBONDATA-3655: --- Summary: Support set base64 string as struct field value. Key: CARBONDATA-3655 URL: https://issues.apache.org/jira/browse/CARBONDATA-3655 Project: CarbonData Issue Type: New Feature Reporter: Zhi Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CARBONDATA-3654) Support set array as struct field value
[ https://issues.apache.org/jira/browse/CARBONDATA-3654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu closed CARBONDATA-3654. --- Resolution: Invalid > Support set array as struct field value > --- > > Key: CARBONDATA-3654 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3654 > Project: CarbonData > Issue Type: New Feature >Reporter: Zhi Liu >Priority: Major > > Currently, only support set string with delimiter as struct field value, > sometime it doesn't work very well on struct field. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3654) Support set array as struct field value
Zhi Liu created CARBONDATA-3654: --- Summary: Support set array as struct field value Key: CARBONDATA-3654 URL: https://issues.apache.org/jira/browse/CARBONDATA-3654 Project: CarbonData Issue Type: New Feature Reporter: Zhi Liu Currently, only support set string with delimiter as struct field value, sometime it doesn't work very well on struct field. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (CARBONDATA-3598) Throw java.util.ConcurrentModifitcationException when close carbon fact data writer.
[ https://issues.apache.org/jira/browse/CARBONDATA-3598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu closed CARBONDATA-3598. --- Resolution: Invalid > Throw java.util.ConcurrentModifitcationException when close carbon fact data > writer. > > > Key: CARBONDATA-3598 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3598 > Project: CarbonData > Issue Type: Bug > Components: core >Reporter: Zhi Liu >Priority: Major > Fix For: 2.0.0 > > Attachments: 20191128201907.png > > Time Spent: 1h > Remaining Estimate: 0h > > See the exception stack in attachment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3651) Add flink-build module to collect runtime packages which depended by flink module
[ https://issues.apache.org/jira/browse/CARBONDATA-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu updated CARBONDATA-3651: Description: Mostly，some of the flink module depends packages were provided by flink runtime environment, we need exclude those packages when build. > Add flink-build module to collect runtime packages which depended by flink > module > - > > Key: CARBONDATA-3651 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3651 > Project: CarbonData > Issue Type: New Feature >Reporter: Zhi Liu >Priority: Major > > Mostly，some of the flink module depends packages were provided by flink > runtime environment, we need exclude those packages when build. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3651) Add flink-build module to collect runtime packages which depended by flink module
Zhi Liu created CARBONDATA-3651: --- Summary: Add flink-build module to collect runtime packages which depended by flink module Key: CARBONDATA-3651 URL: https://issues.apache.org/jira/browse/CARBONDATA-3651 Project: CarbonData Issue Type: New Feature Reporter: Zhi Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3640) Insert from stage command support partition table.
[ https://issues.apache.org/jira/browse/CARBONDATA-3640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhi Liu updated CARBONDATA-3640: Description: Currently, The CarbonInsertFromStageCommand only support writing to a non-partition table. Sometimes, We also need it support writing to a partition table. > Insert from stage command support partition table. > -- > > Key: CARBONDATA-3640 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3640 > Project: CarbonData > Issue Type: New Feature >Reporter: Zhi Liu >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Currently, The CarbonInsertFromStageCommand only support writing to a > non-partition table. > Sometimes, We also need it support writing to a partition table. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3640) Insert from stage command support partition table.
Zhi Liu created CARBONDATA-3640: --- Summary: Insert from stage command support partition table. Key: CARBONDATA-3640 URL: https://issues.apache.org/jira/browse/CARBONDATA-3640 Project: CarbonData Issue Type: New Feature Reporter: Zhi Liu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (CARBONDATA-3598) Throw java.util.ConcurrentModifitcationException when close carbon fact data writer.
Zhi Liu created CARBONDATA-3598: --- Summary: Throw java.util.ConcurrentModifitcationException when close carbon fact data writer. Key: CARBONDATA-3598 URL: https://issues.apache.org/jira/browse/CARBONDATA-3598 Project: CarbonData Issue Type: Bug Components: core Reporter: Zhi Liu Fix For: 2.0.0 Attachments: 20191128201907.png See the exception stack in attachment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3557) Support write Flink streaming data to Carbon
[
https://issues.apache.org/jira/browse/CARBONDATA-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhi Liu updated CARBONDATA-3557:
Attachment: Support write flink streaming data to carbon.docx
> Support write Flink streaming data to Carbon
>
>
> Key: CARBONDATA-3557
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3557
> Project: CarbonData
> Issue Type: New Feature
> Components: spark-integration
>Reporter: Zhi Liu
>Priority: Major
> Fix For: 2.0.0
>
> Attachments: Support write flink streaming data to carbon.docx
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Sometimes, user need to write flink streaming data to carbon, required high
> concurrency and high throughput.
> The write process is:
> # Write flink streaming data to local file system of flink task node use
> flink StreamingFileSink and carbon SDK;
> # Copy local carbon data file to carbon data store system, such as HDFS, S3;
> # Generate and write segment file to ${tablePath}/load_details;
> Run "alter table ${tableName} collect segments" command on server, to compact
> segment files in ${tablePath}/load_details, and then move the compacted
> segment file to ${tablePath}/Metadata/Segments/，update table status file
> finally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3557) Support write Flink streaming data to Carbon
[
https://issues.apache.org/jira/browse/CARBONDATA-3557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhi Liu updated CARBONDATA-3557:
Description:
Sometimes, user need to write flink streaming data to carbon, required high
concurrency and high throughput.
The write process is:
# Write flink streaming data to local file system of flink task node use flink
StreamingFileSink and carbon SDK;
# Copy local carbon data file to carbon data store system, such as HDFS, S3;
# Generate and write segment file to ${tablePath}/load_details;
Run "alter table ${tableName} collect segments" command on server, to compact
segment files in ${tablePath}/load_details, and then move the compacted segment
file to ${tablePath}/Metadata/Segments/，update table status file finally.
was:
Sometimes, user need to write flink streaming data to carbon, required high
concurrency and high throughput.
> Support write Flink streaming data to Carbon
>
>
> Key: CARBONDATA-3557
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3557
> Project: CarbonData
> Issue Type: New Feature
> Components: spark-integration
>Reporter: Zhi Liu
>Priority: Major
> Fix For: 2.0.0
>
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> Sometimes, user need to write flink streaming data to carbon, required high
> concurrency and high throughput.
> The write process is:
> # Write flink streaming data to local file system of flink task node use
> flink StreamingFileSink and carbon SDK;
> # Copy local carbon data file to carbon data store system, such as HDFS, S3;
> # Generate and write segment file to ${tablePath}/load_details;
> Run "alter table ${tableName} collect segments" command on server, to compact
> segment files in ${tablePath}/load_details, and then move the compacted
> segment file to ${tablePath}/Metadata/Segments/，update table status file
> finally.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CARBONDATA-3557) Support write Flink streaming data to Carbon
Zhi Liu created CARBONDATA-3557: --- Summary: Support write Flink streaming data to Carbon Key: CARBONDATA-3557 URL: https://issues.apache.org/jira/browse/CARBONDATA-3557 Project: CarbonData Issue Type: New Feature Components: spark-integration Reporter: Zhi Liu Fix For: 2.0.0 Sometimes, user need to write flink streaming data to carbon, required high concurrency and high throughput. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3553) Support SDK Writer using existing schema file
[
https://issues.apache.org/jira/browse/CARBONDATA-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhi Liu updated CARBONDATA-3553:
Description:
Sometimes, user need to build SDK writer with schema file, rather than with
scheme object. It's will be easier to use.
{code}
val writer =
CarbonWriter.builder.outputPath(...).withSchemaFile("/default/test/Metadata/schema").withCsvInput().build()
{code}
was:
Sometimes, user need to build SDK writer with schema file, rather than with
scheme object. It's will be easier to use.
val writer =
CarbonWriter.builder.outputPath(...).withSchemaFile("/default/test/Metadata/schema").withCsvInput().build()
> Support SDK Writer using existing schema file
> -
>
> Key: CARBONDATA-3553
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3553
> Project: CarbonData
> Issue Type: Improvement
> Components: core, spark-integration
>Reporter: Zhi Liu
>Priority: Major
>
> Sometimes, user need to build SDK writer with schema file, rather than with
> scheme object. It's will be easier to use.
> {code}
> val writer =
> CarbonWriter.builder.outputPath(...).withSchemaFile("/default/test/Metadata/schema").withCsvInput().build()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3553) Support SDK Writer using existing schema file
[
https://issues.apache.org/jira/browse/CARBONDATA-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhi Liu updated CARBONDATA-3553:
Description:
Sometimes, user need to build SDK writer with schema file, rather than with
scheme object. It's will be easier to use.
val writer =
CarbonWriter.builder.outputPath(...).withSchemaFile("/default/test/Metadata/schema").withCsvInput().build()
was:
Sometimes, user need to build SDK writer with schema file, rather than with
scheme object. It's will be easier to use.
val writer =
CarbonWriter.builder.outputPath(...).withSchemaFile("/default/test/Metadata/schema").withCsvInput().build()
> Support SDK Writer using existing schema file
> -
>
> Key: CARBONDATA-3553
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3553
> Project: CarbonData
> Issue Type: Improvement
> Components: core, spark-integration
>Reporter: Zhi Liu
>Priority: Major
>
> Sometimes, user need to build SDK writer with schema file, rather than with
> scheme object. It's will be easier to use.
> val writer =
> CarbonWriter.builder.outputPath(...).withSchemaFile("/default/test/Metadata/schema").withCsvInput().build()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (CARBONDATA-3553) Support SDK Writer using existing schema file
[
https://issues.apache.org/jira/browse/CARBONDATA-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhi Liu updated CARBONDATA-3553:
Description:
Sometimes, user need to build SDK writer with schema file, rather than with
scheme object. It's will be easier to use.
val writer =
CarbonWriter.builder.outputPath(...).withSchemaFile("/default/test/Metadata/schema").withCsvInput().build()
was:Sometimes, user need to build SDK writer with schema file, rather than
with scheme object. It's will be easier to use.
> Support SDK Writer using existing schema file
> -
>
> Key: CARBONDATA-3553
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3553
> Project: CarbonData
> Issue Type: Improvement
> Components: core, spark-integration
>Reporter: Zhi Liu
>Priority: Major
>
> Sometimes, user need to build SDK writer with schema file, rather than with
> scheme object. It's will be easier to use.
>
> val writer =
> CarbonWriter.builder.outputPath(...).withSchemaFile("/default/test/Metadata/schema").withCsvInput().build()
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (CARBONDATA-3553) Support SDK Writer using existing schema file
Zhi Liu created CARBONDATA-3553: --- Summary: Support SDK Writer using existing schema file Key: CARBONDATA-3553 URL: https://issues.apache.org/jira/browse/CARBONDATA-3553 Project: CarbonData Issue Type: Improvement Components: core, spark-integration Reporter: Zhi Liu Sometimes, user need to build SDK writer with schema file, rather than with scheme object. It's will be easier to use. -- This message was sent by Atlassian Jira (v8.3.4#803005)
