[jira] [Closed] (CARBONDATA-3944) Delete stage files was interrupted when IOException happen

2020-08-11 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3944?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu closed CARBONDATA-3944.
-
Resolution: Fixed

> Delete stage files was interrupted when IOException happen
> --
>
> Key: CARBONDATA-3944
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3944
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Xingjun Hao
>Priority: Major
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> In the insertstage flow, the stage files will be deleted with retry 
> mechanism. but then IOException happen due to network abnormal etc, the 
> delete stage flow will be interrupted, which is unexpected.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Closed] (CARBONDATA-3829) Support pagination in SDK reader

2020-07-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu closed CARBONDATA-3829.
-
Resolution: Fixed

> Support pagination in SDK reader
> 
>
> Key: CARBONDATA-3829
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3829
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Ajantha Bhat
>Assignee: Ajantha Bhat
>Priority: Major
> Attachments: CarbonData SDK support pagination.pdf
>
>  Time Spent: 14h
>  Remaining Estimate: 0h
>
> Please find the solution document attached. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Closed] (CARBONDATA-3858) Check CDC deltafiles count in the testcase

2020-06-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu closed CARBONDATA-3858.
-
  Assignee: (was: Bo Xu)
Resolution: Fixed

> Check CDC deltafiles count in the testcase
> --
>
> Key: CARBONDATA-3858
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3858
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Xingjun Hao
>Priority: Minor
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Current there is no deltafiles count check in the testcase, which shall be 
> supplemented.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Reopened] (CARBONDATA-3858) Check CDC deltafiles count in the testcase

2020-06-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu reopened CARBONDATA-3858:
---

> Check CDC deltafiles count in the testcase
> --
>
> Key: CARBONDATA-3858
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3858
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Xingjun Hao
>Priority: Minor
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Current there is no deltafiles count check in the testcase, which shall be 
> supplemented.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Assigned] (CARBONDATA-3858) Check CDC deltafiles count in the testcase

2020-06-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu reassigned CARBONDATA-3858:
-

Assignee: Bo Xu

> Check CDC deltafiles count in the testcase
> --
>
> Key: CARBONDATA-3858
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3858
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Xingjun Hao
>Assignee: Bo Xu
>Priority: Minor
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Current there is no deltafiles count check in the testcase, which shall be 
> supplemented.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Closed] (CARBONDATA-3858) Check CDC deltafiles count in the testcase

2020-06-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu closed CARBONDATA-3858.
-
Resolution: Fixed

Xingjun Hao  fixed it

> Check CDC deltafiles count in the testcase
> --
>
> Key: CARBONDATA-3858
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3858
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Xingjun Hao
>Priority: Minor
>  Time Spent: 3h 40m
>  Remaining Estimate: 0h
>
> Current there is no deltafiles count check in the testcase, which shall be 
> supplemented.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Closed] (CARBONDATA-3860) Fix IndexServer keeps loading some segments index repeatly

2020-06-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu closed CARBONDATA-3860.
-
Resolution: Fixed

> Fix IndexServer keeps loading some segments index repeatly
> --
>
> Key: CARBONDATA-3860
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3860
> Project: CarbonData
>  Issue Type: Bug
>Reporter: Xingjun Hao
>Priority: Major
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> In current 
> getTableBlockIndexUniqueIdentifiers function. if the 
> segmentBlockIndexInfo.getSegmentMetaDataInfo() is null, the IndexServer will 
> keeps loading the index of this segment repeatly. We shall avoid to let it 
> affect query performance, considering MetaDataInfo doesn't matter with the 
> query processing.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (CARBONDATA-3701) Apache CarbonData should provides python interface to support deep learning framework MXNet to ready and write data from/to CarbonData

2020-02-13 Thread Bo Xu (Jira)
Bo Xu created CARBONDATA-3701:
-

 Summary: Apache CarbonData should provides python interface to 
support deep learning framework MXNet to ready and write data from/to CarbonData
 Key: CARBONDATA-3701
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3701
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Bo Xu
Assignee: Bo Xu


Nowadays AI model training is getting more and more popular. Currently many AI 
framework uses raw data files or row format data files for model training, it 
could not provide projection, filtering, and fast scan capability like in 
columnar store. So, if CarbonData supports AI framework, it can speed up model 
training by increase IO throughput, and provide more flexible training set 
selection ability to AI developers

AI compute engine integration:

MXNet integration: New python API in pycarbon to support MXNet to read data 
from CarbonData files for training model



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3271) Apache CarbonData should provides python interface to support deep learning framework TensorFlow to ready and write data from/to CarbonData

2020-02-12 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3271:
--
Description: 
Apache CarbonData should provides python interface to support deep learning 
framework to ready and write data from/to CarbonData, like TensorFlow , MXNet, 
PyTorch and so on. 

Basic framework:

Supports shuffle read, which reads the data in random order when feeding data 
to training model for each epoch.
Supports data cache to improve reading speed for multiple epoch, including 
local-disk and memory-cache.
Supports parallel reading using thread pool and process pool in python.
Supports reading data in object storage
Supports manifest format and CarbonData folder
AI compute engine integration:

Tensorflow integration:  
New python API in pycarbon to support TensorFlow to read data from 
CarbonData files for training model



  was:
Apache CarbonData should provides python interface to support deep learning 
framework to ready and write data from/to CarbonData, like TensorFlow , MXNet, 
PyTorch and so on. It should not dependency Apache Spark.

Goals:
1. CarbonData provides python interface to support TensorFlow to ready data 
from CarbonData for training model
2. CarbonData provides python interface to support MXNet to ready data from 
CarbonData for training model
3. CarbonData provides python interface to support PyTorch to ready data from 
CarbonData for training model
4. CarbonData should support epoch function
5. CarbonData should support cache for speed up performance.



Summary: Apache CarbonData should provides python interface to support 
deep learning framework TensorFlow to ready and write data from/to CarbonData  
(was: Apache CarbonData should provides python interface to support deep 
learning framework to ready and write data from/to CarbonData)

> Apache CarbonData should provides python interface to support deep learning 
> framework TensorFlow to ready and write data from/to CarbonData
> ---
>
> Key: CARBONDATA-3271
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3271
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
> Fix For: 2.0.0
>
>  Time Spent: 14h 40m
>  Remaining Estimate: 0h
>
> Apache CarbonData should provides python interface to support deep learning 
> framework to ready and write data from/to CarbonData, like TensorFlow , 
> MXNet, PyTorch and so on. 
> Basic framework:
> Supports shuffle read, which reads the data in random order when feeding data 
> to training model for each epoch.
> Supports data cache to improve reading speed for multiple epoch, including 
> local-disk and memory-cache.
> Supports parallel reading using thread pool and process pool in python.
> Supports reading data in object storage
> Supports manifest format and CarbonData folder
> AI compute engine integration:
> Tensorflow integration:  
> New python API in pycarbon to support TensorFlow to read data from 
> CarbonData files for training model



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (CARBONDATA-3695) Apache CarbonData should provides python interface to support deep learning framework PyTorch to ready and write data from/to CarbonData

2020-02-12 Thread Bo Xu (Jira)
Bo Xu created CARBONDATA-3695:
-

 Summary: Apache CarbonData should provides python interface to 
support deep learning framework PyTorch to ready and write data from/to 
CarbonData
 Key: CARBONDATA-3695
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3695
 Project: CarbonData
  Issue Type: Sub-task
Reporter: Bo Xu
Assignee: Bo Xu


Nowadays AI model training is getting more and more popular. Currently many AI 
framework uses raw data files or row format data files for model training, it 
could not provide projection, filtering, and fast scan capability like in 
columnar store. So, if CarbonData supports AI framework, it can speed up model 
training by increase IO throughput, and provide more flexible training set 
selection ability to AI developers

AI compute engine integration:

PyTorch integration: New python API in pycarbon to support PyTorch to read data 
from CarbonData files for training model





--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (CARBONDATA-3656) There are some conflicts when concurrently write data to S3

2020-01-08 Thread Bo Xu (Jira)
Bo Xu created CARBONDATA-3656:
-

 Summary: There are some conflicts when concurrently write data to 
S3
 Key: CARBONDATA-3656
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3656
 Project: CarbonData
  Issue Type: Bug
Reporter: Bo Xu
Assignee: Bo Xu


There are some conflicts when concurrently write data to S3



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Created] (CARBONDATA-3627) C++ SDK support write data withSchemaFile

2019-12-22 Thread Bo Xu (Jira)
Bo Xu created CARBONDATA-3627:
-

 Summary: C++ SDK support write data withSchemaFile
 Key: CARBONDATA-3627
 URL: https://issues.apache.org/jira/browse/CARBONDATA-3627
 Project: CarbonData
  Issue Type: New Feature
Reporter: Bo Xu
Assignee: Bo Xu


C++ SDK support write data withSchemaFile, which can be used for add segment 
for transactional table



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3283) Apache CarbonData should provides python interface to manage and analysis data based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap feature in P

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3283:
--
Description: 
Apache CarbonData should provides python interface to manage and analysis data 
based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap 
feature in Python.

Goals:

1). PyCarbon support  read data from local/HDFS/S3 in python code by PySpark 
DataFrame
2). PyCarbon support  write data in python code to local/HDFS/S3 by PySpark 
DataFrame
3). PyCarbon support  DDL in python with sql format
4). PyCarbon support  DML in python with sql format
5). PyCarbon support  DataMap in python with sql format

  was:
Apache CarbonData should provides python interface to manage and analysis data 
based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap 
feature in Python.

TODO: 


> Apache CarbonData should provides python interface to manage and analysis 
> data based on Apache Spark. Apache CarbonData should support DDL, DML, 
> DataMap feature in Python.
> ---
>
> Key: CARBONDATA-3283
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3283
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
> Apache CarbonData should provides python interface to manage and analysis 
> data based on Apache Spark. Apache CarbonData should support DDL, DML, 
> DataMap feature in Python.
> Goals:
> 1). PyCarbon support  read data from local/HDFS/S3 in python code by PySpark 
> DataFrame
> 2). PyCarbon support  write data in python code to local/HDFS/S3 by PySpark 
> DataFrame
> 3). PyCarbon support  DDL in python with sql format
> 4). PyCarbon support  DML in python with sql format
> 5). PyCarbon support  DataMap in python with sql format



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3254) PyCarbon: provide python interface for users to use CarbonData by python code

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3254:
--
Description: 
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData has many great feature and high performance to 
storage, manage and analysis big data. Apache CarbonData not only already 
supported String, Int, Double, Boolean, Char,Date, TImeStamp data types, but 
also supported Binay 
[(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336], 
which can avoid small binary files problem and speed up S3 access performance 
reach dozens or even hundreds of times, also can decrease cost of accessing OBS 
by decreasing the number of calling S3 API. But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code

We already work for these feature several months in  
https://github.com/xubo245/pycarbon

 Goals:
1. Apache CarbonData should provides python interface to support to write and 
read structured and unstructured data in CarbonData, like String, int and 
binary data: image/voice/video. It should not dependency Apache Spark.
2. Apache CarbonData should provides python interface to support deep learning 
framework to ready and write data from/to CarbonData, like TensorFlow , MXNet, 
PyTorch and so on. It should not dependency Apache Spark.
3. Apache CarbonData should provides python interface to manage and analysis 
data based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap 
feature  in Python. 

 



  was:
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData has many great feature and high performance to 
storage, manage and analysis big data. Apache CarbonData not only already 
supported String, Int, Double, Boolean, Char,Date, TImeStamp data types, but 
also supported Binay 
[(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336], 
which can avoid small binary files problem and speed up S3 access performance 
reach dozens or even hundreds of times, also can decrease cost of accessing OBS 
by decreasing the number of calling S3 API. But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code

 Goals:
1. Apache CarbonData should provides python interface to support to write and 
read structured and unstructured data in CarbonData, like String, int and 
binary data: image/voice/video. It should not dependency Apache Spark.
2. Apache CarbonData should provides python interface to support deep learning 
framework to ready and write data from/to CarbonData, like TensorFlow , MXNet, 
PyTorch and so on. It should not dependency Apache Spark.
3. Apache CarbonData should provides python interface to manage and analysis 
data based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap 
feature  in Python. 

 




> PyCarbon: provide python interface for users to use CarbonData by python code
> -
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
>   More and more people use big data to optimize their algorithm, train 
> their model, deploy their model as service and inference image.  It's big 
> challenge to storage, manage and analysis lots of structured and unstructured 
> data, especially unstructured data, like image, video, audio and so on.
>  Many users use python to install their project for these scenario. 
> Apache CarbonData is an indexed columnar data store solution for fast 
> analytics on big data platform. Apache CarbonData has many great feature and 
> high performance to storage, manage and analysis big data. Apache CarbonData 
> not only already supported 

[jira] [Updated] (CARBONDATA-3255) CarbonData provides python interface to support to write and read structured and unstructured data in CarbonData

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3255:
--
Description: 
Apache CarbonData already provide Java/ Scala/C++ interface for users, and more 
and more people use python to manage and analysis big data, so it's better to 
provide python interface to support to write and read structured and 
unstructured data in CarbonData, like String, int and binary data: 
image/voice/video. It should not dependency Apache Spark. We called it is PYSDK.

PYSDK based on CarbonData Java SDK, use pyjnius to call java code in python 
code.  Even though Apache Spark use py4j in PySpark to call java code in 
python, but it's low performance when use py4j to read bigdata with CarbonData 
format in python code, py4j also show low performance when read big data in 
their report: https://www.py4j.org/advanced_topics.html#performance.  JPype is 
also a popular tool to call java code in python, but it already stoped update 
several years ago, so we can not use it.  In our test, pyjnius has high 
performance to read big data by call java code in python, so it's good choice 
for us.

We already work for these feature several months in  
https://github.com/xubo245/pycarbon
Goals:

1. PYSDK should provide interface to support read data
2. PYSDK should provide interface to support write data
3. PYSDK should support basic data types
4. PYSDK should support projection
5. PYSDK should support filter



  was:

Apache CarbonData already provide Java/ Scala/C++ interface for users, and more 
and more people use python to manage and analysis big data, so it's better to 
provide python interface to support to write and read structured and 
unstructured data in CarbonData, like String, int and binary data: 
image/voice/video. It should not dependency Apache Spark. We called it is PYSDK.

PYSDK based on CarbonData Java SDK, use pyjnius to call java code in python 
code.  Even though Apache Spark use py4j in PySpark to call java code in 
python, but it's low performance when use py4j to read bigdata with CarbonData 
format in python code, py4j also show low performance when read big data in 
their report: https://www.py4j.org/advanced_topics.html#performance.  JPype is 
also a popular tool to call java code in python, but it already stoped update 
several years ago, so we can not use it.  In our test, pyjnius has high 
performance to read big data by call java code in python, so it's good choice 
for us.

Goals:

1. PYSDK should provide interface to support read data
2. PYSDK should provide interface to support write data
3. PYSDK should support basic data types
4. PYSDK should support projection
5. PYSDK should support filter




> CarbonData provides python interface to support to write and read structured 
> and unstructured data in CarbonData
> 
>
> Key: CARBONDATA-3255
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3255
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> Apache CarbonData already provide Java/ Scala/C++ interface for users, and 
> more and more people use python to manage and analysis big data, so it's 
> better to provide python interface to support to write and read structured 
> and unstructured data in CarbonData, like String, int and binary data: 
> image/voice/video. It should not dependency Apache Spark. We called it is 
> PYSDK.
> PYSDK based on CarbonData Java SDK, use pyjnius to call java code in python 
> code.  Even though Apache Spark use py4j in PySpark to call java code in 
> python, but it's low performance when use py4j to read bigdata with 
> CarbonData format in python code, py4j also show low performance when read 
> big data in their report: 
> https://www.py4j.org/advanced_topics.html#performance.  JPype is also a 
> popular tool to call java code in python, but it already stoped update 
> several years ago, so we can not use it.  In our test, pyjnius has high 
> performance to read big data by call java code in python, so it's good choice 
> for us.
> We already work for these feature several months in  
> https://github.com/xubo245/pycarbon
> Goals:
> 1. PYSDK should provide interface to support read data
> 2. PYSDK should provide interface to support write data
> 3. PYSDK should support basic data types
> 4. PYSDK should support projection
> 5. PYSDK should support filter



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3283) Apache CarbonData should provides python interface to manage and analysis data based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap feature in P

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3283:
--
Description: 
Apache CarbonData should provides python interface to manage and analysis data 
based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap 
feature in Python.

TODO: 

  was:WIP

Summary: Apache CarbonData should provides python interface to manage 
and analysis data based on Apache Spark. Apache CarbonData should support DDL, 
DML, DataMap feature in Python.  (was: WIP)

> Apache CarbonData should provides python interface to manage and analysis 
> data based on Apache Spark. Apache CarbonData should support DDL, DML, 
> DataMap feature in Python.
> ---
>
> Key: CARBONDATA-3283
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3283
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
> Apache CarbonData should provides python interface to manage and analysis 
> data based on Apache Spark. Apache CarbonData should support DDL, DML, 
> DataMap feature in Python.
> TODO: 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3271) Apache CarbonData should provides python interface to support deep learning framework to ready and write data from/to CarbonData

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3271:
--
Affects Version/s: (was: 1.5.1)
  Description: 
Apache CarbonData should provides python interface to support deep learning 
framework to ready and write data from/to CarbonData, like TensorFlow , MXNet, 
PyTorch and so on. It should not dependency Apache Spark.

Goals:
1. CarbonData provides python interface to support TensorFlow to ready data 
from CarbonData for training model
2. CarbonData provides python interface to support MXNet to ready data from 
CarbonData for training model
3. CarbonData provides python interface to support PyTorch to ready data from 
CarbonData for training model
4. CarbonData should support epoch function
5. CarbonData should support cache for speed up performance.


  Summary: Apache CarbonData should provides python interface to 
support deep learning framework to ready and write data from/to CarbonData  
(was: WIP)

> Apache CarbonData should provides python interface to support deep learning 
> framework to ready and write data from/to CarbonData
> 
>
> Key: CARBONDATA-3271
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3271
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>
> Apache CarbonData should provides python interface to support deep learning 
> framework to ready and write data from/to CarbonData, like TensorFlow , 
> MXNet, PyTorch and so on. It should not dependency Apache Spark.
> Goals:
> 1. CarbonData provides python interface to support TensorFlow to ready data 
> from CarbonData for training model
> 2. CarbonData provides python interface to support MXNet to ready data from 
> CarbonData for training model
> 3. CarbonData provides python interface to support PyTorch to ready data from 
> CarbonData for training model
> 4. CarbonData should support epoch function
> 5. CarbonData should support cache for speed up performance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3254) PyCarbon: provide python interface for users to use CarbonData by python code

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3254:
--
Description: 
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData has many great feature and high performance to 
storage, manage and analysis big data. Apache CarbonData not only already 
supported String, Int, Double, Boolean, Char,Date, TImeStamp data types, but 
also supported Binay 
[(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336], 
which can avoid small binary files problem and speed up S3 access performance 
reach dozens or even hundreds of times, also can decrease cost of accessing OBS 
by decreasing the number of calling S3 API. But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code

 Goals:
1. Apache CarbonData should provides python interface to support to write and 
read structured and unstructured data in CarbonData, like String, int and 
binary data: image/voice/video. It should not dependency Apache Spark.
2. Apache CarbonData should provides python interface to support deep learning 
framework to ready and write data from/to CarbonData, like TensorFlow , MXNet, 
PyTorch and so on. It should not dependency Apache Spark.
3. Apache CarbonData should provides python interface to manage and analysis 
data based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap 
feature  in Python. 

 



  was:
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform. Apache CarbonData has many great feature 
and high performance to storage, manage and analysis big data. Apache 
CarbonData not only already supported String, Int, Double, Boolean, Char,Date, 
TImeStamp data types, but also supported Binay 
[(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336], 
which can avoid small binary files problem and speed up S3 access performance 
reach dozens or even hundreds of times, also can decrease cost of accessing OBS 
by decreasing the number of calling S3 API. But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code

 Goals:
1. Apache CarbonData should provides python interface to support to write and 
read structured and unstructured data in CarbonData, like String, int and 
binary data: image/voice/video. It should not dependency Apache Spark.
2. Apache CarbonData should provides python interface to support deep learning 
framework to ready and write data from/to CarbonData, like TensorFlow , MXNet, 
PyTorch and so on. It should not dependency Apache Spark.
3. Apache CarbonData should provides python interface to manage and analysis 
data based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap 
feature  in Python. 

 




> PyCarbon: provide python interface for users to use CarbonData by python code
> -
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
>   More and more people use big data to optimize their algorithm, train 
> their model, deploy their model as service and inference image.  It's big 
> challenge to storage, manage and analysis lots of structured and unstructured 
> data, especially unstructured data, like image, video, audio and so on.
>  Many users use python to install their project for these scenario. 
> Apache CarbonData is an indexed columnar data store solution for fast 
> analytics on big data platform. Apache CarbonData has many great feature and 
> high performance to storage, manage and analysis big data. Apache CarbonData 
> not only already 

[jira] [Updated] (CARBONDATA-3255) CarbonData provides python interface to support to write and read structured and unstructured data in CarbonData

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3255:
--
Description: 

Apache CarbonData already provide Java/ Scala/C++ interface for users, and more 
and more people use python to manage and analysis big data, so it's better to 
provide python interface to support to write and read structured and 
unstructured data in CarbonData, like String, int and binary data: 
image/voice/video. It should not dependency Apache Spark. We called it is PYSDK.

PYSDK based on CarbonData Java SDK, use pyjnius to call java code in python 
code.  Even though Apache Spark use py4j in PySpark to call java code in 
python, but it's low performance when use py4j to read bigdata with CarbonData 
format in python code, py4j also show low performance when read big data in 
their report: https://www.py4j.org/advanced_topics.html#performance.  JPype is 
also a popular tool to call java code in python, but it already stoped update 
several years ago, so we can not use it.  In our test, pyjnius has high 
performance to read big data by call java code in python, so it's good choice 
for us.

Goals:

1. PYSDK should provide interface to support read data
2. PYSDK should provide interface to support write data
3. PYSDK should support basic data types
4. PYSDK should support projection
5. PYSDK should support filter


Summary: CarbonData provides python interface to support to write and 
read structured and unstructured data in CarbonData  (was: WIP)

> CarbonData provides python interface to support to write and read structured 
> and unstructured data in CarbonData
> 
>
> Key: CARBONDATA-3255
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3255
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>  Time Spent: 7h
>  Remaining Estimate: 0h
>
> Apache CarbonData already provide Java/ Scala/C++ interface for users, and 
> more and more people use python to manage and analysis big data, so it's 
> better to provide python interface to support to write and read structured 
> and unstructured data in CarbonData, like String, int and binary data: 
> image/voice/video. It should not dependency Apache Spark. We called it is 
> PYSDK.
> PYSDK based on CarbonData Java SDK, use pyjnius to call java code in python 
> code.  Even though Apache Spark use py4j in PySpark to call java code in 
> python, but it's low performance when use py4j to read bigdata with 
> CarbonData format in python code, py4j also show low performance when read 
> big data in their report: 
> https://www.py4j.org/advanced_topics.html#performance.  JPype is also a 
> popular tool to call java code in python, but it already stoped update 
> several years ago, so we can not use it.  In our test, pyjnius has high 
> performance to read big data by call java code in python, so it's good choice 
> for us.
> Goals:
> 1. PYSDK should provide interface to support read data
> 2. PYSDK should provide interface to support write data
> 3. PYSDK should support basic data types
> 4. PYSDK should support projection
> 5. PYSDK should support filter



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3254) PyCarbon: provide python interface for users to use CarbonData by python code

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3254:
--
Description: 
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform. Apache CarbonData has many great feature 
and high performance to storage, manage and analysis big data. Apache 
CarbonData not only already supported String, Int, Double, Boolean, Char,Date, 
TImeStamp data types, but also supported Binay 
[(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336], 
which can avoid small binary files problem and speed up S3 access performance 
reach dozens or even hundreds of times, also can decrease cost of accessing OBS 
by decreasing the number of calling S3 API. But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code

 Goals:
1. Apache CarbonData should provides python interface to support to write and 
read structured and unstructured data in CarbonData, like String, int and 
binary data: image/voice/video. It should not dependency Apache Spark.
2. Apache CarbonData should provides python interface to support deep learning 
framework to ready and write data from/to CarbonData, like TensorFlow , MXNet, 
PyTorch and so on. It should not dependency Apache Spark.
3. Apache CarbonData should provides python interface to manage and analysis 
data based on Apache Spark. Apache CarbonData should support DDL, DML, DataMap 
feature  in Python. 

 



  was:
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform. Apache CarbonData has many great feature 
and high performance to storage, manage and analysis big data. Apache 
CarbonData not only already supported String, Int, Double, Boolean, Char,Date, 
TImeStamp data types, but also supported Binay 
[(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336], 
which can avoid small binary files problem and speed up S3 access performance 
reach dozens or even hundreds of times, also can decrease cost of accessing OBS 
by decreasing the number of calling S3 API. But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code
 




> PyCarbon: provide python interface for users to use CarbonData by python code
> -
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
>   More and more people use big data to optimize their algorithm, train 
> their model, deploy their model as service and inference image.  It's big 
> challenge to storage, manage and analysis lots of structured and unstructured 
> data, especially unstructured data, like image, video, audio and so on.
>  Many users use python to install their project for these scenario. 
> Apache CarbonData is an indexed columnar data store solution for fast 
> analytics on big data platform. Apache CarbonData is an indexed columnar data 
> store solution for fast analytics on big data platform. Apache CarbonData has 
> many great feature and high performance to storage, manage and analysis big 
> data. Apache CarbonData not only already supported String, Int, Double, 
> Boolean, Char,Date, TImeStamp data types, but also supported Binay 
> [(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336], 
> which can avoid small binary files problem and speed up S3 access performance 
> reach dozens or even hundreds of times, also can decrease cost of accessing 
> OBS by decreasing the number of calling S3 API. But it's not easy for them to 
> use 

[jira] [Updated] (CARBONDATA-3254) PyCarbon: provide python interface for users to use CarbonData by python code

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3254:
--
Description: 
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform. Apache CarbonData has many great feature 
and high performance to storage, manage and analysis big data. Apache 
CarbonData not only already supported String, Int, Double, Boolean, Char,Date, 
TImeStamp data types, but also supported Binay 
[(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336], 
which can avoid small binary files problem and speed up S3 access performance 
reach dozens or even hundreds of times, also can decrease cost of accessing OBS 
by decreasing the number of calling S3 API. But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code
 



  was:
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform. Apache CarbonData has many great feature 
and high performance to storage, manage and analysis big data. Apache 
CarbonData not only already supported String, Int, Double, Boolean, Char,Date, 
TImeStamp data types, but also supported Binay 
[(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336].But 
it's not easy for them to use carbon by Java/Scala/C++. So it's better to 
provide python interface for these users to use CarbonData by python code
 
Apache CarbonData already 



> PyCarbon: provide python interface for users to use CarbonData by python code
> -
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
>   More and more people use big data to optimize their algorithm, train 
> their model, deploy their model as service and inference image.  It's big 
> challenge to storage, manage and analysis lots of structured and unstructured 
> data, especially unstructured data, like image, video, audio and so on.
>  Many users use python to install their project for these scenario. 
> Apache CarbonData is an indexed columnar data store solution for fast 
> analytics on big data platform. Apache CarbonData is an indexed columnar data 
> store solution for fast analytics on big data platform. Apache CarbonData has 
> many great feature and high performance to storage, manage and analysis big 
> data. Apache CarbonData not only already supported String, Int, Double, 
> Boolean, Char,Date, TImeStamp data types, but also supported Binay 
> [(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336], 
> which can avoid small binary files problem and speed up S3 access performance 
> reach dozens or even hundreds of times, also can decrease cost of accessing 
> OBS by decreasing the number of calling S3 API. But it's not easy for them to 
> use carbon by Java/Scala/C++. So it's better to provide python interface for 
> these users to use CarbonData by python code
>  
> 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3254) PyCarbon: provide python interface for users to use CarbonData by python code

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3254:
--
Description: 
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform. Apache CarbonData has many great feature 
and high performance to storage, manage and analysis big data. Apache 
CarbonData not only already supported String, Int, Double, Boolean, Char,Date, 
TImeStamp data types, but also supported Binay 
[(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336].But 
it's not easy for them to use carbon by Java/Scala/C++. So it's better to 
provide python interface for these users to use CarbonData by python code
 
Apache CarbonData already 


  was:
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform has many great feature and high performance 
to storage, manage and analysis big data.But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code


> PyCarbon: provide python interface for users to use CarbonData by python code
> -
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
>   More and more people use big data to optimize their algorithm, train 
> their model, deploy their model as service and inference image.  It's big 
> challenge to storage, manage and analysis lots of structured and unstructured 
> data, especially unstructured data, like image, video, audio and so on.
>  Many users use python to install their project for these scenario. 
> Apache CarbonData is an indexed columnar data store solution for fast 
> analytics on big data platform. Apache CarbonData is an indexed columnar data 
> store solution for fast analytics on big data platform. Apache CarbonData has 
> many great feature and high performance to storage, manage and analysis big 
> data. Apache CarbonData not only already supported String, Int, Double, 
> Boolean, Char,Date, TImeStamp data types, but also supported Binay 
> [(CARBONDATA-3336)|https://issues.apache.org/jira/browse/CARBONDATA-3336].But 
> it's not easy for them to use carbon by Java/Scala/C++. So it's better to 
> provide python interface for these users to use CarbonData by python code
>  
> Apache CarbonData already 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3254) PyCarbon: provide python interface for users to use CarbonData by python code

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3254:
--
Description: 
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.
 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform has many great feature and high performance 
to storage, manage and analysis big data.But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code

> PyCarbon: provide python interface for users to use CarbonData by python code
> -
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
>   More and more people use big data to optimize their algorithm, train 
> their model, deploy their model as service and inference image.  It's big 
> challenge to storage, manage and analysis lots of structured and unstructured 
> data, especially unstructured data, like image, video, audio and so on.
>  Many users use python to install their project for these scenario. 
> Apache CarbonData is an indexed columnar data store solution for fast 
> analytics on big data platform. Apache CarbonData is an indexed columnar data 
> store solution for fast analytics on big data platform has many great feature 
> and high performance to storage, manage and analysis big data.But it's not 
> easy for them to use carbon by Java/Scala/C++. So it's better to provide 
> python interface for these users to use CarbonData by python code



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3254) PyCarbon: provide python interface for users to use CarbonData by python code

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3254:
--
Description: 
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.


 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform has many great feature and high performance 
to storage, manage and analysis big data.But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code

  was:
  More and more people use big data to optimize their algorithm, train 
their model, deploy their model as service and inference image.  It's big 
challenge to storage, manage and analysis lots of structured and unstructured 
data, especially unstructured data, like image, video, audio and so on.
 Many users use python to install their project for these scenario. Apache 
CarbonData is an indexed columnar data store solution for fast analytics on big 
data platform. Apache CarbonData is an indexed columnar data store solution for 
fast analytics on big data platform has many great feature and high performance 
to storage, manage and analysis big data.But it's not easy for them to use 
carbon by Java/Scala/C++. So it's better to provide python interface for these 
users to use CarbonData by python code


> PyCarbon: provide python interface for users to use CarbonData by python code
> -
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
>   More and more people use big data to optimize their algorithm, train 
> their model, deploy their model as service and inference image.  It's big 
> challenge to storage, manage and analysis lots of structured and unstructured 
> data, especially unstructured data, like image, video, audio and so on.
>  Many users use python to install their project for these scenario. 
> Apache CarbonData is an indexed columnar data store solution for fast 
> analytics on big data platform. Apache CarbonData is an indexed columnar data 
> store solution for fast analytics on big data platform has many great feature 
> and high performance to storage, manage and analysis big data.But it's not 
> easy for them to use carbon by Java/Scala/C++. So it's better to provide 
> python interface for these users to use CarbonData by python code



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Reopened] (CARBONDATA-3283) WIP

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu reopened CARBONDATA-3283:
---

contribute pycarbon to Apache CarbonData

> WIP
> ---
>
> Key: CARBONDATA-3283
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3283
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>
> WIP



--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Reopened] (CARBONDATA-3255) WIP

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu reopened CARBONDATA-3255:
---

contribute pycarbon to Apache CarbonData

> WIP
> ---
>
> Key: CARBONDATA-3255
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3255
> Project: CarbonData
>  Issue Type: Sub-task
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>  Time Spent: 7h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3254) PyCarbon: provide python interface for users to use CarbonData by python code

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3254:
--
Summary: PyCarbon: provide python interface for users to use CarbonData by 
python code  (was: PyCarbon: provide python SDK for user to use CarbonData by 
python code)

> PyCarbon: provide python interface for users to use CarbonData by python code
> -
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Updated] (CARBONDATA-3254) PyCarbon: provide python SDK for user to use CarbonData by python code

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu updated CARBONDATA-3254:
--
Summary: PyCarbon: provide python SDK for user to use CarbonData by python 
code  (was: [WIP] )

> PyCarbon: provide python SDK for user to use CarbonData by python code
> --
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Reopened] (CARBONDATA-3254) [WIP]

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu reopened CARBONDATA-3254:
---

Contribute  Pycarbon to Apache CarbonData

> [WIP] 
> --
>
> Key: CARBONDATA-3254
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3254
> Project: CarbonData
>  Issue Type: New Feature
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)


[jira] [Reopened] (CARBONDATA-3271) WIP

2019-11-23 Thread Bo Xu (Jira)


 [ 
https://issues.apache.org/jira/browse/CARBONDATA-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Xu reopened CARBONDATA-3271:
---

push pycarbon to Apache CarbonData

> WIP
> ---
>
> Key: CARBONDATA-3271
> URL: https://issues.apache.org/jira/browse/CARBONDATA-3271
> Project: CarbonData
>  Issue Type: Sub-task
>Affects Versions: 1.5.1
>Reporter: Bo Xu
>Assignee: Bo Xu
>Priority: Major
>  Time Spent: 7.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)