Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83147351
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/iterators/RecordReaderIterator.java
---
@@ -0,0 +1,40 @@
+pa
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83147018
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java
---
@@ -0,0 +1,40 @@
+package org
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/212#discussion_r83146927
--- Diff:
integration/spark/src/main/scala/org/apache/spark/sql/CarbonDatasourceRelation.scala
---
@@ -55,18 +55,11 @@ class CarbonSource ex
Jacky Li created CARBONDATA-314:
---
Summary: Make CarbonContext to use standard Datasource strategy
Key: CARBONDATA-314
URL: https://issues.apache.org/jira/browse/CARBONDATA-314
Project: CarbonData
Jacky Li created CARBONDATA-313:
---
Summary: Update CarbonSource to use CarbonDatasourceHadoopRelation
Key: CARBONDATA-313
URL: https://issues.apache.org/jira/browse/CARBONDATA-313
Project: CarbonData
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83139950
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ privat
Github user Jay357089 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83139600
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ private
GitHub user foryou2030 opened a pull request:
https://github.com/apache/incubator-carbondata/pull/232
[CARBONDATA-310]Fixed compilation failure when using spark 1.6.2
# Why raise this pr?
Compilation failed when using spark 1.6.2, because class not found:
AggregateExpression
GitHub user Zhangshunyu opened a pull request:
https://github.com/apache/incubator-carbondata/pull/231
[CARBONDATA-311]Log the data size of blocklet during data load.
## Why raise this pr?
The blocklet size is an important parameter for analyzing data load and
query, this info s
zhangshunyu created CARBONDATA-311:
--
Summary: Log the data size of blocklet during data load.
Key: CARBONDATA-311
URL: https://issues.apache.org/jira/browse/CARBONDATA-311
Project: CarbonData
Gin-zhj created CARBONDATA-310:
--
Summary: Compilation failed when using spark 1.6.2
Key: CARBONDATA-310
URL: https://issues.apache.org/jira/browse/CARBONDATA-310
Project: CarbonData
Issue Type:
Jacky Li created CARBONDATA-312:
---
Summary: Unify two datasource: CarbonDatasourceHadoopRelation and
CarbonDatasourceRelation
Key: CARBONDATA-312
URL: https://issues.apache.org/jira/browse/CARBONDATA-312
One issue:
I modified the spark.version in pom.xml,using spark1.6.2, then compliation
failed.
Root cause:
There was a "unused import statement" warinng in CarbonOptimizer class before,
we imported AggregationExpression like the following :
import org.apache.spark.sql.catalyst.expressions.aggre
Jacky Li created CARBONDATA-309:
---
Summary: Support two types of ReadSupport in CarbonRecordReader
Key: CARBONDATA-309
URL: https://issues.apache.org/jira/browse/CARBONDATA-309
Project: CarbonData
Jacky Li created CARBONDATA-308:
---
Summary: Support multiple segment in CarbonHadoopFSRDD
Key: CARBONDATA-308
URL: https://issues.apache.org/jira/browse/CARBONDATA-308
Project: CarbonData
Issue
Jacky Li created CARBONDATA-307:
---
Summary: Support full functionality in CarbonInputFormat
Key: CARBONDATA-307
URL: https://issues.apache.org/jira/browse/CARBONDATA-307
Project: CarbonData
Issu
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/223#discussion_r83133189
--- Diff: docs/DML-Operations-on-Carbon.md ---
@@ -104,8 +109,10 @@ Following are the options that can be used in load
data:
Github user Jay357089 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/223#discussion_r83132221
--- Diff: docs/DML-Operations-on-Carbon.md ---
@@ -104,8 +109,10 @@ Following are the options that can be used in load
data:
Github user foryou2030 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/227#discussion_r83132187
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -197,8 +197,9 @@ public
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83130319
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java
---
@@ -0,0 +1,40 @@
+package org
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83130123
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java
---
@@ -0,0 +1,40 @@
+package org
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83129418
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/CarbonDataLoadConfiguration.java
---
@@ -0,0 +1,185 @@
+pack
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83129008
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/CarbonDataLoadConfiguration.java
---
@@ -0,0 +1,185 @@
+pack
On Tue, Oct 11, 2016 at 2:32 AM, Ravindra Pesala
wrote:
> Currently data is loading to carbon in 2 pass/jobs
> 1. Generating global dictionary using spark job.
Do we have local dictionaries? If not, what if the column has many
distinct values - will the big dictionary loaded into memory?
Regard
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/212#discussion_r83123943
--- Diff:
integration/spark/src/main/scala/org/apache/spark/sql/CarbonDatasourceRelation.scala
---
@@ -55,18 +55,11 @@ class CarbonSource exten
Hi Ravi,
1. I agree with Jihong that creation of global dictionary should be
optional, so that it can be disabled to improve the load performance. User
should be made aware that using global dictionary may boost the query
performance.
2. We should have a generic interface to manage global dictiona
Hi Vishal,
In general, it is good idea to have a cache efficient algorithm.
For solution-1 : how do you want to handle variable length columns and
nulls? may be you will have to maintain variable length columns separately
and use offsets ?
For solution 2: code generation may be more efficient
Hi Jacky,
Yes result preparation in exeutor side.
-Regards
Kumar Vishal
On Wed, Oct 12, 2016 at 9:33 PM, Jacky Li wrote:
> Hi Vishal,
>
> Which part of the preparation are you considering? The column stitching in
> the executor side?
>
> Regards,
> Jacky
>
> > 在 2016年10月12日，下午9:24，Kumar Vishal
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83049479
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java
---
@@ -0,0 +1,40 @@
+package org
Hi Vishal,
Which part of the preparation are you considering? The column stitching in the
executor side?
Regards,
Jacky
> 在 2016年10月12日，下午9:24，Kumar Vishal 写道：
>
> Hi All,
> Currently we are preparing the final result row wise, as number of columns
> present in project list(80 columns) is hig
Github user lion-x commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/200#discussion_r83039531
--- Diff:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestDataLoadWithTrimOption.scala
---
@@ -0,0 +1,78 @@
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83032371
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java
---
@@ -0,0 +1,40 @@
+package org.ap
Github user asfgit closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/205
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the f
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83031958
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java
---
@@ -0,0 +1,40 @@
+package org.ap
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83028336
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/CarbonDataLoadConfiguration.java
---
@@ -0,0 +1,185 @@
+package
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83033524
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/iterators/RecordReaderIterator.java
---
@@ -0,0 +1,40 @@
+packa
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83033746
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/iterators/CarbonArrayWritable.java
---
@@ -0,0 +1,51 @@
+packag
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83030022
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/CarbonDataLoadConfiguration.java
---
@@ -0,0 +1,185 @@
+package
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83033298
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/iterators/RecordReaderIterator.java
---
@@ -0,0 +1,40 @@
+packa
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83032703
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java
---
@@ -0,0 +1,40 @@
+package org.ap
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83028164
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ private st
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83027603
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ privat
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/200#discussion_r82523962
--- Diff:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/dataload/TestDataLoadWithTrimOption.scala
---
@@ -0,0 +1,78 @@
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/230#discussion_r83021340
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -252,6 +252,9 @@ private st
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/204#discussion_r83027008
--- Diff:
integration/spark/src/test/scala/org/apache/carbondata/spark/testsuite/deleteTable/TestDeleteTableNewDDL.scala
---
@@ -97,7 +97,7 @@
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/207#discussion_r83026458
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/util/CarbonTableStatusUtil.java
---
@@ -0,0 +1,92 @@
+/*
+ * Licens
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/207#discussion_r83025827
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/util/CarbonTableStatusUtil.java
---
@@ -0,0 +1,92 @@
+/*
+ * Licens
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/223#discussion_r83022245
--- Diff: docs/DML-Operations-on-Carbon.md ---
@@ -104,8 +109,10 @@ Following are the options that can be used in load
data:
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/229#discussion_r83018479
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/newflow/DataLoadProcessorStep.java
---
@@ -0,0 +1,40 @@
+package org.ap
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/218#discussion_r83014871
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/surrogatekeysgenerator/csvbased/BadRecordslogger.java
---
@@ -69,9 +68,13 @
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/218#discussion_r83015590
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/surrogatekeysgenerator/csvbased/CarbonCSVBasedSeqGenStep.java
---
@@ -458,9
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/218#discussion_r83014256
--- Diff:
integration/spark/src/main/java/org/apache/carbondata/spark/load/CarbonLoadModel.java
---
@@ -117,9 +117,9 @@
private String b
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/227#discussion_r83012391
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/store/writer/AbstractFactDataWriter.java
---
@@ -197,8 +197,9 @@ public Abs
Hi All,
Currently we are preparing the final result row wise, as number of columns
present in project list(80 columns) is high mainly measure column or no
dictionary column there are lots of cpu cache invalidation is happening and
this is resulting to slower the query performance.
*I can think of
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/224#discussion_r82983904
--- Diff:
core/src/main/java/org/apache/carbondata/scan/scanner/impl/FilterScanner.java
---
@@ -78,10 +80,11 @@ public FilterScanner(BlockE
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/200#discussion_r82977592
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/csvreaderstep/UnivocityCsvParser.java
---
@@ -102,8 +102,8 @@ public vo
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/200#discussion_r82977743
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/csvreaderstep/UnivocityCsvParser.java
---
@@ -102,8 +102,8 @@ public vo
Hi Jacky,
1. Yes. It is better to keep all sorting logic to one step so other types
of sorts can be implemented easily. I will update the design.
2. EncoderProcessorStep can do dictionary encoding and converting
nodictionary and complex types to byte[] representation.
Here encoding interface
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/200#discussion_r82968804
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/csvreaderstep/UnivocityCsvParser.java
---
@@ -102,8 +102,8 @@ public vo
Github user sujith71955 commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/200#discussion_r82968468
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/csvreaderstep/UnivocityCsvParser.java
---
@@ -102,8 +102,8 @@ public vo
Github user lion-x commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/200#discussion_r82960653
--- Diff:
processing/src/main/java/org/apache/carbondata/processing/csvreaderstep/UnivocityCsvParser.java
---
@@ -102,8 +102,8 @@ public void in
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/224#discussion_r82958230
--- Diff:
core/src/main/java/org/apache/carbondata/scan/processor/AbstractDataBlockIterator.java
---
@@ -127,11 +133,15 @@ protected boolea
62 matches
Mail list logo
