[GitHub] carbondata pull request #2161: [CARBONDATA-2218] AlluxioCarbonFile while try...
Github user chandrasaripaka commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2161#discussion_r241939242
--- Diff:
core/src/main/java/org/apache/carbondata/core/datastore/impl/FileFactory.java
---
@@ -432,7 +437,7 @@ public static long getDirectorySize(String filePath)
throws IOException {
case VIEWFS:
case S3:
Path path = new Path(filePath);
-FileSystem fs = path.getFileSystem(getConfiguration());
+FileSystem fs = path.getFileSystem(configuration);
--- End diff --
Resolved.
---
[GitHub] carbondata pull request #2984: [CARBONDATA-3165]Protection of Bloom Null Exc...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2984#discussion_r241938979
--- Diff:
datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
---
@@ -227,6 +231,12 @@ private String getAncestorTablePath(CarbonTable
currentTable) {
}
}
}
+if (hitBlocklets == null) {
+ LOGGER.warn(String.format("HitBlocklets is empty in bloom filter
prune method. " +
--- End diff --
can you make it non-null, such as making it an empty set
---
[GitHub] carbondata pull request #2890: [CARBONDATA-3002] Fix some spell error
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2890 ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10034/ ---
[GitHub] carbondata issue #2890: [CARBONDATA-3002] Fix some spell error
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/2890 LGTM ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1987/ ---
[GitHub] carbondata pull request #2990: [CARBONDATA-3149]Support alter table column r...
Github user qiuchenjian commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2990#discussion_r241935317
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/hive/CarbonFileMetastore.scala
---
@@ -286,12 +286,16 @@ class CarbonFileMetastore extends CarbonMetaStore {
newTableIdentifier: CarbonTableIdentifier,
oldTableIdentifier: CarbonTableIdentifier,
thriftTableInfo: org.apache.carbondata.format.TableInfo,
- schemaEvolutionEntry: SchemaEvolutionEntry,
+ schemaEvolutionEntry: List[SchemaEvolutionEntry],
--- End diff --
schemaEvolutionEntries or schemaEvolutionEntryList is better
---
[GitHub] carbondata pull request #2990: [CARBONDATA-3149]Support alter table column r...
Github user qiuchenjian commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2990#discussion_r241935024
--- Diff:
integration/spark-common/src/main/scala/org/apache/spark/sql/catalyst/CarbonDDLSqlParser.scala
---
@@ -1464,31 +1464,46 @@ abstract class CarbonDDLSqlParser extends
AbstractCarbonSparkSQLParser {
* @param values
* @return
*/
- def parseDataType(dataType: String, values: Option[List[(Int, Int)]]):
DataTypeInfo = {
+ def parseDataType(
+ dataType: String,
+ values: Option[List[(Int, Int)]],
+ isColumnRename: Boolean): DataTypeInfo = {
+def validateAndGetDecimalDatatype: DataTypeInfo = {
+ var precision: Int = 0
+ var scale: Int = 0
+ if (values.isDefined) {
+precision = values.get(0)._1
+scale = values.get(0)._2
+ } else {
+throw new MalformedCarbonCommandException("Decimal format provided
is invalid")
+ }
+ // precision should be > 0 and <= 38 and scale should be >= 0 and <=
38
+ if (precision < 1 || precision > 38) {
--- End diff --
Magic number 38 should be defined in constant,
---
[GitHub] carbondata pull request #2988: [CARBONDATA-3174] Fix trailing space issue wi...
Github user qiuchenjian commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2988#discussion_r241934868
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestNonTransactionalCarbonTable.scala
---
@@ -2490,6 +2490,47 @@ class TestNonTransactionalCarbonTable extends
QueryTest with BeforeAndAfterAll {
FileUtils.deleteDirectory(new File(writerPath))
}
+ test("check varchar with trailing space") {
+FileUtils.deleteDirectory(new File(writerPath))
+val fields: Array[Field] = new Array[Field](8)
+
+fields(0) = new Field("Event_ID", DataTypes.STRING)
+fields(1) = new Field("Event_Time", DataTypes.TIMESTAMP)
+fields(2) = new Field("subject", DataTypes.VARCHAR)
+fields(3) = new Field("From_Email", DataTypes.STRING)
+fields(4) = new Field("To_Email",
DataTypes.createArrayType(DataTypes.STRING))
+fields(5) = new Field("CC_Email",
DataTypes.createArrayType(DataTypes.STRING))
+fields(6) = new Field("BCC_Email",
DataTypes.createArrayType(DataTypes.STRING))
+fields(7) = new Field("messagebody ", DataTypes.VARCHAR)
+
+var options = Map("bAd_RECords_action" -> "FORCE").asJava
+CarbonProperties.getInstance()
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "EEE, d
MMM HH:mm:ss Z")
+
+val writer: CarbonWriter = CarbonWriter.builder
+ .outputPath(writerPath)
+ .withCsvInput(new
Schema(fields)).writtenBy("TestNonTransactionalCarbonTable").build()
+
+writer
+ .write(Array("aaa",
+"Fri, 4 May 2001 13:51:00 -0700 (PDT)",
+"Re",
+"c...@ws.com",
+"sd#er",
+"sd",
+"sds",
+"ew"))
+writer.close()
+
+sql("drop table if exists test")
+sql(
+ s"""CREATE TABLE test using carbon
options('long_string_columns'='subject,messagebody')
+ |LOCATION '$writerPath'"""
+.stripMargin)
+sql("select * from test").show()
--- End diff --
Why is this test case not checking the correctness of the results? Such as
checkAnswer ï¼ assert and so on
---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1774/ ---
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10033/ ---
[GitHub] carbondata issue #2899: [CARBONDATA-3073][CARBONDATA-3044] Support configure...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2899 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10032/ ---
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1986/ ---
[GitHub] carbondata issue #2899: [CARBONDATA-3073][CARBONDATA-3044] Support configure...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2899 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1985/ ---
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2914 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1773/ ---
[GitHub] carbondata issue #2890: [CARBONDATA-3002] Fix some spell error
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2890 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1984/ ---
[GitHub] carbondata issue #2925: [CARBONDATA-3102] Fix NoClassDefFoundError when use ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2925 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10030/ ---
[GitHub] carbondata issue #2890: [CARBONDATA-3002] Fix some spell error
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2890 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10031/ ---
[GitHub] carbondata issue #2925: [CARBONDATA-3102] Fix NoClassDefFoundError when use ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2925 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1983/ ---
[GitHub] carbondata issue #2899: [CARBONDATA-3073][CARBONDATA-3044] Support configure...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2899 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1772/ ---
[jira] [Resolved] (CARBONDATA-45) Support MAP type
[
https://issues.apache.org/jira/browse/CARBONDATA-45?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manish Gupta resolved CARBONDATA-45.
Resolution: Fixed
Assignee: Manish Gupta (was: Venkata Ramana G)
Fix Version/s: 1.5.2
> Support MAP type
>
>
> Key: CARBONDATA-45
> URL: https://issues.apache.org/jira/browse/CARBONDATA-45
> Project: CarbonData
> Issue Type: New Feature
> Components: core, sql
>Reporter: cen yuhai
>Assignee: Manish Gupta
>Priority: Major
> Fix For: 1.5.2
>
> Attachments: MAP DATA-TYPE SUPPORT.pdf
>
>
> {code:sql}
> >>CREATE TABLE table1 (
> deviceInformationId int,
> channelsId string,
> props map)
> STORED BY 'org.apache.carbondata.format'
> >>insert into table1 select 10,'channel1', map(1,'user1',101, 'root')
> {code}
> format of data to be read from csv, with '$' as level 1 delimiter and map
> keys terminated by '#'
> {code:sql}
> >>load data local inpath '/tmp/data.csv' into table1 options
> >>('COMPLEX_DELIMITER_LEVEL_1'='$', 'COMPLEX_DELIMITER_LEVEL_2'=':',
> >>'COMPLEX_DELIMITER_FOR_KEY'='#')
> 20,channel2,2#user2$100#usercommon
> 30,channel3,3#user3$100#usercommon
> 40,channel4,4#user3$100#usercommon
> >>select channelId, props[100] from table1 where deviceInformationId > 10;
> 20, usercommon
> 30, usercommon
> 40, usercommon
> >>select channelId, props from table1 where props[2] = 'user2';
> 20, {2,'user2', 100, 'usercommon'}
> {code}
> Following cases needs to be handled:
> ||Sub feature||Pending activity||Remarks||
> |Basic Maptype support|Develop| Create table DDL, Load map data from CSV,
> select * from maptable|
> |Maptype lookup in projection and filter|Develop|Projection and filters needs
> execution at spark|
> |NULL values, UDFs, Describe support|Develop||
> |Compaction support | Test + fix | As compaction works at byte level, no
> changes required. Needs to add test-cases|
> |Insert into table| Develop | Source table data containing Map data needs to
> convert from spark datatype to string , as carbon takes string as input row |
> |Support DDL for Map fields Dictionary include and Dictionary Exclude |
> Develop | Also needs to handle CarbonDictionaryDecoder to handle the same. |
> |Support multilevel Map | Develop | currently DDL is validated to allow only
> 2 levels, remove this restriction|
> |Support Map value to be a measure | Develop | Currently array and struct
> supports only dimensions which needs change|
> |Support Alter table to add and remove Map column | Develop | implement DDL
> and requires default value handling |
> |Projections of Map loopup push down to carbon | Develop | this is an
> optimization, when more number of values are present in Map |
> |Filter map loolup push down to carbon | Develop | this is an optimization,
> when more number of values are present in Map |
> |Update Map values | Develop | update map value|
> h4. Design suggestion:
> Map can be represented internally stored as Array>, So that
> conversion of data is required to Map data type while giving to spark. Schema
> will have new column of map type similar to Array.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2802: [HOTFIX] Correct Create Table documentation contents
Github user sraghunandan commented on the issue: https://github.com/apache/carbondata/pull/2802 Duplicate. Hence closing ---
[GitHub] carbondata pull request #2802: [HOTFIX] Correct Create Table documentation c...
Github user sraghunandan closed the pull request at: https://github.com/apache/carbondata/pull/2802 ---
[GitHub] carbondata pull request #2980: [CARBONDATA-3017] Map DDL Support
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2980 ---
[jira] [Resolved] (CARBONDATA-3017) Create DDL Support for Map Type
[ https://issues.apache.org/jira/browse/CARBONDATA-3017?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manish Gupta resolved CARBONDATA-3017. -- Resolution: Fixed Fix Version/s: 1.5.2 > Create DDL Support for Map Type > --- > > Key: CARBONDATA-3017 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3017 > Project: CarbonData > Issue Type: Sub-task >Reporter: MANISH NALLA >Assignee: MANISH NALLA >Priority: Major > Fix For: 1.5.2 > > Time Spent: 13h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2980 LGTM ---
[GitHub] carbondata issue #2914: [CARBONDATA-3093] Provide property builder for carbo...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2914 rebase again ---
[GitHub] carbondata pull request #2899: [CARBONDATA-3073][CARBONDATA-3044] Support co...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2899#discussion_r241813858 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -77,8 +77,9 @@ /** * Sets the output path of the writer builder + * --- End diff -- I think it's better for formatting the code comments/annotation ---
[GitHub] carbondata issue #2890: [CARBONDATA-3002] Fix some spell error
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2890 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1771/ ---
[GitHub] carbondata issue #2925: [CARBONDATA-3102] Fix NoClassDefFoundError when use ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2925 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1770/ ---
[GitHub] carbondata pull request #2899: [CARBONDATA-3073][CARBONDATA-3044] Support co...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2899#discussion_r241811704
--- Diff: store/CSDK/src/CarbonWriter.cpp ---
@@ -98,6 +127,158 @@ void CarbonWriter::withHadoopConf(char *key, char
*value) {
carbonWriterBuilderObject =
jniEnv->CallObjectMethodA(carbonWriterBuilderObject, methodID, args);
}
+void CarbonWriter::withTableProperty(char *key, char *value) {
+if (key == NULL) {
+throw std::runtime_error("key parameter can't be NULL.");
+}
+if (value == NULL) {
+throw std::runtime_error("value parameter can't be NULL.");
+}
+checkBuilder();
+jclass carbonWriterBuilderClass =
jniEnv->GetObjectClass(carbonWriterBuilderObject);
+jmethodID methodID = jniEnv->GetMethodID(carbonWriterBuilderClass,
"withTableProperty",
+
"(Ljava/lang/String;Ljava/lang/String;)Lorg/apache/carbondata/sdk/file/CarbonWriterBuilder;");
+if (methodID == NULL) {
+throw std::runtime_error("Can't find the method in java:
withTableProperty");
+}
+jvalue args[2];
+args[0].l = jniEnv->NewStringUTF(key);
+args[1].l = jniEnv->NewStringUTF(value);
+carbonWriterBuilderObject =
jniEnv->CallObjectMethodA(carbonWriterBuilderObject, methodID, args);
--- End diff --
ok
---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10028/ ---
[GitHub] carbondata pull request #2925: [CARBONDATA-3102] Fix NoClassDefFoundError wh...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2925#discussion_r241802156
--- Diff:
integration/spark2/src/main/scala/org/apache/carbondata/spark/thriftserver/CarbonThriftServer.scala
---
@@ -48,8 +48,13 @@ object CarbonThriftServer {
System.exit(0)
}
+val master = Option(System.getProperty("spark.master"))
--- End diff --
ok, done
---
[GitHub] carbondata pull request #2925: [CARBONDATA-3102] Fix NoClassDefFoundError wh...
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2925#discussion_r241802106 --- Diff: integration/spark2/pom.xml --- @@ -134,6 +134,11 @@ + --- End diff -- ok, done ---
[GitHub] carbondata issue #2973: [WIP][CARBONDATA-3144] CarbonData support spark-2.4....
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2973 I will work on this maybe next week. ---
[GitHub] carbondata pull request #2890: [CARBONDATA-3002] Fix some spell error
Github user xubo245 commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2890#discussion_r241795430 --- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java --- @@ -569,25 +569,25 @@ public static Dictionary getDictionary(AbsoluteTableIdentifier absoluteTableIden nodes.add(relation.getNode()); } -int noofNodes = (-1 == noOfNodesInput) ? nodes.size() : noOfNodesInput; +int noOfNodes = (-1 == noOfNodesInput) ? nodes.size() : noOfNodesInput; --- End diff -- I change noOfNodes to numOfNodes, there are more than 100+ place use numOf__, Do I need optimize noOfXXX to numOfXXX in similar code? ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10029/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1769/ ---
[GitHub] carbondata issue #2949: [CARBONDATA-3118] support parallel block pruning for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2949 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10026/ ---
[GitHub] carbondata issue #2983: [CARBONDATA-3119] Fixed SDK Write for Complex Array ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2983 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10025/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1981/ ---
[GitHub] carbondata pull request #2925: [CARBONDATA-3102] Fix NoClassDefFoundError wh...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2925#discussion_r241774500 --- Diff: integration/spark2/pom.xml --- @@ -134,6 +134,11 @@ + --- End diff -- I see `org.apache.httpcomponents` is already introduced in other modules, please unify the version for this jar in all places ---
[GitHub] carbondata issue #2973: [WIP][CARBONDATA-3144] CarbonData support spark-2.4....
Github user jackylk commented on the issue: https://github.com/apache/carbondata/pull/2973 Please close this if you are not working on it. Thanks ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1768/ ---
[GitHub] carbondata issue #2983: [CARBONDATA-3119] Fixed SDK Write for Complex Array ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2983 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1767/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1766/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1982/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10027/ ---
[GitHub] carbondata issue #2983: [CARBONDATA-3119] Fixed SDK Write for Complex Array ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2983 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1978/ ---
[GitHub] carbondata issue #2966: [CARBONDATA-3162][CARBONDATA-3163][CARBONDATA-3164] ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2966 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10024/ ---
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2980 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1976/ ---
[GitHub] carbondata issue #2949: [CARBONDATA-3118] support parallel block pruning for...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2949 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1765/ ---
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2980 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10023/ ---
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2980 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1764/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1980/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10022/ ---
[GitHub] carbondata issue #2161: [CARBONDATA-2218] AlluxioCarbonFile while trying to ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2161 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1979/ ---
[GitHub] carbondata issue #2966: [CARBONDATA-3162][CARBONDATA-3163][CARBONDATA-3164] ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2966 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1975/ ---
[GitHub] carbondata issue #2983: [CARBONDATA-3119] Fixed SDK Write for Complex Array ...
Github user shivamasn commented on the issue: https://github.com/apache/carbondata/pull/2983 retest this please ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1974/ ---
[GitHub] carbondata issue #2988: [CARBONDATA-3174] Fix trailing space issue with varc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2988 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10020/ ---
[GitHub] carbondata issue #2988: [CARBONDATA-3174] Fix trailing space issue with varc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2988 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1972/ ---
[GitHub] carbondata issue #2966: [CARBONDATA-3162][CARBONDATA-3163][CARBONDATA-3164] ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2966 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1763/ ---
[GitHub] carbondata issue #2990: [CARBONDATA-3149]Support alter table column rename
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2990 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1762/ ---
[GitHub] carbondata issue #2983: [CARBONDATA-3119] Fixed SDK Write for Complex Array ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2983 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1971/ ---
[GitHub] carbondata issue #2989: [CARBONDATA-3175]Fix Testcase failures in complex de...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2989 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1970/ ---
[GitHub] carbondata pull request #2949: [CARBONDATA-3118] support parallel block prun...
Github user ajantha-bhat commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2949#discussion_r241737142 --- Diff: core/src/main/java/org/apache/carbondata/core/datamap/dev/DataMap.java --- @@ -70,4 +70,8 @@ void init(DataMapModel dataMapModel) */ void finish(); + /* + * Returns number of records information that are stored in datamap. + * */ + int getNumberOfEntries(); --- End diff -- done ---
[GitHub] carbondata pull request #2949: [CARBONDATA-3118] support parallel block prun...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2949#discussion_r241737152
--- Diff:
datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMap.java
---
@@ -436,4 +436,9 @@ public String toString() {
public void finish() {
}
+
+ @Override public int getNumberOfEntries() {
--- End diff --
done
---
[GitHub] carbondata issue #2989: [CARBONDATA-3175]Fix Testcase failures in complex de...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2989 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10018/ ---
[GitHub] carbondata pull request #2925: [CARBONDATA-3102] Fix NoClassDefFoundError wh...
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2925#discussion_r241736440 --- Diff: integration/spark2/pom.xml --- @@ -134,6 +134,11 @@ + --- End diff -- I have never encounter this problem before, can you describe the scenario in more detail, in what case NoClassDefFoundError will be thrown? ---
[GitHub] carbondata pull request #2925: [CARBONDATA-3102] Fix NoClassDefFoundError wh...
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2925#discussion_r241736214
--- Diff:
integration/spark2/src/main/scala/org/apache/carbondata/spark/thriftserver/CarbonThriftServer.scala
---
@@ -48,8 +48,13 @@ object CarbonThriftServer {
System.exit(0)
}
+val master = Option(System.getProperty("spark.master"))
--- End diff --
I think better to let user give --master local, user can control this.
---
[GitHub] carbondata issue #2983: [CARBONDATA-3119] Fixed SDK Write for Complex Array ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2983 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10019/ ---
[GitHub] carbondata pull request #2890: [CARBONDATA-3002] Fix some spell error
Github user jackylk commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2890#discussion_r241735498 --- Diff: processing/src/main/java/org/apache/carbondata/processing/util/CarbonLoaderUtil.java --- @@ -569,25 +569,25 @@ public static Dictionary getDictionary(AbsoluteTableIdentifier absoluteTableIden nodes.add(relation.getNode()); } -int noofNodes = (-1 == noOfNodesInput) ? nodes.size() : noOfNodesInput; +int noOfNodes = (-1 == noOfNodesInput) ? nodes.size() : noOfNodesInput; --- End diff -- ```suggestion int numNodes = (-1 == noOfNodesInput) ? nodes.size() : numNodesInput; ``` ---
[jira] [Resolved] (CARBONDATA-3166) Changes in Document and Displaying Carbon Column Compressor used in Describe Formatted Command
[ https://issues.apache.org/jira/browse/CARBONDATA-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xuchuanyin resolved CARBONDATA-3166. Resolution: Fixed Fix Version/s: 1.5.2 > Changes in Document and Displaying Carbon Column Compressor used in Describe > Formatted Command > -- > > Key: CARBONDATA-3166 > URL: https://issues.apache.org/jira/browse/CARBONDATA-3166 > Project: CarbonData > Issue Type: Improvement >Reporter: Shardul Singh >Assignee: Shardul Singh >Priority: Minor > Fix For: 1.5.2 > > Time Spent: 2h 50m > Remaining Estimate: 0h > > Changes in Document and Displaying Carbon Column Compressor used in Describe > Formatted Command -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2899: [CARBONDATA-3073][CARBONDATA-3044] Support co...
Github user KanakaKumar commented on a diff in the pull request: https://github.com/apache/carbondata/pull/2899#discussion_r241731133 --- Diff: store/sdk/src/main/java/org/apache/carbondata/sdk/file/CarbonWriterBuilder.java --- @@ -77,8 +77,9 @@ /** * Sets the output path of the writer builder + * --- End diff -- Can avoid the space changes in multiple places if not intentional. ---
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user manishgupta88 commented on the issue: https://github.com/apache/carbondata/pull/2980 @manishnalla1994 ...Please rebase and remove the commit for PR 2979 as it is already merged ---
[GitHub] carbondata pull request #2986: [CARBONDATA-3166]Updated Document and added C...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2986 ---
[GitHub] carbondata pull request #2899: [CARBONDATA-3073][CARBONDATA-3044] Support co...
Github user KanakaKumar commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2899#discussion_r241730265
--- Diff: store/CSDK/src/CarbonWriter.cpp ---
@@ -98,6 +127,158 @@ void CarbonWriter::withHadoopConf(char *key, char
*value) {
carbonWriterBuilderObject =
jniEnv->CallObjectMethodA(carbonWriterBuilderObject, methodID, args);
}
+void CarbonWriter::withTableProperty(char *key, char *value) {
+if (key == NULL) {
+throw std::runtime_error("key parameter can't be NULL.");
+}
+if (value == NULL) {
+throw std::runtime_error("value parameter can't be NULL.");
+}
+checkBuilder();
+jclass carbonWriterBuilderClass =
jniEnv->GetObjectClass(carbonWriterBuilderObject);
+jmethodID methodID = jniEnv->GetMethodID(carbonWriterBuilderClass,
"withTableProperty",
+
"(Ljava/lang/String;Ljava/lang/String;)Lorg/apache/carbondata/sdk/file/CarbonWriterBuilder;");
+if (methodID == NULL) {
+throw std::runtime_error("Can't find the method in java:
withTableProperty");
+}
+jvalue args[2];
+args[0].l = jniEnv->NewStringUTF(key);
+args[1].l = jniEnv->NewStringUTF(value);
+carbonWriterBuilderObject =
jniEnv->CallObjectMethodA(carbonWriterBuilderObject, methodID, args);
--- End diff --
Java code may throw IllegalArgumentException for unsuported property. We
should add exception check after the method call
---
[GitHub] carbondata issue #2986: [CARBONDATA-3166]Updated Document and added Column C...
Github user xuchuanyin commented on the issue: https://github.com/apache/carbondata/pull/2986 LGTM ---
[GitHub] carbondata issue #2988: [CARBONDATA-3174] Fix trailing space issue with varc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2988 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1761/ ---
[GitHub] carbondata pull request #2990: [CARBONDATA-3149]Support alter table column r...
GitHub user akashrn5 opened a pull request: https://github.com/apache/carbondata/pull/2990 [CARBONDATA-3149]Support alter table column rename ### Why this PR? This PR is to support column rename feature in carbondata. Carbon already supports datatype change, alter table add column and drop column. This PR uses same DDL as datatype change and supports the column rename. Any column canbe Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/akashrn5/incubator-carbondata alter_rename Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2990.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2990 commit e84b09dd244803746fcacd3e9c6eda4105dd7bef Author: akashrn5 Date: 2018-12-14T11:20:09Z Support alter table column rename ---
[GitHub] carbondata issue #2988: [CARBONDATA-3174] Fix trailing space issue with varc...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2988 add to whitelist ---
[GitHub] carbondata issue #2988: [CARBONDATA-3174] Fix trailing space issue with varc...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2988 retest this please ---
[GitHub] carbondata issue #2983: [CARBONDATA-3119] Fixed SDK Write for Complex Array ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2983 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1760/ ---
[GitHub] carbondata issue #2989: [CARBONDATA-3175]Fix Testcase failures in complex de...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2989 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1759/ ---
[GitHub] carbondata issue #2983: [CARBONDATA-3119] Fixed SDK Write for Complex Array ...
Github user brijoobopanna commented on the issue: https://github.com/apache/carbondata/pull/2983 retest this please ---
[GitHub] carbondata pull request #2989: [CARBONDATA-3175]Fix Testcase failures in com...
GitHub user Indhumathi27 opened a pull request: https://github.com/apache/carbondata/pull/2989 [CARBONDATA-3175]Fix Testcase failures in complex deleimiters Fix Testcase failures in complex deleimiters: Changed the complex delimiter in testcases - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [ ] Testing done - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Indhumathi27/carbondata fixsdv Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2989.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2989 commit ccd2347f6413f0d92e326b4c1013b415eabafbc1 Author: Indhumathi27 Date: 2018-12-14T10:56:00Z [CARBONDATA-3175]Fix Testcase failures in complex deleimiters ---
[jira] [Created] (CARBONDATA-3175) Fix Testcase failures in complex deleimiters
Indhumathi Muthumurugesh created CARBONDATA-3175: Summary: Fix Testcase failures in complex deleimiters Key: CARBONDATA-3175 URL: https://issues.apache.org/jira/browse/CARBONDATA-3175 Project: CarbonData Issue Type: Test Reporter: Indhumathi Muthumurugesh Assignee: Indhumathi Muthumurugesh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2980 Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1969/ ---
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2980 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10017/ ---
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2980 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1758/ ---
[GitHub] carbondata issue #2988: [CARBONDATA-3174] Fix trailing space issue with varc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2988 Can one of the admins verify this patch? ---
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2980 Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10016/ ---
[GitHub] carbondata pull request #2988: [CARBONDATA-3174] Fix trailing space issue wi...
GitHub user Shubh18s opened a pull request: https://github.com/apache/carbondata/pull/2988 [CARBONDATA-3174] Fix trailing space issue with varchar column for SDK What was the issue? After doing SDK Write, Select * was failing for 'long_string_columns' with trailing space. What has been changed? Removed the trailing space in ColumnName. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Added a test case. - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Shubh18s/carbondata master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2988.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2988 commit 7a5d26e6802fef2e40f987e49298a25b78183d20 Author: Shubh18s Date: 2018-12-14T09:11:06Z varchar column trailing space issue fixed ---
[jira] [Created] (CARBONDATA-3174) Fix trailing space issue with varchar column for SDK
Shubhdeep Singh created CARBONDATA-3174: --- Summary: Fix trailing space issue with varchar column for SDK Key: CARBONDATA-3174 URL: https://issues.apache.org/jira/browse/CARBONDATA-3174 Project: CarbonData Issue Type: Bug Reporter: Shubhdeep Singh -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2890: [CARBONDATA-3002] Fix some spell error
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2890 @brijoobopanna @chenliang613 @jackylk CI pass. please check again. ---
[GitHub] carbondata issue #2899: [CARBONDATA-3073][CARBONDATA-3044] Support configure...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2899 @ajantha-bhat @KanakaKumar CI pass, Please review again ---
[GitHub] carbondata issue #2890: [CARBONDATA-3002] Fix some spell error
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2890 Build Success with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10015/ ---
[GitHub] carbondata issue #2980: [CARBONDATA-3017] Map DDL Support
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2980 Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1968/ ---
[GitHub] carbondata issue #2925: [CARBONDATA-3102] Fix NoClassDefFoundError when use ...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2925 @kunal642 In cluster, it also has this problem. ---
