[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3673/



---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2650/



---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3895/



---

[jira] [Commented] (CARBONDATA-2204) Access tablestatus file too many times during query

[xuchuanyin (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376444#comment-16376444
 ] 

xuchuanyin commented on CARBONDATA-2204:


I deleted the unrelated logs and the remaining are shown as below:

 


18/02/25 21:35:38 ERROR TestLoadDataGeneral: 
ScalaTest-run-running-TestLoadDataGeneral XU begin to query data
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/

18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/
18/02/25 21:35:38 ERROR ThriftReader: ScalaTest-run-running-TestLoadDataGeneral 
XU Open thrift reader for file: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Fact/Part0/Segment_0/1519623338082.carbonindexmerge
18/02/25 21:35:38 ERROR ThriftReader: ScalaTest-run-running-TestLoadDataGeneral 
XU Open thrift reader for file: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Fact/Part0/Segment_0/1519623338082.carbonindexmerge

18/02/25 21:35:38 ERROR TestLoadDataGeneral: 
ScalaTest-run-running-TestLoadDataGeneral XU begin to query2 data
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/

18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/tablestatus
18/02/25 21:35:38 ERROR AtomicFileOperationsImpl: 
ScalaTest-run-running-TestLoadDataGeneral XU Open atomic file for read: 
/home/xu/ws/carbondata/integration/spark-common/target/warehouse/loadtest/Metadata/

> Access tablestatus file too many times during query
> ---
>
> Key: CARBONDATA-2204
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2204
> Project: CarbonData
>  Issue Type: Improvement
>  

[GitHub] carbondata issue #1942: [CARBONDATA-2136] Fixed bug related to data load for...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1942
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3671/



---

[jira] [Commented] (CARBONDATA-2204) Access tablestatus file too many times during query

[Jacky Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376439#comment-16376439
 ] 

Jacky Li commented on CARBONDATA-2204:
--

Can you send the log of 7 access here [~xuchuanyin]

> Access tablestatus file too many times during query
> ---
>
> Key: CARBONDATA-2204
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2204
> Project: CarbonData
>  Issue Type: Improvement
>  Components: data-query
>Affects Versions: 1.3.0
>Reporter: xuchuanyin
>Priority: Major
>
> * Problems
> Currently in carbondata, a single query will access tablestatus file 7 times, 
> which will definitely slow down the query performance especially when this 
> file is in remote cluster since reading this file is purely client side 
> operation.
>  
>  *  Steps to reproduce
> 1. Add logger in `AtomicFileOperationsImpl.openForRead` and printout the file 
> name to read.
> 2. Run a query on carbondata table. Here I ran 
> `TestLoadDataGeneral.test("test data loading CSV file without extension 
> name")`.
> 3. Observe the output log and search the keyword 'tablestatus'.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (CARBONDATA-2204) Access tablestatus file too many times during query

[xuchuanyin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

xuchuanyin reassigned CARBONDATA-2204:
--

   Assignee: (was: xuchuanyin)
Description: 
* Problems

Currently in carbondata, a single query will access tablestatus file 7 times, 
which will definitely slow down the query performance especially when this file 
is in remote cluster since reading this file is purely client side operation.

 
 *  Steps to reproduce

1. Add logger in `AtomicFileOperationsImpl.openForRead` and printout the file 
name to read.

2. Run a query on carbondata table. Here I ran `TestLoadDataGeneral.test("test 
data loading CSV file without extension name")`.

3. Observe the output log and search the keyword 'tablestatus'.  

 

  was:
# Problems

Currently in carbondata, a single query will access tablestatus file 7 times, 
which will definitely slow down the query performance especially when this file 
is in remote cluster since reading this file is purely client side operation.

 

# Steps to reproduce

1. Add logger in `AtomicFileOperationsImpl.openForRead` and printout the file 
name to read.

2. Run a query on carbondata table. Here I ran `TestLoadDataGeneral.test("test 
data loading CSV file without extension name")`.

3. Observe the output log and search the keyword 'tablestatus'.  

 


> Access tablestatus file too many times during query
> ---
>
> Key: CARBONDATA-2204
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2204
> Project: CarbonData
>  Issue Type: Improvement
>  Components: data-query
>Affects Versions: 1.3.0
>Reporter: xuchuanyin
>Priority: Major
>
> * Problems
> Currently in carbondata, a single query will access tablestatus file 7 times, 
> which will definitely slow down the query performance especially when this 
> file is in remote cluster since reading this file is purely client side 
> operation.
>  
>  *  Steps to reproduce
> 1. Add logger in `AtomicFileOperationsImpl.openForRead` and printout the file 
> name to read.
> 2. Run a query on carbondata table. Here I ran 
> `TestLoadDataGeneral.test("test data loading CSV file without extension 
> name")`.
> 3. Observe the output log and search the keyword 'tablestatus'.  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CARBONDATA-2167) CarbonDataLoadingException:There is an unexpected error:null

[shining (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16376427#comment-16376427
 ] 

shining commented on CARBONDATA-2167:
-

Anyone know how to resolve it yet? Thanks for look!

[~ravi.pesala]

 

 

> CarbonDataLoadingException:There is an unexpected error:null
> 
>
> Key: CARBONDATA-2167
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2167
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, sql
>Affects Versions: 1.2.0
> Environment: Hadoop2.7
> spark 2.1.2
> carbondata 1.2
>Reporter: shining
>Priority: Major
> Attachments: 00.png, 11.png
>
>
> When insert into cabondta table from a textfile table, I encounter an problem:
> !11.png!
>  
> Here is the table create sql and insert sql:
> !00.png!
> h3. {color:#d04437} The column "content" may be too long, Is this effect the 
> problem??{color}
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1942: [CARBONDATA-2136] Fixed bug related to data load for...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1942
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3894/



---

[jira] [Updated] (CARBONDATA-2198) Streaming data to a table with bad_records_action as IGNORE throws ClassCastException

[Geetika Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geetika Gupta updated CARBONDATA-2198:
--
Description: 
Steps to reproduce:

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * [http://www.apache.org/licenses/LICENSE-2.0]
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.carbondata.examples

import java.io.\{File, PrintWriter}
 import java.net.ServerSocket

import org.apache.spark.sql.\{CarbonEnv, SparkSession}
 import org.apache.spark.sql.streaming.\{ProcessingTime, StreamingQuery}

import org.apache.carbondata.core.constants.CarbonCommonConstants
 import org.apache.carbondata.core.util.CarbonProperties
 import org.apache.carbondata.core.util.path.\{CarbonStorePath, CarbonTablePath}

// scalastyle:off println
 object CarbonStructuredStreamingExample {
 def main(args: Array[String]) {

// setup paths
 val rootPath = new File(this.getClass.getResource("/").getPath
 + "../../../..").getCanonicalPath
 val storeLocation = s"$rootPath/examples/spark2/target/store"
 val warehouse = s"$rootPath/examples/spark2/target/warehouse"
 val metastoredb = s"$rootPath/examples/spark2/target"
 val streamTableName = s"stream_table"

CarbonProperties.getInstance()
 .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "/MM/dd")

import org.apache.spark.sql.CarbonSession._
 val spark = SparkSession
 .builder()
 .master("local")
 .appName("CarbonStructuredStreamingExample")
 .config("spark.sql.warehouse.dir", warehouse)
 .getOrCreateCarbonSession(storeLocation, metastoredb)

spark.sparkContext.setLogLevel("ERROR")

val requireCreateTable = true
 val useComplexDataType = false

if (requireCreateTable) {
 // drop table if exists previously
 spark.sql(s"DROP TABLE IF EXISTS ${ streamTableName }")
 // Create target carbon table and populate with initial data
 if (useComplexDataType) {
 spark.sql(
 s"""
|CREATE TABLE ${ streamTableName }(|
|id INT,|
|name STRING,|
|city STRING,|
|salary FLOAT,|
|file struct|
|)|
|STORED BY 'carbondata'|
|TBLPROPERTIES(|
|'streaming'='true', 'sort_columns'='name', 'dictionary_include'='city')|
|""".stripMargin)
 } else {
 spark.sql(
 s"""|
|CREATE TABLE ${ streamTableName }(|
|id INT,|
|name STRING,|
|city STRING,|
|salary FLOAT|
|)|
|STORED BY 'carbondata'|
|TBLPROPERTIES(|
|'streaming'='true', 'sort_columns'='name')|
|""".stripMargin)
 }|

val carbonTable = CarbonEnv.getCarbonTable(Some("default"), 
streamTableName)(spark)
 val tablePath = 
CarbonStorePath.getCarbonTablePath(carbonTable.getAbsoluteTableIdentifier)

// streaming ingest
 val serverSocket = new ServerSocket(7071)
 val thread1 = startStreaming(spark, tablePath)
 val thread2 = writeSocket(serverSocket)

System.out.println("type enter to interrupt streaming")
 System.in.read()
 thread1.interrupt()
 thread2.interrupt()
 serverSocket.close()
 }

spark.sql(s"select * from $streamTableName").show
 spark.stop()
 System.out.println("streaming finished")
 }

def showTableCount(spark: SparkSession, tableName: String): Thread = {
 val thread = new Thread() {
 override def run(): Unit = {
 for (_ <- 0 to 1000)

{ spark.sql(s"select count(*) from $tableName").show(truncate = false) 
Thread.sleep(1000 * 3) }

}
 }
 thread.start()
 thread
 }

def startStreaming(spark: SparkSession, tablePath: CarbonTablePath): Thread = {
 val thread = new Thread() {
 override def run(): Unit = {
 var qry: StreamingQuery = null
 try

{ val readSocketDF = spark.readStream .format("socket") .option("host", 
"localhost") .option("port", 7071) .load() // Write data from socket stream to 
carbondata file qry = readSocketDF.writeStream .format("carbondata") 
.trigger(ProcessingTime("5 seconds")) .option("checkpointLocation", 
tablePath.getStreamingCheckpointDir) .option("bad_records_action", "ignore") 
.option("dbName", "default") .option("tableName", "stream_table") .start() 
qry.awaitTermination() }

catch

{ case ex: Exception => ex.printStackTrace() println("Done reading and writing 
streaming data") }

finally

{ qry.stop() }

}
 }
 thread.start()
 thread
 }

def writeSocket(serverSocket: ServerSocket): Thread = {
 val thread = new Thread() {
 override def run(): Unit = {
 // wait for client to connection request and accept
 val clientSocket = 

[jira] [Updated] (CARBONDATA-2198) Streaming data to a table with bad_records_action as IGNORE throws ClassCastException

[Geetika Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Geetika Gupta updated CARBONDATA-2198:
--
Description: 
Steps to reproduce:

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements. See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License. You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */

package org.apache.carbondata.examples

import java.io.\{File, PrintWriter}
import java.net.ServerSocket

import org.apache.spark.sql.\{CarbonEnv, SparkSession}
import org.apache.spark.sql.streaming.\{ProcessingTime, StreamingQuery}

import org.apache.carbondata.core.constants.CarbonCommonConstants
import org.apache.carbondata.core.util.CarbonProperties
import org.apache.carbondata.core.util.path.\{CarbonStorePath, CarbonTablePath}

// scalastyle:off println
object CarbonStructuredStreamingExample {
 def main(args: Array[String]) {

 // setup paths
 val rootPath = new File(this.getClass.getResource("/").getPath
 + "../../../..").getCanonicalPath
 val storeLocation = s"$rootPath/examples/spark2/target/store"
 val warehouse = s"$rootPath/examples/spark2/target/warehouse"
 val metastoredb = s"$rootPath/examples/spark2/target"
 val streamTableName = s"stream_table"

 CarbonProperties.getInstance()
 .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "/MM/dd")

 import org.apache.spark.sql.CarbonSession._
 val spark = SparkSession
 .builder()
 .master("local")
 .appName("CarbonStructuredStreamingExample")
 .config("spark.sql.warehouse.dir", warehouse)
 .getOrCreateCarbonSession(storeLocation, metastoredb)

 spark.sparkContext.setLogLevel("ERROR")

 val requireCreateTable = true
 val useComplexDataType = false

 if (requireCreateTable) {
 // drop table if exists previously
 spark.sql(s"DROP TABLE IF EXISTS ${ streamTableName }")
 // Create target carbon table and populate with initial data
 if (useComplexDataType) {
 spark.sql(
 s"""
 | CREATE TABLE ${ streamTableName }(
 | id INT,
 | name STRING,
 | city STRING,
 | salary FLOAT,
 | file struct
 | )
 | STORED BY 'carbondata'
 | TBLPROPERTIES(
 | 'streaming'='true', 'sort_columns'='name', 'dictionary_include'='city')
 | """.stripMargin)
 } else {
 spark.sql(
 s"""
 | CREATE TABLE ${ streamTableName }(
 | id INT,
 | name STRING,
 | city STRING,
 | salary FLOAT
 | )
 | STORED BY 'carbondata'
 | TBLPROPERTIES(
 | 'streaming'='true', 'sort_columns'='name')
 | """.stripMargin)
 }

 val carbonTable = CarbonEnv.getCarbonTable(Some("default"), 
streamTableName)(spark)
 val tablePath = 
CarbonStorePath.getCarbonTablePath(carbonTable.getAbsoluteTableIdentifier)
 
 // streaming ingest
 val serverSocket = new ServerSocket(7071)
 val thread1 = startStreaming(spark, tablePath)
 val thread2 = writeSocket(serverSocket)

 System.out.println("type enter to interrupt streaming")
 System.in.read()
 thread1.interrupt()
 thread2.interrupt()
 serverSocket.close()
 }
 
 spark.sql(s"select * from $streamTableName").show
 spark.stop()
 System.out.println("streaming finished")
 }

 def showTableCount(spark: SparkSession, tableName: String): Thread = {
 val thread = new Thread() {
 override def run(): Unit = {
 for (_ <- 0 to 1000) {
 spark.sql(s"select count(*) from $tableName").show(truncate = false)
 Thread.sleep(1000 * 3)
 }
 }
 }
 thread.start()
 thread
 }

 def startStreaming(spark: SparkSession, tablePath: CarbonTablePath): Thread = {
 val thread = new Thread() {
 override def run(): Unit = {
 var qry: StreamingQuery = null
 try {
 val readSocketDF = spark.readStream
 .format("socket")
 .option("host", "localhost")
 .option("port", 7071)
 .load()

 // Write data from socket stream to carbondata file
 qry = readSocketDF.writeStream
 .format("carbondata")
 .trigger(ProcessingTime("5 seconds"))
 .option("checkpointLocation", tablePath.getStreamingCheckpointDir)
 .option("bad_records_action", "ignore")
 .option("dbName", "default")
 .option("tableName", "stream_table")
 .start()

 qry.awaitTermination()
 } catch {
 case ex: Exception =>
 ex.printStackTrace()
 println("Done reading and writing streaming data")
 } finally {
 qry.stop()
 }
 }
 }
 thread.start()
 thread
 }

 def writeSocket(serverSocket: ServerSocket): Thread = {
 val thread = new Thread() {
 override def run(): Unit = {
 // wait for client to connection request 

[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1959
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3670/



---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3893/



---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2647/



---

[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1959
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2648/



---

[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1959
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3892/



---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3669/



---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2646/



---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3891/



---

[jira] [Created] (CARBONDATA-2203) record detailed metrics information during carbon processing

[xuchuanyin (JIRA)]

xuchuanyin created CARBONDATA-2203:
--

 Summary: record detailed metrics information during carbon 
processing
 Key: CARBONDATA-2203
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2203
 Project: CarbonData
  Issue Type: Improvement
Reporter: xuchuanyin
Assignee: xuchuanyin


We need to record more detailed information during carbondata processing (query 
& load).

These information can be used:
 # for figuring out the possible performance problems in carbondata
 # for tuning
 # as input for some adaptive strategy

Here are some examples:

For data query:
 # Time costs for sql parse/optimize/plan
 # Time to load metadata
 # Time to schedule
 # Time to execute
 # ...

For data loading:
 # Time to schedule
 # Time to execute
 # Disk spill amount
 # ...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3668/



---

[GitHub] carbondata pull request #1997: [CARBONDATA-2201] NPE fixed while triggering ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1997


---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3667/



---

[GitHub] carbondata issue #1994: [CARBONDATA-1114][Tests] Fix bugs in tests in window...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1994
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3889/



---

[GitHub] carbondata issue #1994: [CARBONDATA-1114][Tests] Fix bugs in tests in window...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1994
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2644/



---

[GitHub] carbondata issue #1997: [CARBONDATA-2201] NPE fixed while triggering the Loa...

[QiangCai]

Github user QiangCai commented on the issue:

https://github.com/apache/carbondata/pull/1997
  
LGTM


---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2645/



---

[GitHub] carbondata issue #1998: [HOTFIX] Add dava doc for datamap interface

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1998
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3890/



---

[GitHub] carbondata pull request #1998: [HOTFIX] Add dava doc for datamap interface

[jackylk]

GitHub user jackylk opened a pull request:

https://github.com/apache/carbondata/pull/1998

[HOTFIX] Add dava doc for datamap interface

1. Rename some of the datamap interface
2. Add more java doc for all public class of datamap interface

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jackylk/incubator-carbondata datamap-rename

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1998.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1998


commit 28fe6491d26f613cc78d39c6ce588bb6ccd932dc
Author: Jacky Li 
Date:   2018-02-26T02:04:51Z

rename interface




---

[jira] [Created] (CARBONDATA-2202) Introduce local dictionary encoding for dimensions

[xuchuanyin (JIRA)]

xuchuanyin created CARBONDATA-2202:
--

 Summary: Introduce local dictionary encoding for dimensions
 Key: CARBONDATA-2202
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2202
 Project: CarbonData
  Issue Type: Improvement
Reporter: xuchuanyin
Assignee: xuchuanyin


Currently Carbondata will generate global dictionary for columns with 
'dictionary_include' attribute.

A dimension column without that attribute will only be stored after some simple 
compression. These columns can also be dictionary encoded in file level (called 
‘local dictionary’) to reduce data size.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #1390: [CARBONDATA-1360] fix for update table failures for ...

[zzcclp]

Github user zzcclp commented on the issue:

https://github.com/apache/carbondata/pull/1390
  
@ashwini-krishnakumar 
[PR#1962](https://github.com/apache/carbondata/pull/1962) has been merged, I 
think it don't need to change  CarbonScalaUtil.scala and just remain the test 
case.


---

[GitHub] carbondata issue #1994: [CARBONDATA-1114][Tests] Fix bugs in tests in window...

[xuchuanyin]

Github user xuchuanyin commented on the issue:

https://github.com/apache/carbondata/pull/1994
  
retest this please


---

[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1825
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3666/



---

[GitHub] carbondata issue #1997: [CARBONDATA-2201] NPE fixed while triggering the Loa...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1997
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3665/



---

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1510
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3664/



---

[GitHub] carbondata issue #1988: [CARBONDATA-2193][WIP] Support register analyzer and...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1988
  
SDV Build Fail , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3663/



---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3662/



---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3661/



---

[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1959
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3660/



---

[GitHub] carbondata issue #1997: [CARBONDATA-2201] NPE fixed while triggering the Loa...

[sraghunandan]

Github user sraghunandan commented on the issue:

https://github.com/apache/carbondata/pull/1997
  
LGTM


---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3659/



---

[jira] [Resolved] (CARBONDATA-2091) Enhance data loading performance by specifying range bounds for sort columns

[Jacky Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacky Li resolved CARBONDATA-2091.
--
   Resolution: Fixed
Fix Version/s: 1.3.2

> Enhance data loading performance by specifying range bounds for sort columns
> 
>
> Key: CARBONDATA-2091
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2091
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: xuchuanyin
>Assignee: xuchuanyin
>Priority: Major
> Fix For: 1.3.2
>
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> Currently in carbondata, data loading using node_sort (also known as 
> local_sort) has the following procedures:
>  # convert the input data in batch. (*Convert*)
>  # sort the batch and write to the sort temp files. (*TempSort*)
>  # combine the sort temp files and do merge sort to get a bigger ordered sort 
> temp file. (*MergeSort*)
>  # combine all the sort temp files and do a final sort, its results will feed 
> the next procedure. (*FinalSort*)
>  # get rows in order and convert rows to carbondata columnar format pages. 
> (*produce*)
>  # Write bundles of pages to files and write the corresponding index file. 
> (*consume*)
> The Step1~Step3 are done concurrently using multi-thread. The Step4 is done 
> using only one thread. The Step5 is done using multi-thread. So the Step4 is 
> the bottleneck among all the procedures. When observing the data loading 
> performance, we can see that the CPU usage after Step3 is low.
>  
> We can enhance the data loading performance by parallelizing Step4.
>  
> User can specify range bounds for the sort columns and carbondata internally 
> distributes the records to different ranges and process the data concurrently 
> in different ranges.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (CARBONDATA-2200) Like operation on streaming table throwing Exception

[Jacky Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jacky Li resolved CARBONDATA-2200.
--
   Resolution: Fixed
Fix Version/s: 1.3.1

> Like operation on streaming table throwing Exception
> 
>
> Key: CARBONDATA-2200
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2200
> Project: CarbonData
>  Issue Type: Bug
>Reporter: QiangCai
>Priority: Major
> Fix For: 1.3.1
>
>  Time Spent: 3h 10m
>  Remaining Estimate: 0h
>
> In TestStreamingTableOperation.scala file,
> for test case "query on stream table with dictionary, sort_columns"
> the following SQL is not working:
> select * from stream_table_filter where name like '%me%'
> error log:
> java.io.IOException: Failed to filter row in vector reader
>  at 
> org.apache.carbondata.hadoop.streaming.CarbonStreamRecordReader.scanBlockletAndFillVector(CarbonStreamRecordReader.java:449)
>  at 
> org.apache.carbondata.hadoop.streaming.CarbonStreamRecordReader.nextColumnarBatch(CarbonStreamRecordReader.java:328)
>  at 
> org.apache.carbondata.hadoop.streaming.CarbonStreamRecordReader.nextKeyValue(CarbonStreamRecordReader.java:309)
>  at 
> org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.hasNext(CarbonScanRDD.scala:390)
>  at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown
>  Source)
>  at 
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
>  Source)
>  at 
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
>  at 
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
>  at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231)
>  at 
> org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225)
>  at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826)
>  at 
> org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826)
>  at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
>  at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
>  at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: 
> org.apache.carbondata.core.scan.expression.exception.FilterUnsupportedException:
>  [B cannot be cast to org.apache.spark.unsafe.types.UTF8String
>  at 
> org.apache.spark.sql.SparkUnknownExpression.evaluate(SparkUnknownExpression.scala:50)
>  at 
> org.apache.carbondata.core.scan.filter.executer.RowLevelFilterExecuterImpl.applyFilter(RowLevelFilterExecuterImpl.java:279)
>  at 
> org.apache.carbondata.hadoop.streaming.CarbonStreamRecordReader.scanBlockletAndFillVector(CarbonStreamRecordReader.java:444)
>  ... 20 more
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1996: [CARBONDATA-2200] Fix bug of LIKE operation o...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1996


---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
LGTM


---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2643/



---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3888/



---

[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1825
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3887/



---

[GitHub] carbondata issue #1825: [CARBONDATA-2032][DataLoad] directly write carbon da...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1825
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2642/



---

[GitHub] carbondata issue #1997: [CARBONDATA-2201] NPE fixed while triggering the Loa...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1997
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2641/



---

[GitHub] carbondata issue #1997: [CARBONDATA-2201] NPE fixed while triggering the Loa...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1997
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3886/



---

[GitHub] carbondata issue #1997: [CARBONDATA-2201] NPE fixed while triggering the Loa...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1997
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3658/



---

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1510
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3885/



---

[GitHub] carbondata issue #1510: [CARBONDATA-1543] Supported DataMap chooser and expr...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1510
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2640/



---

[GitHub] carbondata issue #1988: [CARBONDATA-2193][WIP] Support register analyzer and...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1988
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2639/



---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[QiangCai]

Github user QiangCai commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
retest this please


---

[GitHub] carbondata issue #1988: [CARBONDATA-2193][WIP] Support register analyzer and...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1988
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3884/



---

[GitHub] carbondata pull request #1953: [CARBONDATA-2091][DataLoad] Support specifyin...

[xuchuanyin]

Github user xuchuanyin closed the pull request at:

https://github.com/apache/carbondata/pull/1953


---

[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1959
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3881/



---

[GitHub] carbondata issue #1959: [CARBONDATA-2148][CARBONDATA-2147] Add new Row parse...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1959
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2636/



---

[jira] [Updated] (CARBONDATA-2201) firing the LoadTablePreExecutionEvent before streaming causes NPE

[Rahul Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rahul Kumar updated CARBONDATA-2201:

Summary: firing the LoadTablePreExecutionEvent before streaming causes NPE  
(was: firing the event before streaming causes NPE)

> firing the LoadTablePreExecutionEvent before streaming causes NPE
> -
>
> Key: CARBONDATA-2201
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2201
> Project: CarbonData
>  Issue Type: Improvement
>Reporter: Rahul Kumar
>Assignee: Rahul Kumar
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1997: [CARBONDATA-2201] NPE fixed while triggering ...

[rahulforallp]

Github user rahulforallp commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1997#discussion_r170452321
  
--- Diff: 
streaming/src/main/scala/org/apache/carbondata/streaming/StreamSinkFactory.scala
 ---
@@ -67,7 +67,7 @@ object StreamSinkFactory {
   carbonLoadModel.getFactFilePath,
   false,
   parameters.asJava,
-  null,
+  parameters.asJava,
--- End diff --

while triggering the LoadTablePreExecutionEvent we reuire options provided 
by user and the finalOptions . In case of streaming both are same 


---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
Build Failed with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2638/



---

[GitHub] carbondata issue #1953: [CARBONDATA-2091][DataLoad] Support specifying sort ...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/1953
  
merged into carbonstore branch


---

[GitHub] carbondata issue #1953: [CARBONDATA-2091][DataLoad] Support specifying sort ...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/1953
  
LGTM


---

[GitHub] carbondata issue #1997: [CARBONDATA-2201] NPE fixed for Streaming event

[sraghunandan]

Github user sraghunandan commented on the issue:

https://github.com/apache/carbondata/pull/1997
  
update PR description


---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3880/



---

[GitHub] carbondata pull request #1996: [CARBONDATA-2200] Fix bug of LIKE operation o...

[sraghunandan]

Github user sraghunandan commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1996#discussion_r170451788
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
 ---
@@ -276,12 +277,82 @@ public BitSetGroup applyFilter(BlocksChunkHolder 
blockChunkHolder, boolean useBi
   public boolean applyFilter(RowIntf value, int dimOrdinalMax)
   throws FilterUnsupportedException, IOException {
 try {
-  return exp.evaluate(value).getBoolean();
+  Boolean result = exp.evaluate(createRow(value, 
dimOrdinalMax)).getBoolean();
--- End diff --

pls explain why createRow needs to be called.what was the problem without 
this


---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2635/



---

[GitHub] carbondata pull request #1996: [CARBONDATA-2200] Fix bug of LIKE operation o...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1996#discussion_r170451602
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
 ---
@@ -276,12 +277,82 @@ public BitSetGroup applyFilter(BlocksChunkHolder 
blockChunkHolder, boolean useBi
   public boolean applyFilter(RowIntf value, int dimOrdinalMax)
   throws FilterUnsupportedException, IOException {
 try {
-  return exp.evaluate(value).getBoolean();
+  Boolean result = exp.evaluate(createRow(value, 
dimOrdinalMax)).getBoolean();
+  return result == null ? false : result;
 } catch (FilterIllegalMemberException e) {
   throw new FilterUnsupportedException(e);
 }
   }
 
+  /**
+   * create row for row filter to evaluate expression
+   */
+  private RowIntf createRow(RowIntf value, int dimOrdinalMax) throws 
IOException {
+Object[] record = new Object[value.size()];
+String memberString;
+for (int i = 0; i < dimColEvaluatorInfoList.size(); i++) {
+  DimColumnResolvedFilterInfo dimColumnEvaluatorInfo = 
dimColEvaluatorInfoList.get(i);
+  int index = dimColumnEvaluatorInfo.getDimension().getOrdinal();
+  // if filter dimension is not present in the current add its default 
value
+  if (!isDimensionPresentInCurrentBlock[i]) {
+// fill default value here
+record[index] = getDimensionDefaultValue(dimColumnEvaluatorInfo);
+continue;
+  }
+  if 
(!dimColumnEvaluatorInfo.getDimension().getDataType().isComplexType()) {
+if (!dimColumnEvaluatorInfo.isDimensionExistsInCurrentSilce()) {
+  record[index] = 
dimColumnEvaluatorInfo.getDimension().getDefaultValue();
+}
+byte[] memberBytes = (byte[]) value.getVal(index);
+if 
(!dimColumnEvaluatorInfo.getDimension().hasEncoding(Encoding.DICTIONARY)) {
+  if (null != memberBytes) {
+if 
(Arrays.equals(CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY, memberBytes)) {
+  memberBytes = null;
+} else if (memberBytes.length == 0) {
+  memberBytes = null;
+}
+record[index] = 
DataTypeUtil.getDataBasedOnDataTypeForNoDictionaryColumn(memberBytes,
+dimColumnEvaluatorInfo.getDimension().getDataType());
+  }
+} else {
+  int dictionaryValue = ByteUtil.toInt(memberBytes, 0);
+  if 
(dimColumnEvaluatorInfo.getDimension().hasEncoding(Encoding.DICTIONARY)
+  && 
!dimColumnEvaluatorInfo.getDimension().hasEncoding(Encoding.DIRECT_DICTIONARY)) 
{
+memberString =
+
getFilterActualValueFromDictionaryValue(dimColumnEvaluatorInfo, 
dictionaryValue);
+record[index] = 
DataTypeUtil.getDataBasedOnDataType(memberString,
+dimColumnEvaluatorInfo.getDimension().getDataType());
+  } else if (
+  
dimColumnEvaluatorInfo.getDimension().hasEncoding(Encoding.DIRECT_DICTIONARY)) {
+Object member = 
getFilterActualValueFromDirectDictionaryValue(dimColumnEvaluatorInfo,
+dictionaryValue);
+record[index] = member;
+  }
+}
+  } else {
+record[index] = value.getVal(index);
+  }
+}
+
+for (int i = 0; i < msrColEvalutorInfoList.size(); i++) {
+  MeasureColumnResolvedFilterInfo msrColumnEvalutorInfo = 
msrColEvalutorInfoList.get(i);
+  int index = msrColumnEvalutorInfo.getMeasure().getOrdinal() + 
dimOrdinalMax;
+  // add default value for the measure in case filter measure is not 
present
+  // in the current block measure list
+  if (!isMeasurePresentInCurrentBlock[i]) {
+byte[] defaultValue = 
msrColumnEvalutorInfo.getCarbonColumn().getDefaultValue();
+record[index] = RestructureUtil
+
.getMeasureDefaultValue(msrColumnEvalutorInfo.getCarbonColumn().getColumnSchema(),
--- End diff --

move `. getMeasureDefaultValue` to previous line


---

[GitHub] carbondata pull request #1996: [CARBONDATA-2200] Fix bug of LIKE operation o...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1996#discussion_r170451620
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
 ---
@@ -276,12 +277,82 @@ public BitSetGroup applyFilter(BlocksChunkHolder 
blockChunkHolder, boolean useBi
   public boolean applyFilter(RowIntf value, int dimOrdinalMax)
   throws FilterUnsupportedException, IOException {
 try {
-  return exp.evaluate(value).getBoolean();
+  Boolean result = exp.evaluate(createRow(value, 
dimOrdinalMax)).getBoolean();
+  return result == null ? false : result;
 } catch (FilterIllegalMemberException e) {
   throw new FilterUnsupportedException(e);
 }
   }
 
+  /**
+   * create row for row filter to evaluate expression
+   */
+  private RowIntf createRow(RowIntf value, int dimOrdinalMax) throws 
IOException {
+Object[] record = new Object[value.size()];
+String memberString;
+for (int i = 0; i < dimColEvaluatorInfoList.size(); i++) {
+  DimColumnResolvedFilterInfo dimColumnEvaluatorInfo = 
dimColEvaluatorInfoList.get(i);
+  int index = dimColumnEvaluatorInfo.getDimension().getOrdinal();
+  // if filter dimension is not present in the current add its default 
value
+  if (!isDimensionPresentInCurrentBlock[i]) {
+// fill default value here
+record[index] = getDimensionDefaultValue(dimColumnEvaluatorInfo);
+continue;
+  }
+  if 
(!dimColumnEvaluatorInfo.getDimension().getDataType().isComplexType()) {
+if (!dimColumnEvaluatorInfo.isDimensionExistsInCurrentSilce()) {
+  record[index] = 
dimColumnEvaluatorInfo.getDimension().getDefaultValue();
+}
+byte[] memberBytes = (byte[]) value.getVal(index);
+if 
(!dimColumnEvaluatorInfo.getDimension().hasEncoding(Encoding.DICTIONARY)) {
+  if (null != memberBytes) {
+if 
(Arrays.equals(CarbonCommonConstants.MEMBER_DEFAULT_VAL_ARRAY, memberBytes)) {
+  memberBytes = null;
+} else if (memberBytes.length == 0) {
+  memberBytes = null;
+}
+record[index] = 
DataTypeUtil.getDataBasedOnDataTypeForNoDictionaryColumn(memberBytes,
+dimColumnEvaluatorInfo.getDimension().getDataType());
+  }
+} else {
+  int dictionaryValue = ByteUtil.toInt(memberBytes, 0);
+  if 
(dimColumnEvaluatorInfo.getDimension().hasEncoding(Encoding.DICTIONARY)
+  && 
!dimColumnEvaluatorInfo.getDimension().hasEncoding(Encoding.DIRECT_DICTIONARY)) 
{
+memberString =
+
getFilterActualValueFromDictionaryValue(dimColumnEvaluatorInfo, 
dictionaryValue);
+record[index] = 
DataTypeUtil.getDataBasedOnDataType(memberString,
+dimColumnEvaluatorInfo.getDimension().getDataType());
+  } else if (
+  
dimColumnEvaluatorInfo.getDimension().hasEncoding(Encoding.DIRECT_DICTIONARY)) {
+Object member = 
getFilterActualValueFromDirectDictionaryValue(dimColumnEvaluatorInfo,
+dictionaryValue);
+record[index] = member;
+  }
+}
+  } else {
+record[index] = value.getVal(index);
+  }
+}
+
+for (int i = 0; i < msrColEvalutorInfoList.size(); i++) {
+  MeasureColumnResolvedFilterInfo msrColumnEvalutorInfo = 
msrColEvalutorInfoList.get(i);
+  int index = msrColumnEvalutorInfo.getMeasure().getOrdinal() + 
dimOrdinalMax;
+  // add default value for the measure in case filter measure is not 
present
+  // in the current block measure list
+  if (!isMeasurePresentInCurrentBlock[i]) {
+byte[] defaultValue = 
msrColumnEvalutorInfo.getCarbonColumn().getDefaultValue();
+record[index] = RestructureUtil
+
.getMeasureDefaultValue(msrColumnEvalutorInfo.getCarbonColumn().getColumnSchema(),
+defaultValue);
+continue;
--- End diff --

give comment why continue


---

[GitHub] carbondata pull request #1996: [CARBONDATA-2200] Fix bug of LIKE operation o...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1996#discussion_r170451613
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
 ---
@@ -276,12 +277,82 @@ public BitSetGroup applyFilter(BlocksChunkHolder 
blockChunkHolder, boolean useBi
   public boolean applyFilter(RowIntf value, int dimOrdinalMax)
   throws FilterUnsupportedException, IOException {
 try {
-  return exp.evaluate(value).getBoolean();
+  Boolean result = exp.evaluate(createRow(value, 
dimOrdinalMax)).getBoolean();
+  return result == null ? false : result;
 } catch (FilterIllegalMemberException e) {
   throw new FilterUnsupportedException(e);
 }
   }
 
+  /**
+   * create row for row filter to evaluate expression
+   */
+  private RowIntf createRow(RowIntf value, int dimOrdinalMax) throws 
IOException {
+Object[] record = new Object[value.size()];
+String memberString;
+for (int i = 0; i < dimColEvaluatorInfoList.size(); i++) {
+  DimColumnResolvedFilterInfo dimColumnEvaluatorInfo = 
dimColEvaluatorInfoList.get(i);
+  int index = dimColumnEvaluatorInfo.getDimension().getOrdinal();
+  // if filter dimension is not present in the current add its default 
value
+  if (!isDimensionPresentInCurrentBlock[i]) {
+// fill default value here
+record[index] = getDimensionDefaultValue(dimColumnEvaluatorInfo);
+continue;
--- End diff --

give comment why continue


---

[GitHub] carbondata pull request #1996: [CARBONDATA-2200] Fix bug of LIKE operation o...

[jackylk]

Github user jackylk commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1996#discussion_r170451511
  
--- Diff: 
core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java
 ---
@@ -276,12 +277,82 @@ public BitSetGroup applyFilter(BlocksChunkHolder 
blockChunkHolder, boolean useBi
   public boolean applyFilter(RowIntf value, int dimOrdinalMax)
   throws FilterUnsupportedException, IOException {
 try {
-  return exp.evaluate(value).getBoolean();
+  Boolean result = exp.evaluate(createRow(value, 
dimOrdinalMax)).getBoolean();
+  return result == null ? false : result;
 } catch (FilterIllegalMemberException e) {
   throw new FilterUnsupportedException(e);
 }
   }
 
+  /**
+   * create row for row filter to evaluate expression
--- End diff --

describe what is input row and what is the return row


---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3657/



---

[GitHub] carbondata pull request #1997: [CARBONDATA-2201] NPE fixed for Streaming eve...

[sraghunandan]

Github user sraghunandan commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/1997#discussion_r170451392
  
--- Diff: 
streaming/src/main/scala/org/apache/spark/sql/execution/streaming/CarbonAppendableStreamSink.scala
 ---
@@ -102,7 +102,7 @@ class CarbonAppendableStreamSink(
 carbonLoadModel.getFactFilePath,
 false,
 parameters.asJava,
-null,
+parameters.asJava,
--- End diff --

add comment why we need to pass same parameters twice


---

[GitHub] carbondata issue #1997: [CARBONDATA-2201] NPE fixed for Streaming event

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1997
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3879/



---

[GitHub] carbondata issue #1997: [CARBONDATA-2201] NPE fixed for Streaming event

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1997
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2634/



---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[jackylk]

Github user jackylk commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
please add PR description


---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
Build Failed  with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3878/



---

[GitHub] carbondata issue #1996: [CARBONDATA-2200] Fix bug of LIKE operation on strea...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/1996
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/2633/



---

[jira] [Created] (CARBONDATA-2201) firing the event before streaming causes NPE

[Rahul Kumar (JIRA)]

Rahul Kumar created CARBONDATA-2201:
---

 Summary: firing the event before streaming causes NPE
 Key: CARBONDATA-2201
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2201
 Project: CarbonData
  Issue Type: Improvement
Reporter: Rahul Kumar
Assignee: Rahul Kumar






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1997: [CARBONDATA-2201] NPE fixed for Streaming eve...

[rahulforallp]

GitHub user rahulforallp opened a pull request:

https://github.com/apache/carbondata/pull/1997

[CARBONDATA-2201] NPE fixed for Streaming event

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rahulforallp/incubator-carbondata stream_npe

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1997.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1997


commit b94ae7c86f67b6d4eedfc9ed56ca2f7bde2a2ea9
Author: rahulforallp 
Date:   2018-02-25T09:55:26Z

NPE fixed for Streaming event




---

[GitHub] carbondata pull request #1996: [CARBONDATA-2200] Fix bug of LIKE operation o...

[QiangCai]

GitHub user QiangCai opened a pull request:

https://github.com/apache/carbondata/pull/1996

[CARBONDATA-2200] Fix bug of LIKE operation on streaming table


 - [x] Any interfaces changed?
 no
 - [x] Any backward compatibility impacted?
 no
 - [x] Document update required?
 no
 - [x] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
 added UT
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [x] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 
small changes


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/QiangCai/carbondata streaming_like

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/1996.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1996


commit cac99ccda928a44c5e7bc96a4155cc8cf4b2cfe3
Author: QiangCai 
Date:   2018-02-25T10:53:41Z

fix NPE for LIKE operation on streaming table




---

[jira] [Created] (CARBONDATA-2200) Like operation on streaming table throwing Exception

[QiangCai (JIRA)]

QiangCai created CARBONDATA-2200:


 Summary: Like operation on streaming table throwing Exception
 Key: CARBONDATA-2200
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2200
 Project: CarbonData
  Issue Type: Bug
Reporter: QiangCai


In TestStreamingTableOperation.scala file,

for test case "query on stream table with dictionary, sort_columns"

the following SQL is not working:

select * from stream_table_filter where name like '%me%'

error log:

java.io.IOException: Failed to filter row in vector reader
 at 
org.apache.carbondata.hadoop.streaming.CarbonStreamRecordReader.scanBlockletAndFillVector(CarbonStreamRecordReader.java:449)
 at 
org.apache.carbondata.hadoop.streaming.CarbonStreamRecordReader.nextColumnarBatch(CarbonStreamRecordReader.java:328)
 at 
org.apache.carbondata.hadoop.streaming.CarbonStreamRecordReader.nextKeyValue(CarbonStreamRecordReader.java:309)
 at 
org.apache.carbondata.spark.rdd.CarbonScanRDD$$anon$1.hasNext(CarbonScanRDD.scala:390)
 at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.scan_nextBatch$(Unknown
 Source)
 at 
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
 Source)
 at 
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
 at 
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
 at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:231)
 at 
org.apache.spark.sql.execution.SparkPlan$$anonfun$2.apply(SparkPlan.scala:225)
 at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826)
 at 
org.apache.spark.rdd.RDD$$anonfun$mapPartitionsInternal$1$$anonfun$apply$25.apply(RDD.scala:826)
 at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)
 at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)
 at org.apache.spark.rdd.RDD.iterator(RDD.scala:287)
 at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
 at org.apache.spark.scheduler.Task.run(Task.scala:99)
 at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
 at java.lang.Thread.run(Thread.java:745)
Caused by: 
org.apache.carbondata.core.scan.expression.exception.FilterUnsupportedException:
 [B cannot be cast to org.apache.spark.unsafe.types.UTF8String
 at 
org.apache.spark.sql.SparkUnknownExpression.evaluate(SparkUnknownExpression.scala:50)
 at 
org.apache.carbondata.core.scan.filter.executer.RowLevelFilterExecuterImpl.applyFilter(RowLevelFilterExecuterImpl.java:279)
 at 
org.apache.carbondata.hadoop.streaming.CarbonStreamRecordReader.scanBlockletAndFillVector(CarbonStreamRecordReader.java:444)
 ... 20 more

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (CARBONDATA-2149) Displayed complex type data is error when use DataFrame to write complex type data.

[Liang Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Liang Chen resolved CARBONDATA-2149.

Resolution: Fixed

> Displayed complex type data is error when use DataFrame to write complex type 
> data.
> ---
>
> Key: CARBONDATA-2149
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2149
> Project: CarbonData
>  Issue Type: Bug
>  Components: data-load, spark-integration
>Affects Versions: 1.3.1
>Reporter: Zhichao  Zhang
>Assignee: Zhichao  Zhang
>Priority: Minor
> Fix For: 1.3.1
>
>  Time Spent: 3h 20m
>  Remaining Estimate: 0h
>
> The default value of 'complex_delimiter_level_1' and 
> 'complex_delimiter_level_2' is wrong, it must be '$' and ':', not be '\ \ $' 
> and '\ \ :'. Escape characters '\ \ ' need to be added only when using 
> delimiters in ArrayParserImpl or StructParserImpl. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #1962: [CARBONDATA-2149]Fix complex type data displa...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/1962


---

[GitHub] carbondata issue #1962: [CARBONDATA-2149]Fix complex type data displaying er...

[chenliang613]

Github user chenliang613 commented on the issue:

https://github.com/apache/carbondata/pull/1962
  
LGTM


---

[GitHub] carbondata issue #1962: [CARBONDATA-2149]Fix complex type data displaying er...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/1962
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/3656/



---