[GitHub] carbondata pull request #2359: [Hoxfix] Upgrade dev version to 1.5.0-SNAPSHO...
GitHub user chenliang613 opened a pull request: https://github.com/apache/carbondata/pull/2359 [Hoxfix] Upgrade dev version to 1.5.0-SNAPSHOT and fix some small issues 1.Upgrade dev version to 1.5.0-SNAPSHOT 2.Fix carbon-spark-sql issue 3.Remove hadoop 2.2 profile Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [X] Any interfaces changed? NA - [Y] Any backward compatibility impacted? - [Y] Document update required? - [X] Testing done Please provide details on - Whether new unit test cases have been added or why no new tests are required? - How it is tested? Please attach test report. - Is it a performance related change? Please attach the performance test report. - Any additional information to help reviewers in testing this change. - [X] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/chenliang613/carbondata upgrade_1.5 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2359.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2359 commit fa89b051ef990dd9d4c2f2a3874db050c4e0b2c7 Author: chenliang613 Date: 2018-06-02T04:58:57Z upgrade dev version to 1.5.0-SNAPSHOT and fix some small issues ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2345 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5207/ ---
[GitHub] carbondata issue #2335: [WIP] integrate carbonstore mv branch
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2335 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5206/ ---
[jira] [Created] (CARBONDATA-2572) [MV] DataMap Choosing policy should be table size based not 1st Matched based
Babulal created CARBONDATA-2572: --- Summary: [MV] DataMap Choosing policy should be table size based not 1st Matched based Key: CARBONDATA-2572 URL: https://issues.apache.org/jira/browse/CARBONDATA-2572 Project: CarbonData Issue Type: Bug Reporter: Babulal Create Table and create datamap 0: jdbc:hive2://10.18.222.231:23040> show datamap on table babu_2; +--++---+--+ | DataMapName | ClassName | Associated Table | +--++---+--+ | agg_69 | mv | default.agg_69_table | | agg_70 | mv | default.agg_70_table | +--++-- create datamap agg_69 using 'mv' as select unit_id,y_year_id,country_id,sum(dollar_value_id),max(dollar_value),min(dollar_value),sum(quantity),min(quantity),max(quantity) from babu2 group by unit_id,y_year_id,country_id create datamap agg_70 using 'mv' as select unit_id,sum(dollar_value_id) from babu3 group by unit_id; Size of each MV table BLR123613:/srv/spark2.2Bigdata/install/spark/sparkJdbc # hadoop fs -du -s -h /user/hive/warehouse/carbon.store/default/agg_69_table *86.4 K* /user/hive/warehouse/carbon.store/default/agg_69_table BLR123613:/srv/spark2.2Bigdata/install/spark/sparkJdbc # hadoop fs -du -s -h /user/hive/warehouse/carbon.store/default/agg_70_table *2.9 K /*user/hive/warehouse/carbon.store/default/agg_70_table BLR123613:/srv/spark2.2Bigdata/install/spark/sparkJdbc # hadoop fs -du -s -h /user/hive/warehouse/carbon.store/default/agg_68_table Now Run select query which is given during agg_70 , 0: jdbc:hive2://10.18.222.231:23040> explain select unit_id,sum(dollar_value) from babu_2group by unit; + | plan | + | == Physical Plan == *... +- *BatchedScan CarbonDatasourceHadoopRelation [ Database name :default, Table name :*agg_69_table*, Schema But as per Size agg_70 should be selected -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2345 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5075/ ---
[GitHub] carbondata issue #2335: [WIP] integrate carbonstore mv branch
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2335 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5205/ ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2345 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6237/ ---
[GitHub] carbondata issue #2351: [CARBONDATA-2559] task id set for each carbonReader ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2351 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5204/ ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2345 retest this please ---
[GitHub] carbondata issue #2335: [WIP] integrate carbonstore mv branch
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2335 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5073/ ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2345 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6236/ ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2345 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5074/ ---
[GitHub] carbondata issue #2358: [CARBONDATA-2571] Calculating the carbonindex and ca...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2358 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5203/ ---
[GitHub] carbondata pull request #2357: [CARBONDATA-2569] Change the strategy of Sear...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2357#discussion_r192418603
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -101,8 +100,8 @@ class CarbonSession(@transient val sc: SparkContext,
} catch {
case e: Exception =>
logError(String.format(
--- End diff --
It is in scala here, why still use `String.format`? Just use s"$var..."
---
[GitHub] carbondata pull request #2357: [CARBONDATA-2569] Change the strategy of Sear...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2357#discussion_r192418789
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -171,19 +170,25 @@ class CarbonSession(@transient val sc: SparkContext,
*/
private def trySearchMode(qe: QueryExecution, sse: SQLStart): DataFrame
= {
val analyzed = qe.analyzed
+val LOG: LogService =
LogServiceFactory.getLogService(classOf[CarbonSession].getName)
analyzed match {
case _@Project(columns, _@Filter(expr, s: SubqueryAlias))
if s.child.isInstanceOf[LogicalRelation] &&
s.child.asInstanceOf[LogicalRelation].relation
.isInstanceOf[CarbonDatasourceHadoopRelation] =>
+LOG.info(String.format("Search service started and supports: %s",
sse.sqlText))
runSearch(analyzed, columns, expr,
s.child.asInstanceOf[LogicalRelation])
case gl@GlobalLimit(_, ll@LocalLimit(_, p@Project(columns,
_@Filter(expr, s: SubqueryAlias
if s.child.isInstanceOf[LogicalRelation] &&
s.child.asInstanceOf[LogicalRelation].relation
.isInstanceOf[CarbonDatasourceHadoopRelation] =>
val logicalRelation = s.child.asInstanceOf[LogicalRelation]
+LOG.info(String.format("Search service started and supports: %s",
sse.sqlText))
runSearch(analyzed, columns, expr, logicalRelation, gl.maxRows,
ll.maxRows)
case _ =>
+LOG.info(String.format(
--- End diff --
same as above
---
[GitHub] carbondata pull request #2357: [CARBONDATA-2569] Change the strategy of Sear...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2357#discussion_r192420617
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -171,19 +170,25 @@ class CarbonSession(@transient val sc: SparkContext,
*/
private def trySearchMode(qe: QueryExecution, sse: SQLStart): DataFrame
= {
val analyzed = qe.analyzed
+val LOG: LogService =
LogServiceFactory.getLogService(classOf[CarbonSession].getName)
analyzed match {
case _@Project(columns, _@Filter(expr, s: SubqueryAlias))
if s.child.isInstanceOf[LogicalRelation] &&
s.child.asInstanceOf[LogicalRelation].relation
.isInstanceOf[CarbonDatasourceHadoopRelation] =>
+LOG.info(String.format("Search service started and supports: %s",
sse.sqlText))
runSearch(analyzed, columns, expr,
s.child.asInstanceOf[LogicalRelation])
case gl@GlobalLimit(_, ll@LocalLimit(_, p@Project(columns,
_@Filter(expr, s: SubqueryAlias
if s.child.isInstanceOf[LogicalRelation] &&
s.child.asInstanceOf[LogicalRelation].relation
.isInstanceOf[CarbonDatasourceHadoopRelation] =>
val logicalRelation = s.child.asInstanceOf[LogicalRelation]
+LOG.info(String.format("Search service started and supports: %s",
sse.sqlText))
runSearch(analyzed, columns, expr, logicalRelation, gl.maxRows,
ll.maxRows)
case _ =>
+LOG.info(String.format(
+ "Search service started, but don't support: %s, and running it
with SparkSQL",
--- End diff --
and will run it with SparkSQL
---
[GitHub] carbondata pull request #2357: [CARBONDATA-2569] Change the strategy of Sear...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2357#discussion_r192418665
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -171,19 +170,25 @@ class CarbonSession(@transient val sc: SparkContext,
*/
private def trySearchMode(qe: QueryExecution, sse: SQLStart): DataFrame
= {
val analyzed = qe.analyzed
+val LOG: LogService =
LogServiceFactory.getLogService(classOf[CarbonSession].getName)
analyzed match {
case _@Project(columns, _@Filter(expr, s: SubqueryAlias))
if s.child.isInstanceOf[LogicalRelation] &&
s.child.asInstanceOf[LogicalRelation].relation
.isInstanceOf[CarbonDatasourceHadoopRelation] =>
+LOG.info(String.format("Search service started and supports: %s",
sse.sqlText))
--- End diff --
It is in scala here, why still use String.format?
---
[GitHub] carbondata pull request #2357: [CARBONDATA-2569] Change the strategy of Sear...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2357#discussion_r192420491
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -171,19 +170,25 @@ class CarbonSession(@transient val sc: SparkContext,
*/
private def trySearchMode(qe: QueryExecution, sse: SQLStart): DataFrame
= {
val analyzed = qe.analyzed
+val LOG: LogService =
LogServiceFactory.getLogService(classOf[CarbonSession].getName)
analyzed match {
case _@Project(columns, _@Filter(expr, s: SubqueryAlias))
if s.child.isInstanceOf[LogicalRelation] &&
s.child.asInstanceOf[LogicalRelation].relation
.isInstanceOf[CarbonDatasourceHadoopRelation] =>
+LOG.info(String.format("Search service started and supports: %s",
sse.sqlText))
runSearch(analyzed, columns, expr,
s.child.asInstanceOf[LogicalRelation])
case gl@GlobalLimit(_, ll@LocalLimit(_, p@Project(columns,
_@Filter(expr, s: SubqueryAlias
if s.child.isInstanceOf[LogicalRelation] &&
s.child.asInstanceOf[LogicalRelation].relation
.isInstanceOf[CarbonDatasourceHadoopRelation] =>
val logicalRelation = s.child.asInstanceOf[LogicalRelation]
+LOG.info(String.format("Search service started and supports: %s",
sse.sqlText))
runSearch(analyzed, columns, expr, logicalRelation, gl.maxRows,
ll.maxRows)
case _ =>
+LOG.info(String.format(
--- End diff --
the level is `warn` or `info`?
---
[GitHub] carbondata pull request #2357: [CARBONDATA-2569] Change the strategy of Sear...
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2357#discussion_r192418722
--- Diff:
integration/spark2/src/main/scala/org/apache/spark/sql/CarbonSession.scala ---
@@ -171,19 +170,25 @@ class CarbonSession(@transient val sc: SparkContext,
*/
private def trySearchMode(qe: QueryExecution, sse: SQLStart): DataFrame
= {
val analyzed = qe.analyzed
+val LOG: LogService =
LogServiceFactory.getLogService(classOf[CarbonSession].getName)
analyzed match {
case _@Project(columns, _@Filter(expr, s: SubqueryAlias))
if s.child.isInstanceOf[LogicalRelation] &&
s.child.asInstanceOf[LogicalRelation].relation
.isInstanceOf[CarbonDatasourceHadoopRelation] =>
+LOG.info(String.format("Search service started and supports: %s",
sse.sqlText))
runSearch(analyzed, columns, expr,
s.child.asInstanceOf[LogicalRelation])
case gl@GlobalLimit(_, ll@LocalLimit(_, p@Project(columns,
_@Filter(expr, s: SubqueryAlias
if s.child.isInstanceOf[LogicalRelation] &&
s.child.asInstanceOf[LogicalRelation].relation
.isInstanceOf[CarbonDatasourceHadoopRelation] =>
val logicalRelation = s.child.asInstanceOf[LogicalRelation]
+LOG.info(String.format("Search service started and supports: %s",
sse.sqlText))
--- End diff --
same as above
---
[GitHub] carbondata issue #2335: [WIP] integrate carbonstore mv branch
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2335 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6235/ ---
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2357 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5202/ ---
[GitHub] carbondata issue #2351: [CARBONDATA-2559] task id set for each carbonReader ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2351 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5070/ ---
[GitHub] carbondata issue #2335: [WIP] integrate carbonstore mv branch
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2335 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5071/ ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2345 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6234/ ---
[GitHub] carbondata issue #2335: [WIP] integrate carbonstore mv branch
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2335 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6233/ ---
[GitHub] carbondata issue #2181: [CARBONDATA-2355] Support run SQL on carbondata file...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2181 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5201/ ---
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2357 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5069/ ---
[GitHub] carbondata issue #2351: [CARBONDATA-2559] task id set for each carbonReader ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2351 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6232/ ---
[GitHub] carbondata issue #2358: [CARBONDATA-2571] Calculating the carbonindex and ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2358 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5068/ ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2345 retest this please ---
[GitHub] carbondata issue #2351: [CARBONDATA-2559] task id set for each carbonReader ...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2351 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5200/ ---
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2357 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6230/ ---
[GitHub] carbondata issue #2358: [CARBONDATA-2571] Calculating the carbonindex and ca...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2358 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6231/ ---
[GitHub] carbondata issue #2181: [CARBONDATA-2355] Support run SQL on carbondata file...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2181 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5067/ ---
[GitHub] carbondata issue #2351: [CARBONDATA-2559] task id set for each carbonReader ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2351 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5066/ ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2345 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5199/ ---
[jira] [Resolved] (CARBONDATA-2566) Optimize CarbonReaderExample
[ https://issues.apache.org/jira/browse/CARBONDATA-2566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] kumar vishal resolved CARBONDATA-2566. -- Resolution: Fixed > Optimize CarbonReaderExample > > > Key: CARBONDATA-2566 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2566 > Project: CarbonData > Issue Type: Improvement >Reporter: xubo245 >Assignee: xubo245 >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > Optimize CarbonReaderExample -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2357 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5198/ ---
[GitHub] carbondata pull request #2356: [CARBONDATA-2566] Optimize CarbonReaderExampl...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2356 ---
[GitHub] carbondata issue #2356: [CARBONDATA-2566] Optimize CarbonReaderExample
Github user sounakr commented on the issue: https://github.com/apache/carbondata/pull/2356 LGTM ---
[GitHub] carbondata issue #2181: [CARBONDATA-2355] Support run SQL on carbondata file...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2181 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6229/ ---
[GitHub] carbondata issue #2351: [CARBONDATA-2559] task id set for each carbonReader ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2351 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6228/ ---
[jira] [Resolved] (CARBONDATA-2355) Support run SQL on carbon files directly, which is generated by SDk
[
https://issues.apache.org/jira/browse/CARBONDATA-2355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Chen resolved CARBONDATA-2355.
Resolution: Fixed
Fix Version/s: 1.4.1
> Support run SQL on carbon files directly, which is generated by SDk
> ---
>
> Key: CARBONDATA-2355
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2355
> Project: CarbonData
> Issue Type: Improvement
>Reporter: xubo245
>Assignee: xubo245
>Priority: Major
> Fix For: 1.4.1
>
> Time Spent: 11h
> Remaining Estimate: 0h
>
> Spark support run sql on files directly:
> https://spark.apache.org/docs/latest/sql-programming-guide.html#run-sql-on-files-directly
> CarbonData also should support.
> This is for carbonfile generated by SDK
> Now only support directly read the file directory with one carbonfile.
> For example:
> {code:java}
> select * FROM carbonfile.$filePath`
> {code}
> No need to create table and user can read data firectly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #2181: [CARBONDATA-2355] Support run SQL on carbonda...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2181 ---
[GitHub] carbondata issue #2282: [CARBONDATA-2456] Handling request by shard in searc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2282 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5063/ ---
[GitHub] carbondata issue #2181: [CARBONDATA-2355] Support run SQL on carbondata file...
Github user chenliang613 commented on the issue: https://github.com/apache/carbondata/pull/2181 LGTM ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2345 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6227/ ---
[GitHub] carbondata issue #2305: [CARBONDATA-2478] Added datamap-developer-guide.md f...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2305 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5197/ ---
[GitHub] carbondata pull request #2358: [CARBONDATA-2571] Calculating the carbonindex...
GitHub user dhatchayani opened a pull request: https://github.com/apache/carbondata/pull/2358 [CARBONDATA-2571] Calculating the carbonindex and carbondata file size of a table is wrong While calculating the carbonindex and carbondata file size, we have to consider both the files and mergeFileName fields. We should get the list of index files from these 2 fields and then calculate the size of the files. Be sure to do all of the following checklist to help us incorporate your contribution quickly and easily: - [ ] Any interfaces changed? - [ ] Any backward compatibility impacted? - [ ] Document update required? - [x] Testing done Manual Testing - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. You can merge this pull request into a Git repository by running: $ git pull https://github.com/dhatchayani/carbondata CARBONDATA-2571 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/carbondata/pull/2358.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #2358 commit 0490e27271f6d75445c7e7981cfec8c6e21bcc73 Author: dhatchayani Date: 2018-06-01T09:43:38Z [CARBONDATA-2571] Calculating the carbonindex and carbondata file size of a table is wrong ---
[jira] [Created] (CARBONDATA-2571) Calculating the carbonindex and carbondata file size of a table is wrong
dhatchayani created CARBONDATA-2571: --- Summary: Calculating the carbonindex and carbondata file size of a table is wrong Key: CARBONDATA-2571 URL: https://issues.apache.org/jira/browse/CARBONDATA-2571 Project: CarbonData Issue Type: Bug Reporter: dhatchayani Assignee: dhatchayani -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2181: [CARBONDATA-2355] Support run SQL on carbonda...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2181#discussion_r192343540
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/DirectSQLExample.scala
---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples
+
+import java.io.File
+
+import org.apache.commons.io.FileUtils
+
+import org.apache.carbondata.core.metadata.datatype.DataTypes
+import org.apache.carbondata.examples.util.ExampleUtils
+import org.apache.carbondata.sdk.file.{CarbonWriter, Field, Schema}
+
+/**
+ * Running SQL on carbon files directly
+ * No need to create table first
+ * TODO: support more than one carbon file
+ */
+object DirectSQLExample {
+
+ // prepare SDK writer output
+ def buildTestData(
+ path: String,
+ num: Int = 3,
+ persistSchema: Boolean = false): Any = {
+
+// getCanonicalPath gives path with \, so code expects /. Need to
handle in code ?
--- End diff --
ok, done. and optimize all place in the project
---
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2357 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5064/ ---
[GitHub] carbondata issue #2305: [CARBONDATA-2478] Added datamap-developer-guide.md f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2305 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5062/ ---
[GitHub] carbondata pull request #2181: [CARBONDATA-2355] Support run SQL on carbonda...
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2181#discussion_r192340086
--- Diff:
integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/createTable/TestCreateTableUsingSparkCarbonFileFormat.scala
---
@@ -153,6 +153,34 @@ class TestCreateTableUsingSparkCarbonFileFormat
extends QueryTest with BeforeAnd
cleanTestData()
}
+ test("Running SQL directly and read carbondata files (sdk Writer Output)
using the SparkCarbonFileFormat ") {
+buildTestData(false)
+assert(new File(filePath).exists())
+sql("DROP TABLE IF EXISTS sdkOutputTable")
+
+//data source file format
+if (sqlContext.sparkContext.version.startsWith("2.1")) {
+ //data source file format
+ sql(s"""CREATE TABLE sdkOutputTable USING carbonfile OPTIONS (PATH
'$filePath') """)
--- End diff --
if this is a key work, should use 'carbonfile'
---
[jira] [Commented] (CARBONDATA-2570) Carbon SDK Reader, second time reader instance have an issue in cluster test
[
https://issues.apache.org/jira/browse/CARBONDATA-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497763#comment-16497763
]
Ajantha Bhat commented on CARBONDATA-2570:
--
Steps:
# Take sdk jars and dependent jars
# create a intellij test project without spark cluster dependency
# Create a carbon reader on SDK writer's output. Read files and close the
reader.
# Create a reader on another set of SDK writer output (different schema) but
same table name.Now can observe that read fails due to schema mismatch. This is
because old blocklet datamap with same table name is still present
> Carbon SDK Reader, second time reader instance have an issue in cluster test
>
>
> Key: CARBONDATA-2570
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2570
> Project: CarbonData
> Issue Type: Bug
>Reporter: Ajantha Bhat
>Assignee: Ajantha Bhat
>Priority: Major
>
> Debugged the issue, This is happening only in cluster. Not in local.
> root cause: old table's blocklet datamap is not cleared.
>
> solution: In CarbonReader.close() API used for clearing datamap is not
> clearing all the datamap in cluster
> so change
> DataMapStoreManager.getInstance().getDefaultDataMap(queryModel.getTable()).clear();
> to
> DataMapStoreManager.getInstance()
> .clearDataMaps({color:#660e7a}queryModel{color}.getTable().getAbsoluteTableIdentifier());
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata pull request #2181: [CARBONDATA-2355] Support run SQL on carbonda...
Github user chenliang613 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2181#discussion_r192337901
--- Diff:
examples/spark2/src/main/scala/org/apache/carbondata/examples/DirectSQLExample.scala
---
@@ -0,0 +1,100 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples
+
+import java.io.File
+
+import org.apache.commons.io.FileUtils
+
+import org.apache.carbondata.core.metadata.datatype.DataTypes
+import org.apache.carbondata.examples.util.ExampleUtils
+import org.apache.carbondata.sdk.file.{CarbonWriter, Field, Schema}
+
+/**
+ * Running SQL on carbon files directly
+ * No need to create table first
+ * TODO: support more than one carbon file
+ */
+object DirectSQLExample {
+
+ // prepare SDK writer output
+ def buildTestData(
+ path: String,
+ num: Int = 3,
+ persistSchema: Boolean = false): Any = {
+
+// getCanonicalPath gives path with \, so code expects /. Need to
handle in code ?
--- End diff --
please optimize the comment.
---
[GitHub] carbondata issue #2282: [CARBONDATA-2456] Handling request by shard in searc...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2282 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5196/ ---
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2357 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6226/ ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2345 ok, I will try It in my local machine ---
[GitHub] carbondata issue #2305: [CARBONDATA-2478] Added datamap-developer-guide.md f...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2305 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6225/ ---
[GitHub] carbondata issue #2282: [CARBONDATA-2456] Handling request by shard in searc...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2282 @jacky CI pass, please review it. ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user ajantha-bhat commented on the issue: https://github.com/apache/carbondata/pull/2345 @xubo245 : Yes, It is there in PR, refer the CarbonRecordReader.java ---
[GitHub] carbondata issue #2282: [CARBONDATA-2456] Handling request by shard in searc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2282 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6224/ ---
[GitHub] carbondata issue #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDATA-2570...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2345 Have you add it in this PR? so change DataMapStoreManager.getInstance().getDefaultDataMap(queryModel.getTable()).clear(); to DataMapStoreManager.getInstance().clearDataMaps(queryModel.getTable().getAbsoluteTableIdentifie()); ---
[GitHub] carbondata issue #2351: [CARBONDATA-2559] task id set for each carbonReader ...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2351 please add some description for this PR ---
[GitHub] carbondata pull request #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDA...
Github user ajantha-bhat commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2345#discussion_r192333461
--- Diff:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
---
@@ -996,4 +979,30 @@ public boolean canAllow(CarbonTable carbonTable,
TableOperation operation) {
}
return indexColumn;
}
+
+ /**
+ * update the carbon table by using the passed tableInfo
+ * @param table
--- End diff --
done
---
[GitHub] carbondata issue #2282: [CARBONDATA-2456] Handling request by shard in searc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2282 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5060/ ---
[GitHub] carbondata issue #2356: [CARBONDATA-2566] Optimize CarbonReaderExample
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2356 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5061/ ---
[GitHub] carbondata pull request #2345: [CARBONDATA-2557] [CARBONDATA-2472] [CARBONDA...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2345#discussion_r192325777
--- Diff:
core/src/main/java/org/apache/carbondata/core/metadata/schema/table/CarbonTable.java
---
@@ -996,4 +979,30 @@ public boolean canAllow(CarbonTable carbonTable,
TableOperation operation) {
}
return indexColumn;
}
+
+ /**
+ * update the carbon table by using the passed tableInfo
+ * @param table
--- End diff --
please add one blank line
---
[jira] [Commented] (CARBONDATA-2570) Carbon SDK Reader, second time reader instance have an issue in cluster test
[
https://issues.apache.org/jira/browse/CARBONDATA-2570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16497707#comment-16497707
]
xubo245 commented on CARBONDATA-2570:
-
Can you give the test step and exception info in here?
> Carbon SDK Reader, second time reader instance have an issue in cluster test
>
>
> Key: CARBONDATA-2570
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2570
> Project: CarbonData
> Issue Type: Bug
>Reporter: Ajantha Bhat
>Assignee: Ajantha Bhat
>Priority: Major
>
> Debugged the issue, This is happening only in cluster. Not in local.
> root cause: old table's blocklet datamap is not cleared.
>
> solution: In CarbonReader.close() API used for clearing datamap is not
> clearing all the datamap in cluster
> so change
> DataMapStoreManager.getInstance().getDefaultDataMap(queryModel.getTable()).clear();
> to
> DataMapStoreManager.getInstance()
> .clearDataMaps({color:#660e7a}queryModel{color}.getTable().getAbsoluteTableIdentifier());
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] carbondata issue #2356: [CARBONDATA-2566] Optimize CarbonReaderExample
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2356 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5195/ ---
[GitHub] carbondata issue #2356: [CARBONDATA-2566] Optimize CarbonReaderExample
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2356 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6223/ ---
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2357 @jackylk Please review it ---
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2357 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6221/ ---
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2357 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5059/ ---
[GitHub] carbondata issue #2282: [CARBONDATA-2456] Handling request by shard in searc...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2282 SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5194/ ---
[GitHub] carbondata issue #2351: [CARBONDATA-2559] task id set for each carbonReader ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2351 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5058/ ---
[GitHub] carbondata pull request #2356: [CARBONDATA-2566] Optimize CarbonReaderExampl...
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2356#discussion_r192310351
--- Diff: docs/sdk-guide.md ---
@@ -408,17 +408,22 @@ External client can make use of this reader to read
CarbonData files without Car
String path = "./testWriteFiles";
CarbonReader reader = CarbonReader
.builder(path, "_temp")
-.projection(new String[]{"name", "age"})
+.projection(new String[]{"stringField", "shortField", "intField",
"longField",
--- End diff --
ok,done
---
[GitHub] carbondata issue #2282: [CARBONDATA-2456] Handling request by shard in searc...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2282 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6222/ ---
[GitHub] carbondata issue #2351: [CARBONDATA-2559] task id set for each carbonReader ...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2351 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/6220/ ---
[jira] [Resolved] (CARBONDATA-2529) S3 Example not working with Hadoop 2.8.3
[ https://issues.apache.org/jira/browse/CARBONDATA-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Chen resolved CARBONDATA-2529. Resolution: Fixed Fix Version/s: 1.4.1 > S3 Example not working with Hadoop 2.8.3 > > > Key: CARBONDATA-2529 > URL: https://issues.apache.org/jira/browse/CARBONDATA-2529 > Project: CarbonData > Issue Type: Bug > Components: core >Affects Versions: 1.3.1 >Reporter: Bhavya Aggarwal >Assignee: Bhavya Aggarwal >Priority: Minor > Fix For: 1.4.1 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > S3 Example not working with Hadoop 2.8.3 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata pull request #2340: [CARBONDATA-2529] Fixed S3 Issue for Hadoop 2...
Github user asfgit closed the pull request at: https://github.com/apache/carbondata/pull/2340 ---
[GitHub] carbondata issue #2357: [CARBONDATA-2569] Change the strategy of Search mode...
Github user xubo245 commented on the issue: https://github.com/apache/carbondata/pull/2357 retest this please ---
