[jira] [Created] (CARBONDATA-2318) Remove invalid table name(.ds_store) of presto integration

[Liang Chen (JIRA)]

Liang Chen created CARBONDATA-2318:
--

 Summary: Remove invalid table name(.ds_store) of presto 
integration 
 Key: CARBONDATA-2318
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2318
 Project: CarbonData
  Issue Type: Improvement
  Components: presto-integration
Reporter: Liang Chen


For presto integration , will get the invalid table name via "show tables from 
default"

As below.

presto:default> show tables from default;
 Table

 .ds_store
 carbon_table
 carbontable
 partition_bigtable
 partition_table
(5 rows)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null

[Liang Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16429203#comment-16429203
 ] 

Liang Chen commented on CARBONDATA-2282:


ok

> presto carbon does not support reading specific partition on which query is 
> fired mapreduce.input.carboninputformat.partitions.to.prune property is null
> 
>
> Key: CARBONDATA-2282
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2282
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, presto-integration
>Affects Versions: 1.3.0
>Reporter: zhangwei
>Assignee: anubhav tarar
>Priority: Major
> Fix For: 1.3.0
>
> Attachments: partitonToPrune.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2127: [CARBONDATA-2301][SDK] CarbonStore interface ...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2127#discussion_r179905093
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/store/LocalCarbonStore.java ---
@@ -0,0 +1,111 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.store;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.core.datastore.row.CarbonRow;
+import org.apache.carbondata.core.metadata.schema.table.CarbonTable;
+import org.apache.carbondata.core.scan.expression.Expression;
+import org.apache.carbondata.hadoop.CarbonProjection;
+import org.apache.carbondata.hadoop.api.CarbonTableInputFormat;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.mapreduce.InputSplit;
+import org.apache.hadoop.mapreduce.Job;
+import org.apache.hadoop.mapreduce.JobID;
+import org.apache.hadoop.mapreduce.RecordReader;
+import org.apache.hadoop.mapreduce.TaskAttemptID;
+import org.apache.hadoop.mapreduce.task.JobContextImpl;
+import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
+
+/**
+ * A CarbonStore implementation that works locally, without other compute 
framework dependency.
+ * It can be used to read data in local disk.
+ *
+ * Note that this class is experimental, it is not intended to be used in 
production.
+ */
+@InterfaceAudience.Internal
+class LocalCarbonStore extends MetaCachedCarbonStore {
+
+  @Override
+  public CarbonRow[] scan(String path, String[] projectColumns) throws 
IOException {
+return scan(path, projectColumns, null);
+  }
+
+  @Override
+  public CarbonRow[] scan(String path, String[] projectColumns, Expression 
filter)
+  throws IOException {
+CarbonTable table = getTable(path);
+if (table.isStreamingTable() || table.isHivePartitionTable()) {
+  throw new UnsupportedOperationException("streaming and partition 
table is not supported");
+}
+// TODO: use InputFormat to prune data and read data
+
+final CarbonTableInputFormat format = new CarbonTableInputFormat();
+final Job job = new Job(new Configuration());
+format.setTableInfo(job.getConfiguration(), table.getTableInfo());
+format.setTablePath(job.getConfiguration(), table.getTablePath());
+format.setTableName(job.getConfiguration(), table.getTableName());
+format.setDatabaseName(job.getConfiguration(), 
table.getDatabaseName());
+format.setCarbonReadSupport(job.getConfiguration(), 
CarbonRowReadSupport.class);
+if (filter != null) {
+  format.setFilterPredicates(job.getConfiguration(), filter);
+}
+if (projectColumns != null) {
--- End diff --

what is the behavior if the user does not provide projections?


---

[GitHub] carbondata pull request #2127: [CARBONDATA-2301][SDK] CarbonStore interface ...

[ravipesala]

Github user ravipesala commented on a diff in the pull request:

https://github.com/apache/carbondata/pull/2127#discussion_r179905076
  
--- Diff: 
store/sdk/src/main/java/org/apache/carbondata/store/CarbonStore.java ---
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.store;
+
+import java.io.Closeable;
+import java.io.IOException;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.common.annotations.InterfaceStability;
+import org.apache.carbondata.core.datastore.row.CarbonRow;
+import org.apache.carbondata.core.scan.expression.Expression;
+
+/**
+ * User can use {@link CarbonStore} to query data
+ */
+@InterfaceAudience.User
+@InterfaceStability.Unstable
+public interface CarbonStore extends Closeable {
+
+  /**
+   * Scan query on the data in the table path
+   * @param path table path
+   * @param projectColumns column names to read
+   * @return rows
+   * @throws IOException if unable to read files in table path
+   */
+  CarbonRow[] scan(
--- End diff --

I think it is better to return iterator rather than array, because in case 
of returning large rows it is not memory efficient to return all rows to 
driver. Internal implementation can have batch read from each executor and 
gives to drver.


---

[jira] [Commented] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null

[anubhav tarar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428479#comment-16428479
 ] 

anubhav tarar commented on CARBONDATA-2282:
---

pr is already raised for this issue[ 
#2139|https://github.com/apache/carbondata/pull/2139]

> presto carbon does not support reading specific partition on which query is 
> fired mapreduce.input.carboninputformat.partitions.to.prune property is null
> 
>
> Key: CARBONDATA-2282
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2282
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, presto-integration
>Affects Versions: 1.3.0
>Reporter: zhangwei
>Assignee: anubhav tarar
>Priority: Major
> Fix For: 1.3.0
>
> Attachments: partitonToPrune.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null

[anubhav tarar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428473#comment-16428473
 ] 

anubhav tarar commented on CARBONDATA-2282:
---

[~chenliang613] 

title of this issue was not completely correct by zhangwei i corrected it now 
as it can confuse and it is duplicated with my Jira 
https://issues.apache.org/jira/browse/CARBONDATA-2267 on which my pr is linked

> presto carbon does not support reading specific partition on which query is 
> fired mapreduce.input.carboninputformat.partitions.to.prune property is null
> 
>
> Key: CARBONDATA-2282
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2282
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, presto-integration
>Affects Versions: 1.3.0
>Reporter: zhangwei
>Assignee: anubhav tarar
>Priority: Major
> Fix For: 1.3.0
>
> Attachments: partitonToPrune.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired mapreduce.input.carboninputformat.partitions.to.prune property is null

[anubhav tarar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anubhav tarar updated CARBONDATA-2282:
--
Summary: presto carbon does not support reading specific partition on which 
query is fired mapreduce.input.carboninputformat.partitions.to.prune property 
is null  (was: presto carbon does not support reading specific partition on 
which query is fired )

> presto carbon does not support reading specific partition on which query is 
> fired mapreduce.input.carboninputformat.partitions.to.prune property is null
> 
>
> Key: CARBONDATA-2282
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2282
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, presto-integration
>Affects Versions: 1.3.0
>Reporter: zhangwei
>Assignee: anubhav tarar
>Priority: Major
> Fix For: 1.3.0
>
> Attachments: partitonToPrune.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (CARBONDATA-2282) presto carbon does not support reading specific partition on which query is fired

[anubhav tarar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anubhav tarar updated CARBONDATA-2282:
--
Summary: presto carbon does not support reading specific partition on which 
query is fired   (was: presto carbon not support query partition table)

> presto carbon does not support reading specific partition on which query is 
> fired 
> --
>
> Key: CARBONDATA-2282
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2282
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, presto-integration
>Affects Versions: 1.3.0
>Reporter: zhangwei
>Assignee: anubhav tarar
>Priority: Major
> Fix For: 1.3.0
>
> Attachments: partitonToPrune.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (CARBONDATA-2282) presto carbon not support query partition table

[anubhav tarar (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428463#comment-16428463
 ] 

anubhav tarar commented on CARBONDATA-2282:
---

[~chenliang613]  

In this PR we are handling the scenario of querying data to standard hive 
partitioned table in carbondata.

although you are getting the correct data from partition table but in current 
master branch we were iterating over all the partitions to get the data not the 
one which is required,which will be slow you can check value of property 
mapreduce.input.carboninputformat.partitions.to.prune in carboninputformat 
class after executing query on a particular partition it is null so when 
getPartitionsToPrune method is called it is returning null value instead of 
required partition specs 

 

but now in this pr we have identified the required partitions, so the query 
will be performed only on the required partition.
Firstly we are identifying the partition columns through domain constraints in 
presto and after that we are adding the Partition Spec to the configuration 
object to query partition

> presto carbon not support query partition table
> ---
>
> Key: CARBONDATA-2282
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2282
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, presto-integration
>Affects Versions: 1.3.0
>Reporter: zhangwei
>Assignee: anubhav tarar
>Priority: Major
> Fix For: 1.3.0
>
> Attachments: partitonToPrune.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #2143: [CARBONDATA-2317] oncurrent datamap with same name a...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2143
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4322/



---

[GitHub] carbondata issue #2143: [CARBONDATA-2317] oncurrent datamap with same name a...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2143
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3631/



---

[GitHub] carbondata issue #2143: [CARBONDATA-2317] oncurrent datamap with same name a...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2143
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4853/



---

[GitHub] carbondata issue #2142: [CARBONDATA-2316] [WIP] Executor task is failed but ...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2142
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4321/



---

[GitHub] carbondata pull request #2143: [CARBONDATA-2317] oncurrent datamap with same...

[rahulforallp]

GitHub user rahulforallp opened a pull request:

https://github.com/apache/carbondata/pull/2143

[CARBONDATA-2317] oncurrent datamap with same name and schema creation 
throws exception

…n throws exception

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed?
 
 - [ ] Any backward compatibility impacted?
 
 - [ ] Document update required?

 - [ ] Testing done
Please provide details on 
- Whether new unit test cases have been added or why no new tests 
are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance 
test report.
- Any additional information to help reviewers in testing this 
change.
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rahulforallp/incubator-carbondata 
concur_preagg

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2143.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2143


commit c82a046849562dc61b761f3aaefff1653f585a0f
Author: rahulforallp 
Date:   2018-04-06T09:47:54Z

[CARBONDATA-2317] oncurrent datamap with same name and schema creation 
throws exception




---

[jira] [Created] (CARBONDATA-2317) concurrent datamap with same name and schema creation throws exception

[Rahul Kumar (JIRA)]

Rahul Kumar created CARBONDATA-2317:
---

 Summary: concurrent datamap with same name and schema creation 
throws exception 
 Key: CARBONDATA-2317
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2317
 Project: CarbonData
  Issue Type: Improvement
Reporter: Rahul Kumar
Assignee: Rahul Kumar


*Steps to reproduce :* 
 # From Beeline user creates a table.
 #  From 4 concurrent terminals user tries to create datamaps.

*Query used to reproduce:* 
 # CREATE TABLE uniqdata(CUST_ID int,CUST_NAME String,ACTIVE_EMUI_VERSION 
string, DOB timestamp, DOJ timestamp, BIGINT_COLUMN1 bigint,BIGINT_COLUMN2 
bigint,DECIMAL_COLUMN1 decimal(30,10), DECIMAL_COLUMN2 
decimal(36,10),Double_COLUMN1 double, Double_COLUMN2 double,INTEGER_COLUMN1 
int) STORED BY 'org.apache.carbondata.format' 
TBLPROPERTIES('DICTIONARY_INCLUDE'='CUST_ID,CUST_NAME,ACTIVE_EMUI_VERSION,DOB,DOJ,BIGINT_COLUMN1,BIGINT_COLUMN2,DECIMAL_COLUMN1,DECIMAL_COLUMN2,Double_COLUMN1,Double_COLUMN2,INTEGER_COLUMN1');
 # create datamap uniqdata_agg on table uniqdata using 'preaggregate' as select 
cust_name, avg(cust_id) from uniqdata group by cust_id, cust_name;
 # create datamap uniqdata_agg_sum on table uniqdata using 'preaggregate' as 
select cust_name, sum(cust_id) from uniqdata group by cust_id, cust_name;
 # create datamap uniqdata_agg_count on table uniqdata using 'preaggregate' as 
select cust_name, count(cust_id) from uniqdata group by cust_id, cust_name;
 # create datamap uniqdata_agg_min on table uniqdata using 'preaggregate' as 
select cust_name, min(cust_id) from uniqdata group by cust_id, cust_name;
 # create datamap uniqdata_agg_max on table uniqdata using 'preaggregate' as 
select cust_name, max(cust_id) from uniqdata group by cust_id, cust_name;   
--->*The datamaps are tried to be created from 4 concurrent terminals.*

*2 of the datamaps fails with below error*
{quote}0: jdbc:hive2://ha-cluster/default> create datamap uniqdata_agg_min on 
table uniqdata using 'preaggregate' as select cust_name, min(cust_id) from 
uniqdata group by cust_id, cust_name;Error: 
org.apache.carbondata.spark.exception.ProcessMetaDataException: operation 
failed for default.uniqdata_uniqdata_agg_min: Create 
table'uniqdata_uniqdata_agg_min' in database 'default' failed, File does not 
exist: 
/user/hive/warehouse/carbon.store/default/uniqdata_uniqdata_agg_min/Metadata/schema.write
 (inode 20219) Holder DFSClient_NONMAPREDUCE_1307577692_216 does not have any 
open files.at 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2686)at
 
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFileInternal(FSDirWriteFileOp.java:625)at
 
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.completeFile(FSDirWriteFileOp.java:605)at
 
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.completeFile(FSNamesystem.java:2731)at
 
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.complete(NameNodeRpcServer.java:883)at
 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.complete(ClientNamenodeProtocolServerSideTranslatorPB.java:561)at
 
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)at
 
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:447)at
 org.apache.hadoop.ipc.RPC$Server.call(RPC.java:989)at 
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:847)at 
org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:790)at 
java.security.AccessController.doPrivileged(Native Method)at 
javax.security.auth.Subject.doAs(Subject.java:422)at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)at
 org.apache.hadoop.ipc.Server$Handler.run(Server.java:2486) 
(state=,code=0){quote}

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #2142: [CARBONDATA-2316] [WIP] Executor task is failed but ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2142
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3630/



---

[GitHub] carbondata issue #2142: [CARBONDATA-2316] [WIP] Executor task is failed but ...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2142
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4852/



---

[GitHub] carbondata issue #2141: [CARBONDATA-2313] Added comment for SDK writer API m...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2141
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4851/



---

[GitHub] carbondata issue #2141: [CARBONDATA-2313] Added comment for SDK writer API m...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2141
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4320/



---

[GitHub] carbondata issue #2141: [CARBONDATA-2313] Added comment for SDK writer API m...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2141
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3629/



---

[GitHub] carbondata pull request #2142: [CARBONDATA-2316] [WIP] Executor task is fail...

[rahulforallp]

GitHub user rahulforallp opened a pull request:

https://github.com/apache/carbondata/pull/2142

[CARBONDATA-2316] [WIP] Executor task is failed but UI shows success issue 
is fixed …

 - [ ] Any interfaces changed? 
 
 - [ ] Any backward compatibility impacted? No
 
 - [ ] Document update required? No

 - [ ] Testing done
   
 - [ ] For large changes, please consider breaking it into sub-tasks under 
an umbrella JIRA. 



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rahulforallp/incubator-carbondata 
compact_task_faail_issue

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2142.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2142


commit 1ac4d350c4b405b0ff65cb04909b02e3a43d7737
Author: rahulforallp 
Date:   2018-04-06T08:46:06Z

[CARBONDATA-2316] Executor task is failed but UI shows success issue fixed




---

[GitHub] carbondata issue #2139: [CARBONDATA-2267] [Presto] Support Reading CarbonDat...

[geetikagupta16]

Github user geetikagupta16 commented on the issue:

https://github.com/apache/carbondata/pull/2139
  
In this PR we are handling the scenario of querying data to standard hive 
partitioned table in carbondata. 

Earlier we were iterating over all the partitions to get the data but now 
as we have identified the required partitions, so the query will be performed 
only on the required partition.
Firstly we are identifying the partition columns through domain constraints 
in presto and after that we are adding the Partition Spec to the configuration 
object to query partition.



---

[GitHub] carbondata pull request #2141: Added comment for SDK writer API methods

[ajantha-bhat]

GitHub user ajantha-bhat opened a pull request:

https://github.com/apache/carbondata/pull/2141

Added comment for SDK writer API methods

Be sure to do all of the following checklist to help us incorporate 
your contribution quickly and easily:

 - [ ] Any interfaces changed? NO
 
 - [ ] Any backward compatibility impacted? NO
 
 - [ ] Document update required? NO

 - [ ] Testing done. NA
   




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ajantha-bhat/carbondata unmanaged_table

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/carbondata/pull/2141.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2141


commit 49970829d56b91f2a5ca6e46f3957b248cf9ccde
Author: ajantha-bhat 
Date:   2018-04-06T08:40:38Z

Added comment for SDK writer API methods




---

[jira] [Created] (CARBONDATA-2316) Even though one of the Compaction task failed at executor. All the executor task is showing success in UI and Job fails from driver.

[Rahul Kumar (JIRA)]

Rahul Kumar created CARBONDATA-2316:
---

 Summary: Even though one of the Compaction task failed at 
executor. All the executor task is showing success in UI and Job fails from 
driver.
 Key: CARBONDATA-2316
 URL: https://issues.apache.org/jira/browse/CARBONDATA-2316
 Project: CarbonData
  Issue Type: Improvement
Reporter: Rahul Kumar
Assignee: Rahul Kumar


*At UI:* 

*!http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=9wrYLnPvC4cVkbqrwtlHFbeC2%2fnVXKrZXwY7%2bglgnCQfywSBPcodImAcj29S40n0M8Rx%2bHUEMVQ%3d!*

*At Executor :* 

!http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=9wrYLnPvC4cVkbqrwtlHFbeC2%2fnVXKrZXwY7%2bglgnCQfywSBPcodInHgG%2fMfoX2avMU6b0kZ9Jo%3d!

 

*At driver* 

!http://dts.huawei.com/net/dts/fckeditor/download.ashx?Path=9wrYLnPvC4cVkbqrwtlHFbeC2%2fnVXKrZXwY7%2bglgnCQfywSBPcodIruNtISaQlvSdxWgQJLSrNA%3d!



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata pull request #2140: [CARBONDATA-2315] success and failure log fix...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/carbondata/pull/2140


---

[GitHub] carbondata issue #2140: [CARBONDATA-2315] success and failure log fixed when...

[manishgupta88]

Github user manishgupta88 commented on the issue:

https://github.com/apache/carbondata/pull/2140
  
LGTM


---

[jira] [Commented] (CARBONDATA-2282) presto carbon not support query partition table

[Liang Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/CARBONDATA-2282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16428065#comment-16428065
 ] 

Liang Chen commented on CARBONDATA-2282:


Just i tried:

1.create partition table as below in sparkshell

scala> carbon.sql("create table partition_table(id Int,vin String, phonenumber 
Long, country String, area String, salary Int) PARTITIONED BY (logdate Date) 
STORED BY 'carbondata' TBLPROPERTIES('SORT_COLUMNS'='id,vin')")

2.Can query successfully in presto :

presto:default> select * from partition_table;
 id | vin | phonenumber | country | area | salary | logdate
+--+-+-+--++
 6 | A42258434831 | 125371346 | China | Asia | 10005 | 2016-12-13
 7 | A42158475831 | 125371347 | UK | OutSpace | 10006 | 2016-12-13
 8 | A41158494830 | 225371348 | China | Asia | 10007 | 2016-12-13
 12 | A42151477823 | 425371312 | China | Asia | 10011 | 2016-12-13
 14 | A42258434835 | 525371314 | China | Asia | 10013 | 2016-12-13
 20 | A42151477824 | 225371320 | China | Asia | 10019 | 2016-12-13
 22 | A42258434837 | 25371322 | China | Asia | 10021 | 2016-12-13
 24 | A41158494839 | 625371324 | China | Asia | 10023 | 2016-12-13
 15 | A42158475836 | 625371315 | UK | OutSpace | 10014 | 2014-05-15
 16 | A41158494838 | 525371316 | China | Asia | 10015 | 2014-05-15
 18 | A42158473832 | 325371318 | China | Asia | 10017 | 2014-05-15
 19 | A42152474834 | 225371319 | US | America | 10018 | 2014-05-15
 21 | A42158474137 | 325371321 | Japan | Asia | 10020 | 2014-05-15
 23 | A42158475838 | 425371323 | UK | OutSpace | 10022 | 2014-05-15
 25 | A41158494840 | 626381324 | Good | OutSpace | 10024 | 2014-05-15
 26 | A41158494843 | 625378824 | NotGood | OutSpace | 10025 | 2014-05-15
 1 | A42158424831 | 125371341 | China | Asia | 1 | 2016-02-12
 2 | A42158473831 | 125371342 | China | Asia | 10001 | 2016-02-12
 3 | A42152474832 | 125371343 | US | America | 10002 | 2016-02-12
 4 | A42151477823 | 125371344 | China | OutSpace | 10003 | 2016-02-12
 5 | A42158474135 | 125371345 | Japan | OutSpace | 10004 | 2016-02-12
 9 | A42158424831 | 225371349 | China | OutSpace | 10008 | 2016-02-12
 10 | A42158473830 | 225371310 | China | Asia | 10009 | 2016-02-12
 11 | A42152474830 | 325371311 | US | America | 10010 | 2016-02-12
 13 | A42158474133 | 325371313 | Japan | Asia | 10012 | 2016-02-12
 17 | A42158424833 | 425371317 | China | Asia | 10016 | 2016-02-12
(26 rows)

> presto carbon not support query partition table
> ---
>
> Key: CARBONDATA-2282
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2282
> Project: CarbonData
>  Issue Type: Bug
>  Components: core, presto-integration
>Affects Versions: 1.3.0
>Reporter: zhangwei
>Assignee: anubhav tarar
>Priority: Major
> Fix For: 1.3.0
>
> Attachments: partitonToPrune.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[GitHub] carbondata issue #2140: [CARBONDATA-2315] success and failure log fixed when...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2140
  
Build Success with Spark 2.2.1, Please check CI 
http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3628/



---

[GitHub] carbondata issue #2140: [CARBONDATA-2315] success and failure log fixed when...

[ravipesala]

Github user ravipesala commented on the issue:

https://github.com/apache/carbondata/pull/2140
  
SDV Build Success , Please check CI 
http://144.76.159.231:8080/job/ApacheSDVTests/4319/



---

[GitHub] carbondata issue #2140: [CARBONDATA-2315] success and failure log fixed when...

[CarbonDataQA]

Github user CarbonDataQA commented on the issue:

https://github.com/apache/carbondata/pull/2140
  
Build Success with Spark 2.1.0, Please check CI 
http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4850/



---