[GitHub] incubator-carbondata issue #603: Fix the bug of inverted index that store in...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/603 @chenliang613 this will not happen, becasue it is set to true in current code to avoid this problem, but inverted index is not configurable, so i raise this pr to make is can be configurable. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #603: Fix the bug of inverted index that store in...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/603 @ravipesala Could you pls help to check what's the meaning of the CI report taht can not build project? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #603: Fix the bug of inverted index that store in...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/603 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #603: Fix the bug of inverted index that s...
GitHub user Zhangshunyu opened a pull request:
https://github.com/apache/incubator-carbondata/pull/603
Fix the bug of inverted index that store inverted index in metadata by
using Encoding.INVERTED_INDEX.
## Why raise this pr?
This pr is the same to #222 , because pr 222 can not reopen, so open a new
pr.
Problem: In current code, inverted index in ddl info is not stored into
store, and when we restart the cluster, query might mismatch.
To fix problem 1, current code set always true to use inverted index, and
we can not configure inverted index now. We should fix this problem from its
root cause.
## How to solve?
Using the Encoding as the indentifier to check whether using inverted
index, this Encoding is in thrift format now, so we no need to modify the
thrift format.
Here it is the same to the query logic in
CompressedDimensionChunkFileBasedReader:
if
(CarbonUtil.hasEncoding(dimensionColumnChunk.get(blockIndex).getEncodingList(),
Encoding.INVERTED_INDEX)) {
invertedIndexes = CarbonUtil
.getUnCompressColumnIndex(dimensionColumnChunk.get(blockIndex).getRowIdPageLength(),
fileReader.readByteArray(filePath,
dimensionColumnChunk.get(blockIndex).getRowIdPageOffset(),
dimensionColumnChunk.get(blockIndex).getRowIdPageLength()), numberComressor);
// get the reverse index
invertedIndexesReverse = getInvertedReverseIndex(invertedIndexes);
}
it also use Encoding.INVERTED_INDEX to check whether one column is use
inverted index.
## How to test?
Pass all the test cases.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/Zhangshunyu/incubator-carbondata fix_index
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/603.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #603
commit 6c6202b10c9e3bf60830d2fb995b55e5109f59bd
Author: Zhangshunyu
Date: 2016-09-08T07:48:03Z
Save useInvertedIndex info into thrift store
commit 4d50a6a38901af619a56f4352ae0208be9775e50
Author: Zhangshunyu
Date: 2016-09-08T07:48:15Z
Save useInvertedIndex info into thrift store
commit a36bf38ffa7e766e09aa4513d4a4b96ef1218639
Author: Zhangshunyu
Date: 2016-09-08T09:46:12Z
Fix the judge of no_dic_col
commit e3ee06abd032affd098c1d31b6bbab41ce27b747
Author: Zhangshunyu
Date: 2016-09-08T10:04:20Z
add commont
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #595: [CARBONDATA-700]corrected doc for ddl opera...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/595 it seems that it is about no dictionary, not about no inverted index, right? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #591: [CARBONDATA-698]resolved bug for no_inverte...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/591 Hi, except this problem, there is another problem that inverted index info is not stored into thrift info, and when restart cluster, this info would lost. Could you pls check this pr? https://github.com/apache/incubator-carbondata/pull/222 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-676) Code clean
zhangshunyu created CARBONDATA-676: -- Summary: Code clean Key: CARBONDATA-676 URL: https://issues.apache.org/jira/browse/CARBONDATA-676 Project: CarbonData Issue Type: Improvement Reporter: zhangshunyu Assignee: zhangshunyu Priority: Minor To clean some code： Correct the spelling mistake Remove unused function Iterate the Array instead of transform it to List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #531: [CARBONDATA-612]throw exception when trying...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/531 I think it would be better to give an info about the difference instead of throw such an exception. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #567: Code clean
GitHub user Zhangshunyu opened a pull request: https://github.com/apache/incubator-carbondata/pull/567 Code clean To clean the code You can merge this pull request into a Git repository by running: $ git pull https://github.com/Zhangshunyu/incubator-carbondata codeclean Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/567.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #567 commit 5e94cffdc7ad528d602c4fb1a6114607a92c9a91 Author: Zhangshunyu Date: 2017-01-21T15:03:45Z Code clean --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (CARBONDATA-104) To support varchar datatype
[ https://issues.apache.org/jira/browse/CARBONDATA-104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshunyu closed CARBONDATA-104. -- Resolution: Invalid > To support varchar datatype > --- > > Key: CARBONDATA-104 > URL: https://issues.apache.org/jira/browse/CARBONDATA-104 > Project: CarbonData > Issue Type: New Feature > Reporter: zhangshunyu > Assignee: zhangshunyu >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (CARBONDATA-274) Use exist method in CarbonMetastoreCatalog to read/write thrift TableInfo
[ https://issues.apache.org/jira/browse/CARBONDATA-274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshunyu closed CARBONDATA-274. -- Resolution: Invalid > Use exist method in CarbonMetastoreCatalog to read/write thrift TableInfo > - > > Key: CARBONDATA-274 > URL: https://issues.apache.org/jira/browse/CARBONDATA-274 > Project: CarbonData > Issue Type: Improvement > Components: spark-integration >Affects Versions: 0.1.1-incubating > Reporter: zhangshunyu >Assignee: zhangshunyu >Priority: Minor > > Use exist method in CarbonMetastoreCatalog to read/write thrift TableInfo -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #222: [CARBONDATA-221] Fix the bug of inve...
Github user Zhangshunyu closed the pull request at: https://github.com/apache/incubator-carbondata/pull/222 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #254: [CARBONDATA-326] Create wrong table ...
Github user Zhangshunyu closed the pull request at: https://github.com/apache/incubator-carbondata/pull/254 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #538: Multiline is not used now
GitHub user Zhangshunyu opened a pull request: https://github.com/apache/incubator-carbondata/pull/538 Multiline is not used now The parameter "multiLine " in load is not used now, so remove it from code. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Zhangshunyu/incubator-carbondata multiline Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/538.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #538 commit 5a6cde80a25fa31fe0dc32d1517d2309b32b201f Author: Zhangshunyu Date: 2017-01-15T14:32:50Z Multiline is not used now --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #510: Document update for UID
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/510#discussion_r95283671
--- Diff: docs/DML-Operations-on-Carbon.md ---
@@ -1,183 +1,319 @@
-
-
-* [LOAD DATA](#LOAD DATA)
-* [SHOW SEGMENTS](#SHOW SEGMENTS)
-* [DELETE SEGMENT BY ID](#DELETE SEGMENT BY ID)
-* [DELETE SEGMENT BY DATE](#DELETE SEGMENT BY DATE)
-
-***
-
-# LOAD DATA
- This command loads the user data in raw format to the Carbon specific
data format store, this way Carbon provides good performance while querying the
data.Please visit [Data Management](Carbondata-Management.md) for more details
on LOAD
-
-### Syntax
-
- ```ruby
- LOAD DATA [LOCAL] INPATH 'folder_path' INTO TABLE [db_name.]table_name
- OPTIONS(property_name=property_value, ...)
- ```
-
-### Parameter Description
-
-| Parameter | Description | Optional |
-| - | -| |
-| folder_path | Path of raw csv data folder or file. | NO |
-| db_name | Database name, if it is not specified then it uses current
database. | YES |
-| table_name | The name of the table in provided database.| NO |
-| OPTIONS | Extra options provided to Load | YES |
-
-
-### Usage Guideline
-Following are the options that can be used in load data:
-- **DELIMITER:** Delimiters can be provided in the load command.
-
-``` ruby
-OPTIONS('DELIMITER'=',')
-```
-- **QUOTECHAR:** Quote Characters can be provided in the load command.
-
-```ruby
-OPTIONS('QUOTECHAR'='"')
-```
-- **COMMENTCHAR:** Comment Characters can be provided in the load command
if user want to comment lines.
-
-```ruby
-OPTIONS('COMMENTCHAR'='#')
-```
-- **FILEHEADER:** Headers can be provided in the LOAD DATA command if
headers are missing in the source files.
-
-```ruby
-OPTIONS('FILEHEADER'='column1,column2')
-```
-- **MULTILINE:** CSV with new line character in quotes.
-
-```ruby
-OPTIONS('MULTILINE'='true')
-```
-- **ESCAPECHAR:** Escape char can be provided if user want strict
validation of escape character on CSV.
-
-```ruby
-OPTIONS('ESCAPECHAR'='\')
-```
-- **COMPLEX_DELIMITER_LEVEL_1:** Split the complex type data column in a
row (eg., a$b$c --> Array = {a,b,c}).
-
-```ruby
-OPTIONS('COMPLEX_DELIMITER_LEVEL_1'='$')
-```
-- **COMPLEX_DELIMITER_LEVEL_2:** Split the complex type nested data column
in a row. Applies level_1 delimiter & applies level_2 based on complex data
type (eg., a:b$c:d --> Array> = {{a,b},{c,d}}).
-
-```ruby
-OPTIONS('COMPLEX_DELIMITER_LEVEL_2'=':')
-```
-- **ALL_DICTIONARY_PATH:** All dictionary files path.
-
-```ruby
-OPTIONS('ALL_DICTIONARY_PATH'='/opt/alldictionary/data.dictionary')
-```
-
-**Example:**
-
- ```ruby
- LOAD DATA local inpath '/opt/rawdata/data.csv' INTO table carbontable
- options('DELIMITER'=',', 'QUOTECHAR'='"',
'COMMENTCHAR'='#',
- 'FILEHEADER'='empno,empname,
- designation,doj,workgroupcategory,
-
workgroupcategoryname,deptno,deptname,projectcode,
-
projectjoindate,projectenddate,attendance,utilization,salary',
- 'MULTILINE'='true', 'ESCAPECHAR'='\',
- 'COMPLEX_DELIMITER_LEVEL_1'='$',
- 'COMPLEX_DELIMITER_LEVEL_2'=':',
-
'ALL_DICTIONARY_PATH'='/opt/alldictionary/data.dictionary'
- )
- ```
-
-***
-
-# SHOW SEGMENTS
-This command is to show the segments of carbon table to the user.
-
- ```ruby
- SHOW SEGMENTS FOR TABLE [db_name.]table_name LIMIT number_of_segments;
- ```
-
-### Parameter Description
-
-| Parameter | Description | Optional |
-| - | -| - |
-| db_name | Database name, if it is not specified then it uses current
database. | YES |
-| table_name | The name of the table in provided database.| NO |
-| number_of_segments | limit the output to this number. | YES |
-
[GitHub] incubator-carbondata issue #416: [CARBONDATA-518]Fix the bug of CarbonExampl...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/416 @chenliang613 Fixed, pls check. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #416: [CARBONDATA-518]Fix the bug of Carbo...
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/416#discussion_r91847903
--- Diff:
examples/spark/src/main/scala/org/apache/carbondata/examples/CarbonExample.scala
---
@@ -26,11 +28,15 @@ object CarbonExample {
def main(args: Array[String]) {
val cc = ExampleUtils.createCarbonContext("CarbonExample")
val testData = ExampleUtils.currentPath +
"/src/main/resources/data.csv"
-
+val rootPath = new File(this.getClass.getResource("/").getPath
--- End diff --
@chenliang613 Hi liang, now the path is not get from carboncontext, they
are got from carbonproperties, the code has been changed by this commit:
https://github.com/apache/incubator-carbondata/commit/f67ec0125247e98d7925866fd3062ee3ddf16e28https://github.com/apache/incubator-carbondata/commit/f67ec0125247e98d7925866fd3062ee3ddf16e28
And you can try carbon example, its not run now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[jira] [Created] (CARBONDATA-518) CarbonExample of spark moudle can not run as kettlehome and storepath shoug get form carbonproperties now
zhangshunyu created CARBONDATA-518: -- Summary: CarbonExample of spark moudle can not run as kettlehome and storepath shoug get form carbonproperties now Key: CARBONDATA-518 URL: https://issues.apache.org/jira/browse/CARBONDATA-518 Project: CarbonData Issue Type: Bug Components: examples Affects Versions: 0.2.0-incubating, 1.0.0-incubating Reporter: zhangshunyu Priority: Minor Fix For: 1.0.0-incubating -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata pull request #416: Fix the bug of CarbonExample in spar...
GitHub user Zhangshunyu opened a pull request: https://github.com/apache/incubator-carbondata/pull/416 Fix the bug of CarbonExample in spark1.5 moudle You can merge this pull request into a Git repository by running: $ git pull https://github.com/Zhangshunyu/incubator-carbondata carbonexample Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/416.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #416 commit 784384c1bf87a69da78f35cd87d5ca1c657c2eca Author: Zhangshunyu Date: 2016-12-10T06:08:09Z Fix the bug of CarbonExample in spark1.5 moudle --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #254: [CARBONDATA-326] Create wrong table using '...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/254 retest this please --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #222: [CARBONDATA-221] Fix the bug of inverted in...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/222 Rebased, Could any one pls check this pr? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #376: [WIP]TO support insert 1 line into c...
Github user Zhangshunyu closed the pull request at: https://github.com/apache/incubator-carbondata/pull/376 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #376: [WIP]TO support insert 1 line into c...
GitHub user Zhangshunyu opened a pull request: https://github.com/apache/incubator-carbondata/pull/376 [WIP]TO support insert 1 line into carbon table. WIP You can merge this pull request into a Git repository by running: $ git pull https://github.com/Zhangshunyu/incubator-carbondata insert1line Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/376.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #376 commit 43c363315ff980f02fe060ad0c25a4d028d463f7 Author: Zhangshunyu Date: 2016-12-01T08:20:04Z To support insert into one line commit 600fc29e24c9766a63f239e543ca23ead53c235e Author: Zhangshunyu Date: 2016-12-01T08:20:59Z To support insert into one line commit 3b0b0bc16ff1ffe0e59d338ec94d36a3317e4a1e Author: Zhangshunyu Date: 2016-12-01T08:24:18Z To support insert into one line --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #357: [CARBONDATA-451]Fix the bug of storePath.su...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/357 Added comments and rebased. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #357: [CARBONDATA-451]Fix the bug of storePath.su...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/357 @jackylk pls review this pr, thank you! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #357: [CARBONDATA-451]Fix the bug of store...
GitHub user Zhangshunyu opened a pull request: https://github.com/apache/incubator-carbondata/pull/357 [CARBONDATA-451]Fix the bug of storePath.substring on windows which lead to query can not run ## Why rasie this pr? To fix the bug that when query execute on windows can not run, this is becasue when table path is got on windows, the patch might contains '\' but current code substring the path not format it. ## How to solve? Using the formatted table path. ## How to test? Pass all the test cases. You can merge this pull request into a Git repository by running: $ git pull https://github.com/Zhangshunyu/incubator-carbondata path Alternatively you can review and apply these changes as the patch at: https://github.com/apache/incubator-carbondata/pull/357.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #357 commit 3443686e8df8791fbb92af55608ee90aa874faf8 Author: Zhangshunyu Date: 2016-11-26T10:22:22Z Fix the bug of storePath substring on windows --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (CARBONDATA-451) Can not run query on windows now
zhangshunyu created CARBONDATA-451: -- Summary: Can not run query on windows now Key: CARBONDATA-451 URL: https://issues.apache.org/jira/browse/CARBONDATA-451 Project: CarbonData Issue Type: Bug Components: core Reporter: zhangshunyu Assignee: zhangshunyu Fix For: 0.2.0-incubating As tablePath on windows has '/' and not replaced when substring, it would throw error when execute query. I have fixed this and will raise a pr. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] incubator-carbondata issue #330: [CARBONDATA-423] - Added Example to Load Da...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/330 LGTM --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata pull request #330: [CARBONDATA-423] - Added Example to ...
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/330#discussion_r88653600
--- Diff:
examples/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala
---
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples
+
+import org.apache.spark.sql.{DataFrame, SaveMode}
+import org.apache.spark.rdd.RDD
--- End diff --
pls change the order of line21 and line 22ï¼because CI would check scala
style strictly.
first rdd then sql
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #330: [CARBONDATA-423] - Added Example to ...
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/330#discussion_r88651390
--- Diff:
examples/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala
---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples
+
+import org.apache.spark.sql.{DataFrame, SaveMode}
+
--- End diff --
like GenerateDictionaryExample
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #330: [CARBONDATA-423] - Added Example to ...
Github user Zhangshunyu commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/330#discussion_r88651217
--- Diff:
examples/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala
---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples
+
+import org.apache.spark.sql.{DataFrame, SaveMode}
+
--- End diff --
pls remove this empty lineï¼no need here. only line23 is needed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #330: [CARBONDATA-423] - Added Example to ...
Github user Zhangshunyu commented on a diff in the pull request: https://github.com/apache/incubator-carbondata/pull/330#discussion_r88646359 --- Diff: examples/src/main/scala/org/apache/carbondata/examples/CaseClassDataFrameAPIExample.scala --- @@ -0,0 +1,48 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + *http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.carbondata.examples + +import org.apache.carbondata.examples.util.ExampleUtils --- End diff -- import order is not properï¼pls refer to DatasourceExample that fisrt spark lib and then carbon libï¼using an empty line to separate them. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] incubator-carbondata issue #254: [CARBONDATA-326] Create wrong table using '...
Github user Zhangshunyu commented on the issue: https://github.com/apache/incubator-carbondata/pull/254 Could anyone pls review this prï¼ @jackylk @ravipesala --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Closed] (CARBONDATA-411) test
[ https://issues.apache.org/jira/browse/CARBONDATA-411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshunyu closed CARBONDATA-411. -- Resolution: Invalid > test > > > Key: CARBONDATA-411 > URL: https://issues.apache.org/jira/browse/CARBONDATA-411 > Project: CarbonData > Issue Type: Improvement > Components: core > Reporter: zhangshunyu >Priority: Minor > Fix For: 0.2.0-incubating > > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-411) test
zhangshunyu created CARBONDATA-411: -- Summary: test Key: CARBONDATA-411 URL: https://issues.apache.org/jira/browse/CARBONDATA-411 Project: CarbonData Issue Type: Improvement Components: core Reporter: zhangshunyu Priority: Minor Fix For: 0.2.0-incubating -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CARBONDATA-332) Create successfully Database, tables and columns using carbon reserve keywords
[ https://issues.apache.org/jira/browse/CARBONDATA-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15599708#comment-15599708 ] zhangshunyu edited comment on CARBONDATA-332 at 10/23/16 1:40 PM: -- i think this is not a problem, since we dont need keep all behavior as hive. was (Author: zhangshunyu): i think this is not a problem, since we need not keep all behavior as hive. > Create successfully Database, tables and columns using carbon reserve keywords > -- > > Key: CARBONDATA-332 > URL: https://issues.apache.org/jira/browse/CARBONDATA-332 > Project: CarbonData > Issue Type: Bug >Reporter: Harmeet Singh > > Hey team, I am trying to create database, tables and columns with carbon > reserve keywords name and carbon allow us for creating. I am expecting an > error. In hive, we are facing an error. Following are the steps : > Step1: > 0: jdbc:hive2://127.0.0.1:1> create database double; > +-+--+ > | result | > +-+--+ > +-+--+ > No rows selected (6.225 seconds) > Step 2: > 0: jdbc:hive2://127.0.0.1:1> use double; > +-+--+ > | result | > +-+--+ > +-+--+ > No rows selected (0.104 seconds) > Step 3: > 0: jdbc:hive2://127.0.0.1:1> create table decimal(int int, string string) > stored by 'carbondata'; > +-+--+ > | Result | > +-+--+ > +-+--+ > No rows selected (2.372 seconds) > Step 4: > 0: jdbc:hive2://127.0.0.1:1> show tables; > ++--+--+ > | tableName | isTemporary | > ++--+--+ > | decimal| false| > ++--+--+ > 1 row selected (0.071 seconds) > Step 5: > 0: jdbc:hive2://127.0.0.1:1> desc decimal; > +---++--+--+ > | col_name | data_type | comment | > +---++--+--+ > | string| string | | > | int | bigint | | > +---++--+--+ > 2 rows selected (0.556 seconds) > Step 6: > 0: jdbc:hive2://127.0.0.1:1> load data inpath > 'hdfs://localhost:54310/home/harmeet/reservewords.csv' into table decimal; > +-+--+ > | Result | > +-+--+ > +-+--+ > No rows selected (0.863 seconds) > Step 7: > 0: jdbc:hive2://127.0.0.1:1> select * from decimal; > +-+--+--+ > | string | int | > +-+--+--+ > | james | 10 | > +-+--+--+ > 1 row selected (0.413 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CARBONDATA-332) Create successfully Database, tables and columns using carbon reserve keywords
[ https://issues.apache.org/jira/browse/CARBONDATA-332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15599708#comment-15599708 ] zhangshunyu commented on CARBONDATA-332: i think this is not a problem, since we need not keep all behavior as hive. > Create successfully Database, tables and columns using carbon reserve keywords > -- > > Key: CARBONDATA-332 > URL: https://issues.apache.org/jira/browse/CARBONDATA-332 > Project: CarbonData > Issue Type: Bug >Reporter: Harmeet Singh > > Hey team, I am trying to create database, tables and columns with carbon > reserve keywords name and carbon allow us for creating. I am expecting an > error. In hive, we are facing an error. Following are the steps : > Step1: > 0: jdbc:hive2://127.0.0.1:1> create database double; > +-+--+ > | result | > +-+--+ > +-+--+ > No rows selected (6.225 seconds) > Step 2: > 0: jdbc:hive2://127.0.0.1:1> use double; > +-+--+ > | result | > +-+--+ > +-+--+ > No rows selected (0.104 seconds) > Step 3: > 0: jdbc:hive2://127.0.0.1:1> create table decimal(int int, string string) > stored by 'carbondata'; > +-+--+ > | Result | > +-+--+ > +-+--+ > No rows selected (2.372 seconds) > Step 4: > 0: jdbc:hive2://127.0.0.1:1> show tables; > ++--+--+ > | tableName | isTemporary | > ++--+--+ > | decimal| false| > ++--+--+ > 1 row selected (0.071 seconds) > Step 5: > 0: jdbc:hive2://127.0.0.1:1> desc decimal; > +---++--+--+ > | col_name | data_type | comment | > +---++--+--+ > | string| string | | > | int | bigint | | > +---++--+--+ > 2 rows selected (0.556 seconds) > Step 6: > 0: jdbc:hive2://127.0.0.1:1> load data inpath > 'hdfs://localhost:54310/home/harmeet/reservewords.csv' into table decimal; > +-+--+ > | Result | > +-+--+ > +-+--+ > No rows selected (0.863 seconds) > Step 7: > 0: jdbc:hive2://127.0.0.1:1> select * from decimal; > +-+--+--+ > | string | int | > +-+--+--+ > | james | 10 | > +-+--+--+ > 1 row selected (0.413 seconds) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-311) Log the data size of blocklet during data load.
zhangshunyu created CARBONDATA-311: -- Summary: Log the data size of blocklet during data load. Key: CARBONDATA-311 URL: https://issues.apache.org/jira/browse/CARBONDATA-311 Project: CarbonData Issue Type: Improvement Affects Versions: 0.1.1-incubating Reporter: zhangshunyu Assignee: zhangshunyu Priority: Minor Fix For: 0.2.0-incubating Log the data size of blocklet during data load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (CARBONDATA-295) Abstract Snappy interface and seperate it from Compressor interface
[ https://issues.apache.org/jira/browse/CARBONDATA-295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshunyu closed CARBONDATA-295. -- Resolution: Invalid > Abstract Snappy interface and seperate it from Compressor interface > --- > > Key: CARBONDATA-295 > URL: https://issues.apache.org/jira/browse/CARBONDATA-295 > Project: CarbonData > Issue Type: Improvement > Components: data-load >Affects Versions: 0.1.1-incubating > Reporter: zhangshunyu >Assignee: zhangshunyu >Priority: Minor > Fix For: 0.2.0-incubating > > > Currently, we only have snappy compressor who extends form Compressor > interface, for future expansion, we need to abstract Snappy interface and > seperate it from Compressor interface, it means Compressor interface is the > parent of all compressors, and SnappyCompressor interface and the other > compressor's interface(or abstract class) should extends Compressor > interface, as to different data type for different compressor, it would > extend its own interface/abstract class. > for example: Compressor -> SnappyCompressor -> SnappyDoubleCompression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-295) Abstract Snappy interface and seperate it from Compressor interface
zhangshunyu created CARBONDATA-295: -- Summary: Abstract Snappy interface and seperate it from Compressor interface Key: CARBONDATA-295 URL: https://issues.apache.org/jira/browse/CARBONDATA-295 Project: CarbonData Issue Type: Improvement Components: data-load Affects Versions: 0.1.1-incubating Reporter: zhangshunyu Assignee: zhangshunyu Priority: Minor Fix For: 0.2.0-incubating Currently, we only have snappy compressor who extends form Compressor interface, for future expansion, we need to abstract Snappy interface and seperate it from Compressor interface, it means Compressor interface is the parent of all compressors, and SnappyCompressor interface and the other compressor's interface(or abstract class) should extends Compressor interface, as to different data type for different compressor, it would extend its own interface/abstract class. for example: Compressor -> SnappyCompressor -> SnappyDoubleCompression. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-293) Add scan_blocklet_num for query statistics
zhangshunyu created CARBONDATA-293: -- Summary: Add scan_blocklet_num for query statistics Key: CARBONDATA-293 URL: https://issues.apache.org/jira/browse/CARBONDATA-293 Project: CarbonData Issue Type: Improvement Components: data-query Affects Versions: 0.1.1-incubating Reporter: zhangshunyu Assignee: zhangshunyu Fix For: 0.2.0-incubating Add scan_blocklet_num for query statistics -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-289) Support MB/M for table block size and update the doc about this new feature.
zhangshunyu created CARBONDATA-289: -- Summary: Support MB/M for table block size and update the doc about this new feature. Key: CARBONDATA-289 URL: https://issues.apache.org/jira/browse/CARBONDATA-289 Project: CarbonData Issue Type: Bug Components: spark-integration Affects Versions: 0.1.0-incubating Reporter: zhangshunyu Assignee: zhangshunyu Priority: Minor Fix For: 0.2.0-incubating Support MB/M for table block size and update the doc about this new feature. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-280) when table properties is repeated it only set the last one
zhangshunyu created CARBONDATA-280:
--
Summary: when table properties is repeated it only set the last
one
Key: CARBONDATA-280
URL: https://issues.apache.org/jira/browse/CARBONDATA-280
Project: CarbonData
Issue Type: Bug
Components: sql
Affects Versions: 0.1.1-incubating
Reporter: zhangshunyu
Assignee: zhangshunyu
Priority: Minor
Fix For: 0.2.0-incubating
when table properties is repeated it only set the last one:
For example,
CREATE TABLE IF NOT EXISTS carbontable
(ID Int, date Timestamp, country String,
name String, phonetype String, serialname String, salary Int)
STORED BY 'carbondata'
TBLPROPERTIES('DICTIONARY_EXCLUDE'='country','DICTIONARY_INCLUDE'='ID',
'DICTIONARY_EXCLUDE'='phonetype', 'DICTIONARY_INCLUDE'='salary')
only salary is set to DICTIONARY_INCLUDE and only phonetype is set to
DICTIONARY_EXCLUDE.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Closed] (CARBONDATA-266) support delete all carbon tables under one database
[ https://issues.apache.org/jira/browse/CARBONDATA-266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshunyu closed CARBONDATA-266. -- Resolution: Invalid Assignee: (was: zhangshunyu) Fix Version/s: (was: 0.2.0-incubating) > support delete all carbon tables under one database > --- > > Key: CARBONDATA-266 > URL: https://issues.apache.org/jira/browse/CARBONDATA-266 > Project: CarbonData > Issue Type: New Feature > Components: spark-integration >Affects Versions: 0.1.0-incubating > Reporter: zhangshunyu >Priority: Minor > > support delete all carbon tables under one database, but nor delete the other > tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-274) Use exist method in CarbonMetastoreCatalog to read/write thrift TableInfo
zhangshunyu created CARBONDATA-274: -- Summary: Use exist method in CarbonMetastoreCatalog to read/write thrift TableInfo Key: CARBONDATA-274 URL: https://issues.apache.org/jira/browse/CARBONDATA-274 Project: CarbonData Issue Type: Improvement Components: spark-integration Affects Versions: 0.1.1-incubating Reporter: zhangshunyu Assignee: zhangshunyu Priority: Minor Fix For: 0.2.0-incubating Use exist method in CarbonMetastoreCatalog to read/write thrift TableInfo -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-267) Set block_size for table on table level
zhangshunyu created CARBONDATA-267: -- Summary: Set block_size for table on table level Key: CARBONDATA-267 URL: https://issues.apache.org/jira/browse/CARBONDATA-267 Project: CarbonData Issue Type: New Feature Affects Versions: 0.1.0-incubating Reporter: zhangshunyu Assignee: zhangshunyu Fix For: 0.2.0-incubating Set block_size for table on table level -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-266) support delete all carbon tables under one database
zhangshunyu created CARBONDATA-266: -- Summary: support delete all carbon tables under one database Key: CARBONDATA-266 URL: https://issues.apache.org/jira/browse/CARBONDATA-266 Project: CarbonData Issue Type: New Feature Components: spark-integration Affects Versions: 0.1.0-incubating Reporter: zhangshunyu Assignee: zhangshunyu Priority: Minor Fix For: 0.2.0-incubating support delete all carbon tables under one database, but nor delete the other tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (CARBONDATA-199) when subquery with sort and filter the result is empty
[ https://issues.apache.org/jira/browse/CARBONDATA-199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshunyu resolved CARBONDATA-199. Resolution: Fixed Assignee: zhangshunyu Fix Version/s: 0.2.0-incubating > when subquery with sort and filter the result is empty > -- > > Key: CARBONDATA-199 > URL: https://issues.apache.org/jira/browse/CARBONDATA-199 > Project: CarbonData > Issue Type: Bug > Reporter: zhangshunyu > Assignee: zhangshunyu > Fix For: 0.2.0-incubating > > > When the query has subquery with sort and filter, it can not return resullt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CARBONDATA-254) Code Inspection Optiminization
[ https://issues.apache.org/jira/browse/CARBONDATA-254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangshunyu updated CARBONDATA-254: --- Affects Version/s: 0.1.0-incubating Fix Version/s: 0.2.0-incubating > Code Inspection Optiminization > -- > > Key: CARBONDATA-254 > URL: https://issues.apache.org/jira/browse/CARBONDATA-254 > Project: CarbonData > Issue Type: Improvement >Affects Versions: 0.1.0-incubating > Reporter: zhangshunyu > Fix For: 0.2.0-incubating > > > Code Inspection Optiminization -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-255) keyword SEGMENT should be used instead of LOAD In data management dml because LOAD is not supported now
zhangshunyu created CARBONDATA-255: -- Summary: keyword SEGMENT should be used instead of LOAD In data management dml because LOAD is not supported now Key: CARBONDATA-255 URL: https://issues.apache.org/jira/browse/CARBONDATA-255 Project: CarbonData Issue Type: Bug Affects Versions: 0.1.0-incubating Reporter: zhangshunyu Fix For: 0.2.0-incubating keyword SEGMENT should be used instead of LOAD In data management dml because LOAD is not supported now -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-254) Code Inspection Optiminization
zhangshunyu created CARBONDATA-254: -- Summary: Code Inspection Optiminization Key: CARBONDATA-254 URL: https://issues.apache.org/jira/browse/CARBONDATA-254 Project: CarbonData Issue Type: Improvement Reporter: zhangshunyu Code Inspection Optiminization -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-231) Rename repeared table names in same test file and add drop tables.
zhangshunyu created CARBONDATA-231: -- Summary: Rename repeared table names in same test file and add drop tables. Key: CARBONDATA-231 URL: https://issues.apache.org/jira/browse/CARBONDATA-231 Project: CarbonData Issue Type: Improvement Reporter: zhangshunyu -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CARBONDATA-199) when subquery with sort and filter the result is empty
zhangshunyu created CARBONDATA-199: -- Summary: when subquery with sort and filter the result is empty Key: CARBONDATA-199 URL: https://issues.apache.org/jira/browse/CARBONDATA-199 Project: CarbonData Issue Type: Bug Reporter: zhangshunyu When the query has subquery with sort and filter, it can not return resullt -- This message was sent by Atlassian JIRA (v6.3.4#6332)
