[GitHub] carbondata pull request #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStor...
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207721977
--- Diff:
store/core/src/main/java/org/apache/carbondata/store/impl/MetaOperation.java ---
@@ -145,26 +152,61 @@ private void createTable(TableInfo tableInfo, boolean
ifNotExists) throws IOExce
}
}
- public void dropTable(TableIdentifier table) throws IOException {
-String tablePath = store.getTablePath(table.getTableName(),
table.getDatabaseName());
+ public void dropTable(TableIdentifier table) throws CarbonException {
+String tablePath = getTablePath(table.getTableName(),
table.getDatabaseName());
cache.remove(tablePath);
-FileFactory.deleteFile(tablePath);
+try {
+ FileFactory.deleteFile(tablePath);
+} catch (IOException e) {
+ throw new CarbonException(e);
+}
+ }
+
+ public TableInfo getTable(TableIdentifier table) throws CarbonException {
+return getTable(table, storeConf);
}
- public CarbonTable getTable(TableIdentifier table) throws IOException {
-String tablePath = store.getTablePath(table.getTableName(),
table.getDatabaseName());
+ public static TableInfo getTable(TableIdentifier table, StoreConf
storeConf)
+ throws CarbonException {
+String tablePath = getTablePath(table.getTableName(),
table.getDatabaseName(), storeConf);
if (cache.containsKey(tablePath)) {
return cache.get(tablePath);
} else {
- org.apache.carbondata.format.TableInfo formatTableInfo =
-
CarbonUtil.readSchemaFile(CarbonTablePath.getSchemaFilePath(tablePath));
+ org.apache.carbondata.format.TableInfo formatTableInfo = null;
+ try {
+formatTableInfo =
CarbonUtil.readSchemaFile(CarbonTablePath.getSchemaFilePath(tablePath));
+ } catch (IOException e) {
+throw new CarbonException(e);
+ }
SchemaConverter schemaConverter = new
ThriftWrapperSchemaConverterImpl();
TableInfo tableInfo = schemaConverter.fromExternalToWrapperTableInfo(
formatTableInfo, table.getDatabaseName(), table.getTableName(),
tablePath);
tableInfo.setTablePath(tablePath);
- CarbonTable carbonTable = CarbonTable.buildFromTableInfo(tableInfo);
- cache.put(tablePath, carbonTable);
- return carbonTable;
+ cache.put(tablePath, tableInfo);
+ return tableInfo;
}
}
-}
\ No newline at end of file
+
+ public List listTable() throws CarbonException {
+throw new UnsupportedOperationException();
+ }
+
+ public TableDescriptor getDescriptor(TableIdentifier table) throws
CarbonException {
+throw new UnsupportedOperationException();
+ }
+
+ public void alterTable(TableIdentifier table, TableDescriptor newTable)
throws CarbonException {
+throw new UnsupportedOperationException();
+ }
+
+ public String getTablePath(String tableName, String databaseName) {
+Objects.requireNonNull(tableName);
+Objects.requireNonNull(databaseName);
+return String.format("%s/%s", storeConf.storeLocation(), tableName);
+ }
+ public static String getTablePath(String tableName, String databaseName,
StoreConf storeConf) {
+Objects.requireNonNull(tableName);
+Objects.requireNonNull(databaseName);
+return String.format("%s/%s", storeConf.storeLocation(), tableName);
--- End diff --
must consider database name in path
---
[GitHub] carbondata pull request #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStor...
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207721786
--- Diff:
store/core/src/main/java/org/apache/carbondata/store/impl/master/RegistryServiceImpl.java
---
@@ -20,14 +20,14 @@
import java.io.IOException;
import org.apache.carbondata.common.annotations.InterfaceAudience;
-import org.apache.carbondata.store.impl.rpc.RegistryService;
-import org.apache.carbondata.store.impl.rpc.model.RegisterWorkerRequest;
-import org.apache.carbondata.store.impl.rpc.model.RegisterWorkerResponse;
+import org.apache.carbondata.store.impl.service.RegistryService;
+import
org.apache.carbondata.store.impl.service.model.RegisterWorkerRequest;
+import
org.apache.carbondata.store.impl.service.model.RegisterWorkerResponse;
import org.apache.hadoop.ipc.ProtocolSignature;
@InterfaceAudience.Internal
-class RegistryServiceImpl implements RegistryService {
+class egistryServiceImpl implements RegistryService {
--- End diff --
typo, pls rename
---
[GitHub] carbondata issue #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStore and I...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2589 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6167/ ---
[GitHub] carbondata issue #2589: [CARBONDATA-2825][CARBONDATA-2828] CarbonStore and I...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2589 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6509/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2589 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7784/ ---
[jira] [Created] (CARBONDATA-2828) InternalCarbonStore Interface
Jacky Li created CARBONDATA-2828: Summary: InternalCarbonStore Interface Key: CARBONDATA-2828 URL: https://issues.apache.org/jira/browse/CARBONDATA-2828 Project: CarbonData Issue Type: Sub-task Reporter: Jacky Li Assignee: Jacky Li As CarbonStore Interface is User level API, there should be Developer API for developer to integrate Distribute CarbonStore into compute engine like Spark, Presto, etc -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2589 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6166/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2589 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6165/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2589 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6508/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2589 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7783/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2589 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6507/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2589 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7782/ ---
[GitHub] carbondata pull request #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user ajithme commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2589#discussion_r207704316
--- Diff:
hadoop/src/main/java/org/apache/carbondata/hadoop/CarbonInputSplit.java ---
@@ -252,9 +262,24 @@ public Segment getSegment() {
if (dataMapWriterPathExists) {
dataMapWritePath = in.readUTF();
}
+boolean filePathExists = in.readBoolean();
+if (filePathExists) {
+ filePath = in.readUTF();
+} else {
+ filePath = super.getPath().toString();
+}
}
@Override public void write(DataOutput out) throws IOException {
+if (super.getPath() != null) {
+ super.write(out);
+} else {
+ // see HADOOP-13519, after Java deserialization, super.filePath is
+ // null, so write our filePath instead
+ Text.writeString(out, filePath);
+ out.writeLong(getStart());
+ out.writeLong(getLength());
+}
super.write(out);
--- End diff --
can delete this line else read will fail
---
[GitHub] carbondata issue #2590: [CARBONDATA-2750] Updated documentation on Local Dic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2590 Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6505/ ---
[GitHub] carbondata issue #2590: [CARBONDATA-2750] Updated documentation on Local Dic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2590 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7780/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2589 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6164/ ---
[GitHub] carbondata issue #2590: [CARBONDATA-2750] Updated documentation on Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2590 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6163/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2589 Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7781/ ---
[GitHub] carbondata issue #2590: [CARBONDATA-2750] Updated documentation on Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2590 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6162/ ---
[GitHub] carbondata issue #2589: [WIP][CARBONSTORE] Refactor CarbonStore API
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2589 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6506/ ---
[GitHub] carbondata issue #2590: [CARBONDATA-2750] Updated documentation on Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2590 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6161/ ---
[GitHub] carbondata issue #2590: [CARBONDATA-2750] Updated documentation on Local Dic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2590 Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7778/ ---
[GitHub] carbondata issue #2590: [CARBONDATA-2750] Updated documentation on Local Dic...
Github user CarbonDataQA commented on the issue: https://github.com/apache/carbondata/pull/2590 Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6502/ ---
[GitHub] carbondata issue #2590: [CARBONDATA-2750] Updated documentation on Local Dic...
Github user ravipesala commented on the issue: https://github.com/apache/carbondata/pull/2590 SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/6160/ ---
