[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #365: [IOTDB-189] Support compatibility of HDFS File and Java File
jt2594838 commented on a change in pull request #365: [IOTDB-189] Support
compatibility of HDFS File and Java File
URL: https://github.com/apache/incubator-iotdb/pull/365#discussion_r321575841
##
File path: server/src/main/java/org/apache/iotdb/db/conf/IoTDBConfig.java
##
@@ -793,4 +800,16 @@ public long getAllocateMemoryForChumkMetaDataCache() {
public void setAllocateMemoryForChumkMetaDataCache(long
allocateMemoryForChumkMetaDataCache) {
this.allocateMemoryForChumkMetaDataCache =
allocateMemoryForChumkMetaDataCache;
}
+
+ public FSType getStorageFs() {
+return storageFs;
+ }
+
+ public void setStorageFs(String storageFs) {
+if (storageFs.equals("HDFS")) {
+ this.storageFs = FSType.HDFS;
+} else if (storageFs.equals("LOCAL")) {
+ this.storageFs = FSType.LOCAL;
+}
Review comment:
You may use `FSType.valueOf()` for simplicity.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on issue #375: [IOTDB-191]Add the delete interface to Session API
qiaojialin commented on issue #375: [IOTDB-191]Add the delete interface to Session API URL: https://github.com/apache/incubator-iotdb/pull/375#issuecomment-528714078 After you finish the coding. Please add an IT for Session Interfaces. You can refer to the IoTDBCompleteIT. Use session api to set storage group, create timeseries, insert data and delete data... Then use JDBC API to query. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #375: [IOTDB-191]Add the delete interface to Session API
qiaojialin commented on a change in pull request #375: [IOTDB-191]Add the
delete interface to Session API
URL: https://github.com/apache/incubator-iotdb/pull/375#discussion_r321577700
##
File path: session/src/main/java/org/apache/iotdb/session/Session.java
##
@@ -173,6 +173,19 @@ public synchronized TSRPCResp insert(String deviceId,
long time, List me
}
}
+ public synchronized TSRPCResp delete(String deviceId, String measurementId,
long time) throws IoTDBSessionException {
Review comment:
The parameters could include a path and a time
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #375: [IOTDB-191]Add the delete interface to Session API
qiaojialin commented on a change in pull request #375: [IOTDB-191]Add the
delete interface to Session API
URL: https://github.com/apache/incubator-iotdb/pull/375#discussion_r321578282
##
File path: example/session/src/main/java/org/apache/iotdb/SessionExample.java
##
@@ -63,6 +64,14 @@ private static void insert() throws IoTDBSessionException {
}
}
+ private static void delete() throws IoTDBSessionException {
+String deviceId = "root.sg1.d1";
+String measurementId = "s1";
+for (long time = 0; time < 3; time++) {
+ session.delete(deviceId, measurementId, time);
Review comment:
The deleteTime in DeletePlan means deleting all data points in this
timeseries whose time is <= deleteTime. Therefore, onle one statement could do
the for(): session.delete(deviceId, measurementId, 2); You can add a
comment for the deleteTime field in the DeletePlan.
An easier interface: session.delete("root.sg1.d1.s1", 2);
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin opened a new pull request #376: delete storage group 30%
qiaojialin opened a new pull request #376: delete storage group 30% URL: https://github.com/apache/incubator-iotdb/pull/376 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321558739
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/SimpleExternalSortEngine.java
##
@@ -0,0 +1,160 @@
+ /**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ package org.apache.iotdb.db.query.externalsort;
+
+ import java.io.File;
+ import java.io.IOException;
+ import java.util.ArrayList;
+ import java.util.List;
+ import org.apache.commons.io.FileUtils;
+ import org.apache.iotdb.db.conf.IoTDBDescriptor;
+ import org.apache.iotdb.db.exception.StorageEngineFailureException;
+ import
org.apache.iotdb.db.query.externalsort.adapter.ByTimestampReaderAdapter;
+ import org.apache.iotdb.db.query.reader.IPointReader;
+ import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
+ import org.apache.iotdb.db.query.reader.chunkRelated.ChunkReaderWrap;
+
+
+ public class SimpleExternalSortEngine implements ExternalSortJobEngine {
+
+ private ExternalSortJobScheduler scheduler;
+
+ private String queryDir;
+ private int minExternalSortSourceCount;
+
+ private SimpleExternalSortEngine() {
+ queryDir = IoTDBDescriptor.getInstance().getConfig().getQueryDir();
+ minExternalSortSourceCount = IoTDBDescriptor.getInstance().getConfig()
+ .getExternalSortThreshold();
+ scheduler = ExternalSortJobScheduler.getInstance();
+
+ // create queryDir
+ try {
+ FileUtils.forceMkdir(new File(queryDir));
+ } catch (IOException e) {
+ throw new StorageEngineFailureException("create system directory
failed!");
+ }
+ }
+
+ // This class is used in test.
+ public SimpleExternalSortEngine(String queryDir, int
minExternalSortSourceCount) {
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321553510
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/serialize/impl/SimpleTimeValuePairSerializer.java
##
@@ -0,0 +1,66 @@
+/**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.db.query.externalsort.serialize.impl;
+
+import java.io.BufferedOutputStream;
+import java.io.File;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.ObjectOutputStream;
+import
org.apache.iotdb.db.query.externalsort.serialize.TimeValuePairSerializer;
+import org.apache.iotdb.db.utils.TimeValuePair;
+
+/**
+ * Serializer timeValuePair.
+ */
+public class SimpleTimeValuePairSerializer implements TimeValuePairSerializer {
Review comment:
deleted
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321553523
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/serialize/impl/SimpleTimeValuePairDeserializer.java
##
@@ -0,0 +1,84 @@
+/**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.db.query.externalsort.serialize.impl;
+
+import java.io.BufferedInputStream;
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.ObjectInputStream;
+import
org.apache.iotdb.db.query.externalsort.serialize.TimeValuePairDeserializer;
+import org.apache.iotdb.db.utils.TimeValuePair;
+
+/**
+ * Deserializer TimeValuePair.
+ */
+public class SimpleTimeValuePairDeserializer implements
TimeValuePairDeserializer {
Review comment:
deleted
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #375: [IOTDB-191]Add the delete interface to Session API
samperson1997 commented on a change in pull request #375: [IOTDB-191]Add the
delete interface to Session API
URL: https://github.com/apache/incubator-iotdb/pull/375#discussion_r321548734
##
File path: docs/Documentation-CHN/UserGuide/7-Session API/1-Session API.md
##
@@ -94,6 +94,14 @@ public class SessionExample {
}
}
Review comment:
In this md file you forget to add `delete()` in main method : )
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] jack870131 opened a new pull request #375: [IOTDB-191] Enrich Session interfaces
jack870131 opened a new pull request #375: [IOTDB-191] Enrich Session interfaces URL: https://github.com/apache/incubator-iotdb/pull/375 Corresponding JIRA issue: https://issues.apache.org/jira/browse/IOTDB-191 Add the delete interface for Session API. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321265807
##
File path:
server/src/test/java/org/apache/iotdb/db/query/externalsort/ExternalSortEngineTest.java
##
@@ -0,0 +1,174 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.query.externalsort;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Random;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.query.control.QueryResourceManager;
+import org.apache.iotdb.db.query.reader.IPointReader;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkReaderWrap;
+import org.apache.iotdb.db.query.reader.universal.FakedSeriesReader;
+import org.apache.iotdb.db.query.reader.universal.PriorityMergeReader;
+import org.apache.iotdb.db.utils.EnvironmentUtils;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class ExternalSortEngineTest {
+
+ private String baseDir = "externalSortTestTmp";
+ private long queryId = EnvironmentUtils.TEST_QUERY_JOB_ID;
+
+ @After
+ public void after() throws IOException, StorageEngineException {
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321250027
##
File path:
server/src/test/java/org/apache/iotdb/db/query/externalsort/SimpleTimeValuePairSerializerTest.java
##
@@ -0,0 +1,146 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.db.query.externalsort;
+
+import java.io.File;
+import java.io.IOException;
+import
org.apache.iotdb.db.query.externalsort.serialize.TimeValuePairDeserializer;
+import
org.apache.iotdb.db.query.externalsort.serialize.TimeValuePairSerializer;
+import
org.apache.iotdb.db.query.externalsort.serialize.impl.FixLengthTimeValuePairDeserializer;
+import
org.apache.iotdb.db.query.externalsort.serialize.impl.FixLengthTimeValuePairSerializer;
+import
org.apache.iotdb.db.query.externalsort.serialize.impl.SimpleTimeValuePairDeserializer;
+import
org.apache.iotdb.db.query.externalsort.serialize.impl.SimpleTimeValuePairSerializer;
+import org.apache.iotdb.db.utils.TimeValuePair;
+import org.apache.iotdb.db.utils.TsPrimitiveType;
+import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType;
+import org.apache.iotdb.tsfile.utils.Binary;
+import org.junit.Assert;
+import org.junit.Test;
+
+/**
+ * Created by zhangjinrui on 2018/1/20.
+ */
+public class SimpleTimeValuePairSerializerTest {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] samperson1997 closed pull request #362: [IOTDB-187] Enable to choose storage in local file system or HDFS
samperson1997 closed pull request #362: [IOTDB-187] Enable to choose storage in local file system or HDFS URL: https://github.com/apache/incubator-iotdb/pull/362 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321242110
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/ExternalSortJobPart.java
##
@@ -0,0 +1,44 @@
+/**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.db.query.externalsort;
+
+import java.io.IOException;
+import org.apache.iotdb.db.query.reader.IPointReader;
+
+
+public abstract class ExternalSortJobPart {
+
+ private ExternalSortJobPartType type;
+
+ public ExternalSortJobPart(ExternalSortJobPartType type) {
+this.type = type;
+ }
+
+ public abstract IPointReader executeWithGlobalTimeFilter() throws
IOException;
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321242185
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/MultiSourceExternalSortJobPart.java
##
@@ -0,0 +1,61 @@
+ /**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ package org.apache.iotdb.db.query.externalsort;
+
+ import java.io.IOException;
+ import java.util.ArrayList;
+ import java.util.List;
+ import org.apache.iotdb.db.query.reader.IPointReader;
+
+
+ public class MultiSourceExternalSortJobPart extends ExternalSortJobPart {
+
+ private String tmpFilePath;
+ private List source;
+ private long queryId;
+
+ public MultiSourceExternalSortJobPart(long queryId, String tmpFilePath,
+ List source) {
+ super(ExternalSortJobPartType.MULTIPLE_SOURCE);
+ this.source = source;
+ this.tmpFilePath = tmpFilePath;
+ this.queryId = queryId;
+ }
+
+ public MultiSourceExternalSortJobPart(long queryId, String tmpFilePath,
+ ExternalSortJobPart... externalSortJobParts) {
+ super(ExternalSortJobPartType.MULTIPLE_SOURCE);
+ source = new ArrayList<>();
+ for (ExternalSortJobPart externalSortJobPart : externalSortJobParts) {
+ source.add(externalSortJobPart);
+ }
+ this.tmpFilePath = tmpFilePath;
+ }
+
+ @Override
+ public IPointReader executeWithGlobalTimeFilter() throws IOException {
+ List prioritySeriesReaders = new ArrayList<>();
+ for (ExternalSortJobPart part : source) {
+ prioritySeriesReaders.add(part.executeWithGlobalTimeFilter());
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321242041
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/ExternalSortJob.java
##
@@ -0,0 +1,51 @@
+/**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.db.query.externalsort;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import org.apache.iotdb.db.query.reader.IPointReader;
+
+/**
+ * This class represents an external sort job. Every job will use a separated
directory.
+ */
+public class ExternalSortJob {
+
+ private long jobId;
+ private List partList;
+
+ public ExternalSortJob(long jobId, List partList) {
+this.jobId = jobId;
+this.partList = partList;
+ }
+
+ public List executeWithGlobalTimeFilter() throws IOException {
+List readers = new ArrayList<>();
+for (ExternalSortJobPart part : partList) {
+ readers.add(part.executeWithGlobalTimeFilter());
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321241397
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/SingleSourceExternalSortJobPart.java
##
@@ -0,0 +1,42 @@
+ /**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ package org.apache.iotdb.db.query.externalsort;
+
+ import java.io.IOException;
+ import org.apache.iotdb.db.query.reader.IPointReader;
+ import org.apache.iotdb.db.query.reader.chunkRelated.ChunkReaderWrap;
+
+
+ public class SingleSourceExternalSortJobPart extends ExternalSortJobPart {
+
+ private IPointReader timeValuePairReader;
+ private ChunkReaderWrap chunkReaderWrap;
+
+ public SingleSourceExternalSortJobPart(ChunkReaderWrap chunkReaderWrap) {
+ super(ExternalSortJobPartType.SINGLE_SOURCE);
+ this.chunkReaderWrap = chunkReaderWrap;
+ }
+
+ @Override
+ public IPointReader executeWithGlobalTimeFilter() throws IOException {
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321240221
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/SimpleExternalSortEngine.java
##
@@ -0,0 +1,160 @@
+ /**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ package org.apache.iotdb.db.query.externalsort;
+
+ import java.io.File;
+ import java.io.IOException;
+ import java.util.ArrayList;
+ import java.util.List;
+ import org.apache.commons.io.FileUtils;
+ import org.apache.iotdb.db.conf.IoTDBDescriptor;
+ import org.apache.iotdb.db.exception.StorageEngineFailureException;
+ import
org.apache.iotdb.db.query.externalsort.adapter.ByTimestampReaderAdapter;
+ import org.apache.iotdb.db.query.reader.IPointReader;
+ import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
+ import org.apache.iotdb.db.query.reader.chunkRelated.ChunkReaderWrap;
+
+
+ public class SimpleExternalSortEngine implements ExternalSortJobEngine {
+
+ private ExternalSortJobScheduler scheduler;
+
+ private String queryDir;
+ private int minExternalSortSourceCount;
+
+ private SimpleExternalSortEngine() {
+ queryDir = IoTDBDescriptor.getInstance().getConfig().getQueryDir();
+ minExternalSortSourceCount = IoTDBDescriptor.getInstance().getConfig()
+ .getExternalSortThreshold();
+ scheduler = ExternalSortJobScheduler.getInstance();
+
+ // create queryDir
+ try {
+ FileUtils.forceMkdir(new File(queryDir));
+ } catch (IOException e) {
+ throw new StorageEngineFailureException("create system directory
failed!");
+ }
+ }
+
+ // This class is used in test.
+ public SimpleExternalSortEngine(String queryDir, int
minExternalSortSourceCount) {
+ this.queryDir = queryDir;
+ this.minExternalSortSourceCount = minExternalSortSourceCount;
+ scheduler = ExternalSortJobScheduler.getInstance();
+ }
+
+ @Override
+ public List executeForIPointReader(long queryId,
+ List chunkReaderWraps)
+ throws IOException {
+ if (chunkReaderWraps.size() < minExternalSortSourceCount) {
+ return generateIPointReader(chunkReaderWraps, 0,
chunkReaderWraps.size());
+ }
+ ExternalSortJob job = createJob(queryId, chunkReaderWraps);
+ return job.executeWithGlobalTimeFilter();
+ }
+
+ @Override
+ public List executeForByTimestampReader(long queryId,
+ List chunkReaderWraps) throws IOException {
+ if (chunkReaderWraps.size() < minExternalSortSourceCount) {
+ return generateIReaderByTimestamp(chunkReaderWraps, 0,
chunkReaderWraps.size());
+ }
+ ExternalSortJob job = createJob(queryId, chunkReaderWraps);
+ return convert(job.executeWithGlobalTimeFilter());
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321239771
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/SimpleExternalSortEngine.java
##
@@ -0,0 +1,160 @@
+ /**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+ package org.apache.iotdb.db.query.externalsort;
+
+ import java.io.File;
+ import java.io.IOException;
+ import java.util.ArrayList;
+ import java.util.List;
+ import org.apache.commons.io.FileUtils;
+ import org.apache.iotdb.db.conf.IoTDBDescriptor;
+ import org.apache.iotdb.db.exception.StorageEngineFailureException;
+ import
org.apache.iotdb.db.query.externalsort.adapter.ByTimestampReaderAdapter;
+ import org.apache.iotdb.db.query.reader.IPointReader;
+ import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
+ import org.apache.iotdb.db.query.reader.chunkRelated.ChunkReaderWrap;
+
+
+ public class SimpleExternalSortEngine implements ExternalSortJobEngine {
+
+ private ExternalSortJobScheduler scheduler;
+
+ private String queryDir;
+ private int minExternalSortSourceCount;
+
+ private SimpleExternalSortEngine() {
+ queryDir = IoTDBDescriptor.getInstance().getConfig().getQueryDir();
+ minExternalSortSourceCount = IoTDBDescriptor.getInstance().getConfig()
+ .getExternalSortThreshold();
+ scheduler = ExternalSortJobScheduler.getInstance();
+
+ // create queryDir
+ try {
+ FileUtils.forceMkdir(new File(queryDir));
+ } catch (IOException e) {
+ throw new StorageEngineFailureException("create system directory
failed!");
+ }
+ }
+
+ // This class is used in test.
+ public SimpleExternalSortEngine(String queryDir, int
minExternalSortSourceCount) {
+ this.queryDir = queryDir;
+ this.minExternalSortSourceCount = minExternalSortSourceCount;
+ scheduler = ExternalSortJobScheduler.getInstance();
+ }
+
+ @Override
+ public List executeForIPointReader(long queryId,
+ List chunkReaderWraps)
+ throws IOException {
+ if (chunkReaderWraps.size() < minExternalSortSourceCount) {
+ return generateIPointReader(chunkReaderWraps, 0,
chunkReaderWraps.size());
+ }
+ ExternalSortJob job = createJob(queryId, chunkReaderWraps);
+ return job.executeWithGlobalTimeFilter();
Review comment:
ok
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External
sort
URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321235790
##
File path:
server/src/main/java/org/apache/iotdb/db/query/externalsort/ExternalSortJobEngine.java
##
@@ -0,0 +1,55 @@
+/**
+ * Copyright © 2019 Apache IoTDB(incubating) (d...@iotdb.apache.org)
+ *
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.iotdb.db.query.externalsort;
+
+import java.io.IOException;
+import java.util.List;
+import org.apache.iotdb.db.query.reader.IPointReader;
+import org.apache.iotdb.db.query.reader.IReaderByTimestamp;
+import org.apache.iotdb.db.query.reader.chunkRelated.ChunkReaderWrap;
+
+
+public interface ExternalSortJobEngine {
+
+ /**
+ * Receive a list of ChunkReaderWraps and judge whether it should be
processed using external
+ * sort. If needed, do the merge sort for all ChunkReaderWraps using specific
strategy.
+ *
+ * @param chunkReaderWraps A list include a set of ChunkReaderWraps
Review comment:
fixed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort
little-emotion commented on a change in pull request #336: [ IOTDB-160]External sort URL: https://github.com/apache/incubator-iotdb/pull/336#discussion_r321216367 ## File path: server/src/test/java/org/apache/iotdb/db/query/externalsort/SimpleTimeValuePairSerializerTest.java ## @@ -0,0 +1,146 @@ +/** + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.iotdb.db.query.externalsort; + +import java.io.File; +import java.io.IOException; +import org.apache.iotdb.db.query.externalsort.serialize.TimeValuePairDeserializer; +import org.apache.iotdb.db.query.externalsort.serialize.TimeValuePairSerializer; +import org.apache.iotdb.db.query.externalsort.serialize.impl.FixLengthTimeValuePairDeserializer; +import org.apache.iotdb.db.query.externalsort.serialize.impl.FixLengthTimeValuePairSerializer; +import org.apache.iotdb.db.query.externalsort.serialize.impl.SimpleTimeValuePairDeserializer; +import org.apache.iotdb.db.query.externalsort.serialize.impl.SimpleTimeValuePairSerializer; +import org.apache.iotdb.db.utils.TimeValuePair; +import org.apache.iotdb.db.utils.TsPrimitiveType; +import org.apache.iotdb.tsfile.file.metadata.enums.TSDataType; +import org.apache.iotdb.tsfile.utils.Binary; +import org.junit.Assert; +import org.junit.Test; + +/** + * Created by zhangjinrui on 2018/1/20. Review comment: This PR reuses some of the previous code, and I didn't delete the original author's information. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #365: [IOTDB-189] Support compatibility of HDFS File and Java File
qiaojialin commented on a change in pull request #365: [IOTDB-189] Support
compatibility of HDFS File and Java File
URL: https://github.com/apache/incubator-iotdb/pull/365#discussion_r321201107
##
File path:
tsfile/src/main/java/org/apache/iotdb/tsfile/fileSystem/IoTDBFileFactory.java
##
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.fileSystem;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+
+import java.io.File;
+import java.net.URI;
+
+public enum IoTDBFileFactory {
+
+ INSTANCE;
+
+ public File getIoTDBFile(String pathname) {
+if (TSFileConfig.storageFs.equals("HDFS")) {
+ return new HdfsFile(pathname);
+} else {
+ return new File(pathname);
+}
+ }
+
+ public File getIoTDBFile(String parent, String child) {
Review comment:
rename to getFile or newFile, we do not have IoTDBFile anymore
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #365: [IOTDB-189] Support compatibility of HDFS File and Java File
qiaojialin commented on a change in pull request #365: [IOTDB-189] Support
compatibility of HDFS File and Java File
URL: https://github.com/apache/incubator-iotdb/pull/365#discussion_r321201107
##
File path:
tsfile/src/main/java/org/apache/iotdb/tsfile/fileSystem/IoTDBFileFactory.java
##
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.fileSystem;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+
+import java.io.File;
+import java.net.URI;
+
+public enum IoTDBFileFactory {
+
+ INSTANCE;
+
+ public File getIoTDBFile(String pathname) {
+if (TSFileConfig.storageFs.equals("HDFS")) {
+ return new HdfsFile(pathname);
+} else {
+ return new File(pathname);
+}
+ }
+
+ public File getIoTDBFile(String parent, String child) {
Review comment:
rename to getFile or newFile
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #365: [IOTDB-189] Support compatibility of HDFS File and Java File
qiaojialin commented on a change in pull request #365: [IOTDB-189] Support
compatibility of HDFS File and Java File
URL: https://github.com/apache/incubator-iotdb/pull/365#discussion_r321196673
##
File path:
tsfile/src/main/java/org/apache/iotdb/tsfile/fileSystem/IoTDBFileFactory.java
##
@@ -0,0 +1,63 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.tsfile.fileSystem;
+
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+
+import java.io.File;
+import java.net.URI;
+
+public enum IoTDBFileFactory {
+
+ INSTANCE;
+
+ public File getIoTDBFile(String pathname) {
+if (TSFileConfig.storageFs.equals("HDFS")) {
Review comment:
use an enum
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #365: [IOTDB-189] Support compatibility of HDFS File and Java File
qiaojialin commented on a change in pull request #365: [IOTDB-189] Support compatibility of HDFS File and Java File URL: https://github.com/apache/incubator-iotdb/pull/365#discussion_r321196449 ## File path: server/src/assembly/resources/conf/tsfile-format.properties ## @@ -47,4 +47,7 @@ value_encoder=PLAIN # Compression configuration # Data compression method, TsFile supports UNCOMPRESSED or SNAPPY. Default value is UNCOMPRESSED which means no compression -compressor=UNCOMPRESSED \ No newline at end of file +compressor=UNCOMPRESSED + +# Storage file system. Currently, Tsfile supports to be storied in LOCAL file system or HDFS. +storage_fs=LOCAL Review comment: this parameter should be in the iotdb-engine.properties This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #366: [IOTDB-190] upgrade from antlr3 to antlr4
Ring-k commented on a change in pull request #366: [IOTDB-190] upgrade from
antlr3 to antlr4
URL: https://github.com/apache/incubator-iotdb/pull/366#discussion_r321159498
##
File path:
server/src/test/java/org/apache/iotdb/db/qp/plan/logicalPlan/TestListStatement.java
##
@@ -0,0 +1,117 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.qp.plan.logicalPlan;
+
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.qp.constant.SQLConstant;
+import org.apache.iotdb.db.qp.logical.RootOperator;
+import org.apache.iotdb.db.qp.logical.sys.AuthorOperator;
+import org.apache.iotdb.db.qp.strategy.LogicalGenerator;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.junit.Before;
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNull;
+
+public class TestListStatement {
Review comment:
Thanks for your suggestion. I renamed the test classes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #366: [IOTDB-190] upgrade from antlr3 to antlr4
Ring-k commented on a change in pull request #366: [IOTDB-190] upgrade from
antlr3 to antlr4
URL: https://github.com/apache/incubator-iotdb/pull/366#discussion_r321156306
##
File path: server/src/main/java/org/apache/iotdb/db/service/TSServiceImpl.java
##
@@ -636,7 +636,7 @@ public TSExecuteStatementResp
executeQueryStatement(TSExecuteStatementReq req) {
PhysicalPlan physicalPlan;
try {
physicalPlan = processor.parseSQLToPhysicalPlan(statement,
zoneIds.get());
-} catch (QueryProcessorException | ArgsErrorException |
MetadataErrorException e) {
+} catch (QueryProcessorException e) {
Review comment:
Thanks for your suggestion. I removed the unused packages.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] Ring-k commented on a change in pull request #366: [IOTDB-190] upgrade from antlr3 to antlr4
Ring-k commented on a change in pull request #366: [IOTDB-190] upgrade from
antlr3 to antlr4
URL: https://github.com/apache/incubator-iotdb/pull/366#discussion_r321155945
##
File path: server/src/main/antlr4/org/apache/iotdb/db/sql/parse/TSParser.g4
##
@@ -0,0 +1,535 @@
+//
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+//
+
+parser grammar TSParser;
+
+options
+{
+tokenVocab=TSLexer;
+}
+
+// starting rule
+statement
+ : execStatement (SEMICOLON)? EOF
+ ;
+
+nonNegativeInteger
+: PositiveInteger
+| UnsignedInteger
+;
+
+integer
+: NegativeInteger
+| nonNegativeInteger
+;
+
+// ATTENTION: DO NOT USE NAME 'float'!
+floatValue
+: PositiveFloat # x_positiveFloat
+| NegativeFloat # x_negativeInteger
+| PositiveInteger DOT # x_positiveIntegerDot
+| NegativeInteger DOT # x_negativeIntegerDot
+| UnsignedInteger DOT UnsignedInteger # x_unsignedIntegerDotUnsignedInteger
+| UnsignedInteger DOT # x_unsignedIntegerDot
+| DOT UnsignedInteger # x_dotUnsignedInteger
+| UnsignedInteger DoubleInScientificNotationSuffix #
x_unsignedIntegerDoubleInScientificNotationSuffix
+| DoubleInScientificNotationSuffix # x_doubleInScientificNotationSuffix
+;
+
+number
+: integer # integerString
+| floatValue # floatString
+| Boolean # booleanString
+;
+
+numberOrString // identifier is string or integer
+: Boolean | floatValue | identifier
+;
+
+numberOrStringWidely
+: number
+| StringLiteral
+;
+
+execStatement
+: authorStatement # exeAuthorStat
+| deleteStatement # exeDeleteStat
+| updateStatement # exeUpdateStat
+| insertStatement # exeInsertStat
+| queryStatement # exeQueryStat
+| metadataStatement # exeMetaStat
+| mergeStatement # exeMergeStat
+| indexStatement # exeIndexStat
+| quitStatement # exeQuitStat
+| listStatement # exeListStat
+;
+
+
+
+dateFormat
+: datetime=DATETIME #getDateTime
+| func=Identifier LPAREN RPAREN #getIdentifier
+;
+
+dateFormatWithNumber
+: dateFormat #getDateTimeFromDateFormat
+| integer #getDateTimeFromInteger
+;
+
+
+/*
+
+*
+metadata
+*
+
+*/
+
+
+metadataStatement
+: createTimeseries
+| setStorageGroup
+| deleteStorageGroup // TODO to implement
+| addAPropertyTree
+| addALabelProperty
+| deleteALabelFromPropertyTree
+| linkMetadataToPropertyTree
+| unlinkMetadataNodeFromPropertyTree
+| deleteTimeseries
+| showMetadata
+| describePath
+;
+
+describePath
+: KW_DESCRIBE prefixPath
+;
+
+showMetadata
+ : KW_SHOW KW_METADATA
+ ;
+
+createTimeseries
+ : KW_CREATE KW_TIMESERIES timeseries KW_WITH propertyClauses
+ ;
+
+timeseries
+ : KW_ROOT (DOT identifier)+
+ ;
+
+propertyClauses
+ : KW_DATATYPE EQUAL propertyName=identifier COMMA KW_ENCODING EQUAL
pv=propertyValue (COMMA KW_COMPRESSOR EQUAL compressor=propertyValue)? (COMMA
propertyClause)*
+ ;
+
+propertyClause
+ : propertyName=identifier EQUAL pv=propertyValue
+ ;
+
+propertyValue
+ : numberOrString
+ ;
+
+setStorageGroup
+ : KW_SET KW_STORAGE KW_GROUP KW_TO prefixPath
+ ;
+
+deleteStorageGroup
+ : KW_DELETE KW_STORAGE KW_GROUP prefixPath
+ ;
+
+addAPropertyTree
+ : KW_CREATE KW_PROPERTY property=identifier
+ ;
+
+addALabelProperty
+ : KW_ADD KW_LABEL label=identifier KW_TO KW_PROPERTY property=identifier
+ ;
+
+deleteALabelFromPropertyTree
+ : KW_DELETE KW_LABEL label=identifier KW_FROM KW_PROPERTY property=identifier
+ ;
+
+linkMetadataToPropertyTree
+ : KW_LINK prefixPath KW_TO propertyPath
+ ;
+
+
+propertyPath
+ : property=identifier DOT label=identifier
+ ;
+
+unlinkMetadataNodeFromPropertyTree
+ :KW_UNLINK prefixPath KW_FROM propertyPath
+ ;
+
+deleteTimeseries
+ : KW_DELETE KW_TIMESERIES prefixPath (COMMA prefixPath)*
+ ;
+
+
+/*
+
+*
+crud & author
+*
+
+*/
+mergeStatement
+:
+KW_MERGE
+;
+
+quitStatement
+:
+KW_QUIT
+;
+
+queryStatement
+ :
+ selectClause
+ whereClause?
+ specialClause?
+ ;
+
+specialClause
+:
+limitClause slimitClause?
+|slimitClause limitClause?
+
[GitHub] [incubator-iotdb] qiaojialin merged pull request #371: fix some small problems discovered when reading code
qiaojialin merged pull request #371: fix some small problems discovered when reading code URL: https://github.com/apache/incubator-iotdb/pull/371 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #303: transform old form to new form and use a parameter to control
qiaojialin commented on a change in pull request #303: transform old form to
new form and use a parameter to control
URL: https://github.com/apache/incubator-iotdb/pull/303#discussion_r321150077
##
File path: spark-tsfile/src/test/scala/org/apache/iotdb/tsfile/TSFileSuit.scala
##
@@ -214,4 +223,28 @@ class TSFileSuit extends FunSuite with BeforeAndAfterAll {
Assert.assertEquals(expected, df.schema)
}
+ test("testTransform1") {
+val df = spark.read.tsfile(tsfile1)
+df.createOrReplaceTempView("tsfile_table")
+val newDf = spark.sql("select * from tsfile_table " +
+ "where (`device_1.sensor_1`>0 or `device_1.sensor_2` < 22) and time < 4")
+Assert.assertEquals(3, newDf.count())
+ }
+
+ test("testTransform2") {
+val df = spark.read.tsfile(tsfile1, true)
+df.createOrReplaceTempView("tsfile_table")
+val newDf = spark.sql("select * from tsfile_table " +
+ "where `device_name` = 'device_1' and (`sensor_1`>0 or `sensor_2` < 22)
and time < 4")
+Assert.assertEquals(3, newDf.count())
+ }
+
+ test("testTransform3") {
Review comment:
You can try dfs.block.size, and blockSize should be a multiple of
io.bytes.per.checksum (512).
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin removed a comment on issue #370: [IOTDB-199]Add a log tool
qiaojialin removed a comment on issue #370: [IOTDB-199]Add a log tool URL: https://github.com/apache/incubator-iotdb/pull/370#issuecomment-528267997 this is a big pr, it's better to leave more time for reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on issue #370: [IOTDB-199]Add a log tool
qiaojialin commented on issue #370: [IOTDB-199]Add a log tool URL: https://github.com/apache/incubator-iotdb/pull/370#issuecomment-528267997 this is a big pr, it's better to leave more time for reviewing... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #370: [IOTDB-199]Add a log tool
qiaojialin commented on a change in pull request #370: [IOTDB-199]Add a log tool URL: https://github.com/apache/incubator-iotdb/pull/370#discussion_r321145105 ## File path: docs/Documentation-CHN/UserGuide/9-System Tools/3-Log Visualizer.md ## @@ -0,0 +1,146 @@ + + + +## 日志可视化工具LogVisualizer + + + +- [日志可视化工具LogVisualizer](#日志可视化工具LogVisualizer) +- [介绍](#介绍) +- [准备工作](#准备工作) +- [使用方法(图形界面)](#使用方法（图形界面）) + - [启动图形界面](#启动图形界面) + - [选择日志文件](#选择日志文件) + - [选择解析器配置文件](#选择解析器配置文件) + - [载入日志](#载入日志) + - [新建可视化计划](#新建可视化计划) + - [编辑可视化计划](#编辑可视化计划) + - [删除可视化计划](#删除可视化计划) + - [执行可视化计划](#执行可视化计划) +- [使用方法（命令行）](#使用方法(命令行)) + + + + +# 介绍 Review comment: 牛逼啊 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #365: [IOTDB-189] Support compatibility of HDFS File and Java File
samperson1997 commented on a change in pull request #365: [IOTDB-189] Support
compatibility of HDFS File and Java File
URL: https://github.com/apache/incubator-iotdb/pull/365#discussion_r321120563
##
File path:
tsfile/src/main/java/org/apache/iotdb/tsfile/fileSystem/IoTDBFile.java
##
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.fileSystem;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.*;
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+
+public class IoTDBFile extends File {
Review comment:
Thanks for such a good suggestion! I'd like to refactor codes as suggested.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] samperson1997 commented on a change in pull request #365: [IOTDB-189] Support compatibility of HDFS File and Java File
samperson1997 commented on a change in pull request #365: [IOTDB-189] Support
compatibility of HDFS File and Java File
URL: https://github.com/apache/incubator-iotdb/pull/365#discussion_r321120274
##
File path:
tsfile/src/main/java/org/apache/iotdb/tsfile/fileSystem/IoTDBFile.java
##
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.fileSystem;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.*;
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+
+public class IoTDBFile extends File {
+
+ private Path hdfsPath;
+
+ public IoTDBFile(String pathname) {
+super(pathname);
+if (TSFileConfig.isHdfsStorage) {
+ hdfsPath = new Path(pathname);
+}
+ }
+
+ public IoTDBFile(String parent, String child) {
+super(parent, child);
+ }
+
+ public IoTDBFile(File parent, String child) {
+super(parent, child);
+ }
+
+ public IoTDBFile(URI uri) {
+super(uri);
+ }
+
+ @Override
+ public String getAbsolutePath() {
+if (TSFileConfig.isHdfsStorage) {
+ return hdfsPath.toUri().toString();
+}
+return super.getAbsolutePath();
+ }
+
+ @Override
+ public long length() {
+if (TSFileConfig.isHdfsStorage) {
+ try {
+FileSystem fs = hdfsPath.getFileSystem(new Configuration());
+return fs.getFileStatus(hdfsPath).getLen();
+ } catch (IOException e) {
+e.printStackTrace();
+ }
+}
+return super.length();
+ }
+
+ @Override
+ public boolean exists() {
+if (TSFileConfig.isHdfsStorage) {
+ try {
+FileSystem fs = hdfsPath.getFileSystem(new Configuration());
+return fs.exists(hdfsPath);
+ } catch (IOException e) {
+e.printStackTrace();
+ }
+}
+return super.exists();
+ }
+
+ public IoTDBFile[] listFiles() {
+if (TSFileConfig.isHdfsStorage) {
+ ArrayList files = new ArrayList<>();
+ try {
+FileSystem fs = hdfsPath.getFileSystem(new Configuration());
+RemoteIterator iterator = fs.listFiles(hdfsPath,
true);
+while (iterator.hasNext()) {
+ LocatedFileStatus fileStatus = iterator.next();
+ Path fullPath = fileStatus.getPath();
+ files.add(new IoTDBFile(fullPath.toUri()));
+}
+return files.toArray(new IoTDBFile[files.size()]);
+ } catch (IOException e) {
+e.printStackTrace();
+ }
+}
+
+File[] rawFiles = super.listFiles();
+IoTDBFile[] files = new IoTDBFile[rawFiles.length];
+for (int i = 0; i < rawFiles.length; i++) {
+ files[i] = new IoTDBFile(rawFiles[i].getPath());
+}
+return files;
+ }
+
+ public IoTDBFile[] listFiles(FileFilter filter) {
+if (TSFileConfig.isHdfsStorage) {
+ ArrayList files = new ArrayList<>();
+ try {
+PathFilter pathFilter = new GlobFilter(filter.toString()); // TODO
Review comment:
`PathFilter` / `GlobFilter` in hadoop file system needs _RegExpr_ as param
to filter files, however every `FileFilter` in java file system needs to
implement interface `FileFilter`. I intend to fix this method to support hadoop
file in [next step](https://github.com/apache/incubator-iotdb/pull/362) since
this PR just encapsulated "IoTDBFile" to ensure java file system works well...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [incubator-iotdb] qiaojialin commented on a change in pull request #365: [IOTDB-189] Support compatibility of HDFS File and Java File
qiaojialin commented on a change in pull request #365: [IOTDB-189] Support
compatibility of HDFS File and Java File
URL: https://github.com/apache/incubator-iotdb/pull/365#discussion_r321117612
##
File path:
tsfile/src/main/java/org/apache/iotdb/tsfile/fileSystem/IoTDBFile.java
##
@@ -0,0 +1,183 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.iotdb.tsfile.fileSystem;
+
+import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.fs.*;
+import org.apache.iotdb.tsfile.common.conf.TSFileConfig;
+
+import java.io.File;
+import java.io.FileFilter;
+import java.io.IOException;
+import java.net.URI;
+import java.util.ArrayList;
+
+public class IoTDBFile extends File {
+
+ private Path hdfsPath;
+
+ public IoTDBFile(String pathname) {
+super(pathname);
+if (TSFileConfig.isHdfsStorage) {
+ hdfsPath = new Path(pathname);
+}
+ }
+
+ public IoTDBFile(String parent, String child) {
+super(parent, child);
+ }
+
+ public IoTDBFile(File parent, String child) {
+super(parent, child);
+ }
+
+ public IoTDBFile(URI uri) {
+super(uri);
+ }
+
+ @Override
+ public String getAbsolutePath() {
Review comment:
Consider another way: Use a IoTDBFileFactory to generate different files:
java.io.File and HDFSFile(extends java.io.File). Then, we only need to judge
the storage type when create and no need to judge in each method.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
