[jira] [Created] (IOTDB-419) Rename first/last aggregate query to first_value/last_value
Lei Rui created IOTDB-419: - Summary: Rename first/last aggregate query to first_value/last_value Key: IOTDB-419 URL: https://issues.apache.org/jira/browse/IOTDB-419 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-417) [0.9.0] With German Locale it does NOT work as one test fails due to decimal comma vs decimal point
Lei Rui created IOTDB-417: - Summary: [0.9.0] With German Locale it does NOT work as one test fails due to decimal comma vs decimal point Key: IOTDB-417 URL: https://issues.apache.org/jira/browse/IOTDB-417 Project: Apache IoTDB Issue Type: Bug Affects Versions: 0.9.1 Reporter: Lei Rui Fix For: 0.10.0-SNAPSHOT Credit to Julian Feinauer. See [https://lists.apache.org/thread.html/rfef7f0035f28c3ee81ce32135e46e53774ce19b4af0bc06db426d4e3%40%3Cdev.iotdb.apache.org%3E] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-416) [0.9.1] ASF header format issues
Lei Rui created IOTDB-416:
-
Summary: [0.9.1] ASF header format issues
Key: IOTDB-416
URL: https://issues.apache.org/jira/browse/IOTDB-416
Project: Apache IoTDB
Issue Type: Improvement
Reporter: Lei Rui
see
[https://lists.apache.org/thread.html/209371ab5158fc61f2342115c1eef199f78555059dfb98bfdad70d72%40%3Cdev.iotdb.apache.org%3E]
Hi,
using `find **/src -type f -exec grep -i "copyright" {} \; -print | sort
-u`, many apache header format issues can be found..
e.g.,
1. showTTLPlan.java
2. all files for QueryMetricUI function...
3. All scala files are using /** so the blank space is two chars while it
is one char in all java files.
4. there are two blank spaces between "#" and content in all Python files
while it is one blank space in all sh files.
I do not vote -1 but at least we need to fix these issues in next release.
Best,
---
Xiangdong Huang
School of Software, Tsinghua University
黄向东
清华大学 软件学院
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IOTDB-364) Add TsFileDataDirViewer Tool
Lei Rui created IOTDB-364: - Summary: Add TsFileDataDirViewer Tool Key: IOTDB-364 URL: https://issues.apache.org/jira/browse/IOTDB-364 Project: Apache IoTDB Issue Type: New Feature Reporter: Lei Rui To have a quick overview of the data distribution in the data directories -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-359) [0.9.0] TsFile Sketch Tool prints wrong results when the tsfile is very large
Lei Rui created IOTDB-359: - Summary: [0.9.0] TsFile Sketch Tool prints wrong results when the tsfile is very large Key: IOTDB-359 URL: https://issues.apache.org/jira/browse/IOTDB-359 Project: Apache IoTDB Issue Type: Bug Reporter: Lei Rui I used the TsFile sketch tool on a TsFile sized 2.2G, and the printed result is wrong for that the ChunkGroupMetaDatas are not printed in their address ascending order. I analyzed this bug and find that `int` is used when comparing the address of ChunkGroupMetaDatas. However, the address should be long data type. I'll fix this bug soon :) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (IOTDB-150) Try some other web fonts
[ https://issues.apache.org/jira/browse/IOTDB-150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei Rui closed IOTDB-150. - Resolution: Won't Do > Try some other web fonts > > > Key: IOTDB-150 > URL: https://issues.apache.org/jira/browse/IOTDB-150 > Project: Apache IoTDB > Issue Type: Wish >Reporter: Lei Rui >Priority: Trivial > > My suggestion is that the website of > [IoTDB|[https://iotdb.apache.org|https://iotdb.apache.org/#/]] will look more > professional with some other web fonts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-346) StorageGroupProcessor.sequenceFileList is ordered by fileName rather than dataTime
Lei Rui created IOTDB-346:
-
Summary: StorageGroupProcessor.sequenceFileList is ordered by
fileName rather than dataTime
Key: IOTDB-346
URL: https://issues.apache.org/jira/browse/IOTDB-346
Project: Apache IoTDB
Issue Type: Bug
Reporter: Lei Rui
`StorageGroupProcessor.sequenceFileList` is ordered by fileName rather than by
time of data, as reflected in the `StorageGroupProcessor.getAllFiles` method
code:
{code:java}
tsFiles.sort(this::compareFileName);
{code}
I use the following examples to expose the bug when the order of fileName is
inconsistent with that of dataTime.
First, for preparation, I created three tsfiles using the following sql:
{code:java}
SET STORAGE GROUP TO root.ln.wf01.wt01
CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN, ENCODING=PLAIN
CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=DOUBLE,
ENCODING=PLAIN
CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32, ENCODING=PLAIN
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(1,
1.1, false, 11)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(2,
2.2, true, 22)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(3,
3.3, false, 33)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(4,
4.4, false, 44)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(5,
5.5, false, 55)
flush
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(100, 100.1, false, 110)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(150, 200.2, true, 220)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(200, 300.3, false, 330)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(250, 400.4, false, 440)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(300, 500.5, false, 550)
flush
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(10, 10.1, false, 110)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(20, 20.2, true, 220)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(30, 30.3, false, 330)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(40, 40.4, false, 440)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(50, 50.5, false, 550)
flush
{code}
The tsfiles created are organized in the following directory structure:
{code:java}
|data
|--sequence
|root.ln.wf01.wt01
|--1575813520203-101-0.tsfile
|--1575813520203-101-0.tsfile.resource
|--1575813520669-103-0.tsfile
|--1575813520669-103-0.tsfile.resource
|--unsequence
|root.ln.wf01.wt01
|--1575813521063-105-0.tsfile
|--1575813521063-105-0.tsfile.resource
{code}
||File Name||Data Time||
|(a) 1575813520203-101-0.tsfile|1-5|
|(c) 1575813521063-105-0.tsfile|10-50|
|(b) 1575813520669-103-0.tsfile|100-300|
Note how the order of fileName is inconsistent with that of dataTime.
By the way, if you look into the code, you will know how the file name is
generated:
{code:java}
System.currentTimeMillis() + IoTDBConstant.TSFILE_NAME_SEPARATOR +
versionController.nextVersion() + IoTDBConstant.TSFILE_NAME_SEPARATOR + "0" +
TSFILE_SUFFIX
{code}
Then, I loaded the three tsfiles into another brand new IoTDB. I did two
experiments with different loading orders each.
In the first experiment, the tsfiles were loaded in their data time order. That
is,
{code:java}
IoTDB> load 1575813520203-101-0.tsfile // tsfile (a), with data time 1-5
IoTDB> load 1575813521063-105-0.tsfile // tsfile (c), with data time 10-50
IoTDB> load 1575813520669-103-0.tsfile // tsfile (b), with data time
100-300{code}
After loading successfully, I did the following query in the same client window
and got the wrong result:
{code:java}
IoTDB> select * from root
+---+-+-+-+
| Time|root.ln.wf01.wt01.temperature|
root.ln.wf01.wt01.status| root.ln.wf01.wt01.hardware|
+---+-+-+-+
| 1970-01-01T08:00:00.001+08:00| 1.1|
false| 11|
| 1970-01-01T08:00:00.002+08:00| 2.2|
true| 22|
| 1970-01-01T08:00:00.003+08:00| 3.3|
false| 33|
| 1970-01-01T08:00:00.004+08:00| 4.4|
false| 44|
|
[jira] [Created] (IOTDB-335) Separate query execution of the same timeseries with different aggregate functions
Lei Rui created IOTDB-335: - Summary: Separate query execution of the same timeseries with different aggregate functions Key: IOTDB-335 URL: https://issues.apache.org/jira/browse/IOTDB-335 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui For example, select avg(s1),sum(s1) from root.sg.d0 select first_value(s1),first_time(s1) from root.sg.d0 select count(s1), sum(s1) from root.sg.d0 group by ([1,100],2ms) Probably the results of the same timeseries with different aggregate functions can be achieved in one query execution process. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-332) Bugs when using Chinese character in the timeseries path
Lei Rui created IOTDB-332:
-
Summary: Bugs when using Chinese character in the timeseries path
Key: IOTDB-332
URL: https://issues.apache.org/jira/browse/IOTDB-332
Project: Apache IoTDB
Issue Type: Bug
Reporter: Lei Rui
Note: At the beginning of every case, the data directory is deleted and the
server is restarted.
Case 1: insert into root."哈哈".b(timestamp,c) values(1,2)
{code:java}
IoTDB> insert into root."哈哈".b(timestamp,c) values(1,2)
Msg: Fail to execute insert into root."哈哈".b(timestamp,c) values(1,2) after
reconnecting. please check server status
{code}
Case 2: insert into root.a.b(timestamp,"哈哈") values(1,2)
{code:java}
IoTDB> insert into root.a.b(timestamp,"哈哈") values(1,2)
It costs 0.240sIoTDB> select * from root
+---+---+
| Time|root.a.b.哈哈|
+---+---+
| 1970-01-01T08:00:00.001+08:00| 2|
+---+---+
Total line number = 1
It costs 0.145s
IoTDB> flush
It costs 0.441sIoTDB> select * from root
.
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
0, 0, 0, 0, 0, 0, 0, 0, 0, 0]. Actual:26
It costs 1.587
{code}
Case 3: insert into root.a.b(timestamp,"哈哈") values(1,"哈哈")
{code:java}
IoTDB> insert into root.a.b(timestamp,"哈哈") values(1,"哈哈")
It costs 0.281s
IoTDB> select * from root
+---+---+
| Time|root.a.b.哈哈|
+---+---+
| 1970-01-01T08:00:00.001+08:00| 哈哈|
+---+---+
Total line number = 1
It costs 0.566s
IoTDB> flush
It costs 0.388s
IoTDB> select * from root
Msg: Cannot fetch result from server, because of network connection: {}
It costs 0.080s{code}
Case 4: insert into root.哈哈.b(timestamp,c) values(1,100)
{code:java}
IoTDB> insert into root.哈哈.b(timestamp,c) values(1,100)
Msg: Statement format is not right: Parsing error, statement [insert into
root.哈哈.b(timestamp,c) values(1,100)] failed when parsing AST tree to generate
logical operator. Detailed information: [line 1:17 character '哈' not supported
here. (Note that time format should be something like 1) number: eg.123456 2)
function: eg.now() 3) datatime: eg.-MM-dd HH:mm:ss, please check whether
inputting correct time format or referring to sql document)
line 1:18 character '哈' not supported here. (Note that time format should be
something like 1) number: eg.123456 2) function: eg.now() 3) datatime:
eg.-MM-dd HH:mm:ss, please check whether inputting correct time format or
referring to sql document)]
It costs 0.117s
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IOTDB-331) [0.9.0]A groupBy query bug when axisOrigin-startTimeOfWindow is an integral multiple of interval
Lei Rui created IOTDB-331:
-
Summary: [0.9.0]A groupBy query bug when
axisOrigin-startTimeOfWindow is an integral multiple of interval
Key: IOTDB-331
URL: https://issues.apache.org/jira/browse/IOTDB-331
Project: Apache IoTDB
Issue Type: Bug
Reporter: Lei Rui
{code:java}
SET STORAGE GROUP TO root.ln.wf01.wt01
CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN, ENCODING=PLAIN
CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=DOUBLE,
ENCODING=PLAIN
CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32, ENCODING=PLAIN
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(1,
1.1, false, 11)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(2,
2.2, true, 22)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(3,
3.3, false, 33)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(4,
4.4, false, 44)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(5,
5.5, false, 55)
flush
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(100, 100.1, false, 110)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(150, 200.2, true, 220)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(200, 300.3, false, 330)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(250, 400.4, false, 440)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(300, 500.5, false, 550)
flush
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(10, 10.1, false, 110)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(20, 20.2, true, 220)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(30, 30.3, false, 330)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(40, 40.4, false, 440)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(50, 50.5, false, 550)
flush
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(500, 100.1, false, 110)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(510, 200.2, true, 220)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(520, 300.3, false, 330)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(530, 400.4, false, 440)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(540, 500.5, false, 550)
flush
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(580, 100.1, false, 110)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(590, 200.2, true, 220)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(600, 300.3, false, 330)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(610, 400.4, false, 440)
INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware)
values(620, 500.5, false, 550)
select count(temperature), sum(temperature), avg(temperature) from
root.ln.wf01.wt01 where temperature > 3 GROUP BY (20ms, 25,[5,30], [35,37],
[50, 160], [310, 314])
{code}
Note that in the above groupBy query, 25-5=1*20.
The result of the above query is:
{code:java}
5,0,0.0,null // This is not right
5,3,35.8,11.932
25,1,30.3,30.3
50,1,50.5,50.5
65,0,0.0,null
85,1,100.1,100.1
105,0,0.0,null
125,0,0.0,null
145,1,200.2,200.2
310,0,0.0,null
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IOTDB-321) Lack the definition of time expression in 4-SQL Reference.md
Lei Rui created IOTDB-321: - Summary: Lack the definition of time expression in 4-SQL Reference.md Key: IOTDB-321 URL: https://issues.apache.org/jira/browse/IOTDB-321 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui In the "4-SQL Reference.md/Select Record Statement", the definition of TimeExpr is not updated to accomodate the time expression feature developed in [https://github.com/apache/incubator-iotdb/pull/462|https://github.com/apache/incubator-iotdb/pull/462/files]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-317) [0.8.1] flush data, execute a wrong aggregation query and the server fails
Lei Rui created IOTDB-317: - Summary: [0.8.1] flush data, execute a wrong aggregation query and the server fails Key: IOTDB-317 URL: https://issues.apache.org/jira/browse/IOTDB-317 Project: Apache IoTDB Issue Type: Bug Reporter: Lei Rui *tag release/0.8.1* I start a new server and run the following commands in a client: ``` SET STORAGE GROUP TO root.ln.wf01.wt01; CREATE TIMESERIES root.ln.wf01.wt01.status WITH DATATYPE=BOOLEAN, ENCODING=PLAIN CREATE TIMESERIES root.ln.wf01.wt01.temperature WITH DATATYPE=FLOAT, ENCODING=PLAIN CREATE TIMESERIES root.ln.wf01.wt01.hardware WITH DATATYPE=INT32, ENCODING=PLAIN INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(1, 1.1, false, 11) INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(2, 2.2, true, 22) INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(3, 3.3, false, 33 ) INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(4, 4.4, false, 44) INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(5, 5.5, false, 55) INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(6, 1.1, false, 11) INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(7, 2.2, true, 22) INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(8, 3.3, false, 33 ) INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(9, 4.4, false, 44) INSERT INTO root.ln.wf01.wt01(timestamp,temperature,status, hardware) values(10, 5.5, false, 55) select mean(temperature) from root.ln.wf01.wt01 *select abc(temperature) from root.ln.wf01.wt01* // Msg: Cannot get column abc(root.ln.wf01.wt01.temperature) data type // It costs 0.014s select mean(temperature) from root.ln.wf01.wt01 select count(temperature) from root.ln.wf01.wt01 *flush* select count(temperature) from root.ln.wf01.wt01 // Msg: null // It costs 0.011s select * from root // Msg: java.nio.channels.ClosedByInterruptException // It costs 0.015s ``` As can be seen, after executing a wrong aggregation query "select abc(temperature) from root.ln.wf01.wt01" and a flush command (you can also flush before executing this wrong aggregation query), the following two queries fail. I then close the client and restart a new client, queries still fail. I close the server and restart, queries work finally. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-316) The "mean" function in 4-SQL Reference.md should be "avg" for latest version
Lei Rui created IOTDB-316: - Summary: The "mean" function in 4-SQL Reference.md should be "avg" for latest version Key: IOTDB-316 URL: https://issues.apache.org/jira/browse/IOTDB-316 Project: Apache IoTDB Issue Type: Bug Reporter: Lei Rui In version 0.8.1, the function name is "mean“. In version 0.9.0 and master, the function name has changed to "avg", while the doc is not updated yet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-315) Can't start iotdb on windows
Lei Rui created IOTDB-315: - Summary: Can't start iotdb on windows Key: IOTDB-315 URL: https://issues.apache.org/jira/browse/IOTDB-315 Project: Apache IoTDB Issue Type: Bug Reporter: Lei Rui D:\new\incubator-iotdb\server\target\iotdb-server-0.9.0-SNAPSHOT\sbin>.\start-server.bat Starting IoTDB If you want to change this configuration, please check conf/iotdb-env.sh(Unix or OS X, if you use Windows, check conf/iotdb-env.bat). 此时不应有 (。 I use master branch. commit id=def4daf2addef0062fe04f698a825301ffa343bc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-306) count query is not that fast
Lei Rui created IOTDB-306: - Summary: count query is not that fast Key: IOTDB-306 URL: https://issues.apache.org/jira/browse/IOTDB-306 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui According to my test, *q1: select count(s_10) from root.group_0.d_17 where time>=2018-09-20T00:00:00+08:00 and time<=2018-09-20T23:59:59+08:00* ||Total time cost||readTsFileMetaData||readTsDeviceMetaData||readMemChunk|| |23,998|1,367|13,591|7,592| Unit: ms *q2: select s_10 from root.group_0.d_17 where time>=2018-09-20T00:00:00+08:00 and time<=2018-09-20T23:59:59+08:00* ||Total time cost||readTsFileMetaData||readTsDeviceMetaData||readMemChunk|| |27,783|31.2+2,068|134+13,880|14.9+9,587| Unit: ms (The "+" is because the step happens in both `createNewData` and `convertQueryDataSetByFetchSize` phases.) As is shown, the total time cost of q1 is just a little bit smaller than q2. The costs of the three major steps - `readTsFileMetaData`, `readTsDeviceMetaData`, and `readMemChunk` - are very close. The reason for this consequence is that the query execution process of count query reads chunk data from disk into memory anyway and in the best cases utilizes statistics (i,e., numOfPoints) in the pageHeader instead of reading page data. However, the time cost of reading page data from ChunkBuffer (see `ChunkReader.nextBatch`) is not that large, as it is performed in memory. Therefore, the execution process of count query overlaps mostly with that of without count query. And probably other aggregate queries have the similar results. A direction of performance improvement is to avoid `readMemChunk` whenever the statistics in the ChunkMetaData can be utilized. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-305) a sql to query a group of devices separately
Lei Rui created IOTDB-305:
-
Summary: a sql to query a group of devices separately
Key: IOTDB-305
URL: https://issues.apache.org/jira/browse/IOTDB-305
Project: Apache IoTDB
Issue Type: New Feature
Reporter: Lei Rui
First of all, if Bob knows exactly what devices to query (root.sg.d1 and
root.sg.d2 in this case) , he can write sqls for every device:
{code:java}
sql1: select * from root.sg.d1 where s1=1
sql2: select * from root.sg.d2 where s1=1{code}
However, when there are many devices or devices are not specified in advance,
Bob wants to query like:
{code:java}
select * from root.sg.d1,root.sg.d2 where s1=1 (a demo
conception)
/ select * from root.sg.* where s1=1 (a demo conception)
{code}
to return the result same as the concatenation of the results of the above two
queries (i.e., sql1 and sql2) while eliminating the trouble of writing two
separate sqls.
"group by device" sql can't satisfy this demand. It is because the where
condition of "group by device" is shared by all devices. For example,
{code:java}
select * from root.sg.* where s1=1 group by device
{code}
equals
{code:java}
select * from root.sg.d1 where root.sg.d1.s1=1 and root.sg.d2.s1=1
select * from root.sg.d2 where root.sg.d1.s1=1 and root.sg.d2.s1=1
{code}
Note the "and" in the where condition. The following example further
demonstrates the effect:
{code:java}
SET STORAGE GROUP TO root.ChangSha;
CREATE TIMESERIES root.ChangSha.A.aa WITH DATATYPE=INT64, ENCODING=RLE;
CREATE TIMESERIES root.ChangSha.A.ab WITH DATATYPE=INT64, ENCODING=RLE;
CREATE TIMESERIES root.ChangSha.A.ac WITH DATATYPE=INT64, ENCODING=RLE;
CREATE TIMESERIES root.ChangSha.B.aa WITH DATATYPE=INT64, ENCODING=RLE;
CREATE TIMESERIES root.ChangSha.B.ab WITH DATATYPE=INT64, ENCODING=RLE;
CREATE TIMESERIES root.ChangSha.B.ad WITH DATATYPE=INT64, ENCODING=RLE;
insert into root.ChangSha.A(timestamp,aa,ab,ac) values(1,1,1,1);
insert into root.ChangSha.B(timestamp,aa,ab,ad) values(2,1,2,2);
select * from root.ChangSha.A, root.ChangSha.B where aa=1 group by device{code}
The above select query equals
{code:java}
select * from root.ChangSha.A where root.ChangSha.A.aa=1 and
root.ChangSha.B.aa=1
select * from root.ChangSha.B where root.ChangSha.A.aa=1 and
root.ChangSha.B.aa=1{code}
This select query will get an empty result because there is no timestamp under
which both root.ChangSha.A.aa=1 and root.ChangSha.B.aa=1.
What Bob wants is:
{code:java}
select * from root.ChangSha.A, root.ChangSha.B where aa=1 (a
demo conception)
{code}
equals
{code:java}
select * from root.ChangSha.A where root.ChangSha.A.aa=1
select * from root.ChangSha.B where root.ChangSha.B.aa=1{code}
to get the result like:
||Time||Device||aa||ab||ac||ad||
|1970-01-01T08:00:00.001+08:00|root.ChangSha.A|1|1|1|null|
|1970-01-01T08:00:00.002+08:00|root.ChangSha.B|1|2|null|2|
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (IOTDB-302) Bug when executing "show child paths "
[ https://issues.apache.org/jira/browse/IOTDB-302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei Rui closed IOTDB-302. - Resolution: Not A Problem > Bug when executing "show child paths " > --- > > Key: IOTDB-302 > URL: https://issues.apache.org/jira/browse/IOTDB-302 > Project: Apache IoTDB > Issue Type: Bug >Reporter: Lei Rui >Priority: Major > Attachments: image-2019-11-11-23-55-59-103.png > > > !image-2019-11-11-23-55-59-103.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-302) Bug when executing "show child paths "
Lei Rui created IOTDB-302: - Summary: Bug when executing "show child paths " Key: IOTDB-302 URL: https://issues.apache.org/jira/browse/IOTDB-302 Project: Apache IoTDB Issue Type: Bug Reporter: Lei Rui Attachments: image-2019-11-11-23-55-59-103.png !image-2019-11-11-23-55-59-103.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-301) execute "count nodes root" in client gets "Msg:3"
Lei Rui created IOTDB-301: - Summary: execute "count nodes root" in client gets "Msg:3" Key: IOTDB-301 URL: https://issues.apache.org/jira/browse/IOTDB-301 Project: Apache IoTDB Issue Type: Bug Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (IOTDB-237) client fetches all data in spite of maxPrintRowCount
[ https://issues.apache.org/jira/browse/IOTDB-237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei Rui closed IOTDB-237. - Resolution: Duplicate > client fetches all data in spite of maxPrintRowCount > > > Key: IOTDB-237 > URL: https://issues.apache.org/jira/browse/IOTDB-237 > Project: Apache IoTDB > Issue Type: Improvement >Reporter: Lei Rui >Priority: Major > > In line 202-208 of AbstractClient.java, the client fetches all data from the > server, showing maxPrintRowCount rows. > In line 223, the client shows the total row count of the whole queried result. > Can the client only fetch data within the maxPrintRowCount constraint and > still be able to know the total result count? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-277) Lost docs about usage of import-csv.sh and export-csv.sh
Lei Rui created IOTDB-277: - Summary: Lost docs about usage of import-csv.sh and export-csv.sh Key: IOTDB-277 URL: https://issues.apache.org/jira/browse/IOTDB-277 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui As in the readme of the branch rel/0.8([https://github.com/apache/incubator-iotdb/tree/rel/0.8]), there are docs about usage of import-csv.sh and export-csv.sh. But now I can't find them in the docs of the master branch. Are they missing? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-263) Refactor IoTDBQueryResultSet of JDBC
Lei Rui created IOTDB-263:
-
Summary: Refactor IoTDBQueryResultSet of JDBC
Key: IOTDB-263
URL: https://issues.apache.org/jira/browse/IOTDB-263
Project: Apache IoTDB
Issue Type: Improvement
Reporter: Lei Rui
Proposed by Tian Jiang, there are at least two points that can be improved of
IoTDBQueryResultSet.
The first point: Batch structure may be preferable to List (as shown
in the following code). The data structure of List looks bloated
(chinese: 臃肿的).
{code:java}
// as in nextWithoutConstraints method
TSQueryDataSet tsQueryDataSet = resp.getQueryDataSet();
List records = Utils.convertRowRecords(tsQueryDataSet);
recordItr = records.iterator();
{code}
The batch structure may incur compact memory layout and possible beneficial
batch processing strategy. So I think this point is worth trying, especially
when we have already developed a new TSQueryDataSet with batch-like structure
(pr#455).
The second point: Some inefficient interface realizations. For example, as
shown in the following code, the getDouble method first calls the getString
method and then parses the string to get the double value. However, the double
value in the field could otherwise be directly got (if the data type of this
field is double).
{code:java}
public double getDouble(String columnName) throws SQLException {
return Double.parseDouble(getValueByName(columnName));
}
private String getValueByName(String columnName) throws SQLException {
checkRecord();
if (columnName.equals(TIMESTAMP_STR)) {
return String.valueOf(record.getTimestamp());
}
int tmp = columnInfoMap.get(columnName);
int i = 0;
for (Field field : record.getFields()) {
i++;
if (i == tmp - 1) {
return field.isNull() ? null : field.getStringValue();
}
}
return null;
}
{code}
This point is indeed a true problem. Although right now only the getString
method is actually in use, we do need to improve the other getXXX methods
(e.g., getDouble, getInt, getFloat) and inform our users of these interfaces.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IOTDB-252) bugs in the print-tsfile-resource-files tool
Lei Rui created IOTDB-252:
-
Summary: bugs in the print-tsfile-resource-files tool
Key: IOTDB-252
URL: https://issues.apache.org/jira/browse/IOTDB-252
Project: Apache IoTDB
Issue Type: Bug
Reporter: Lei Rui
- lack execution command in the print-tsfile-resource-files.bat
- unnecessary check of the existence of ${IOTDB_HOME}/data/wal in the
print-tsfile-resource-files.sh
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (IOTDB-244) wrong duplicated columns query result
Lei Rui created IOTDB-244:
-
Summary: wrong duplicated columns query result
Key: IOTDB-244
URL: https://issues.apache.org/jira/browse/IOTDB-244
Project: Apache IoTDB
Issue Type: Bug
Reporter: Lei Rui
I use the following sql to insert data,
{code:java}
SET STORAGE GROUP TO root.vehicle.d0
SET STORAGE GROUP TO root.vehicle.d1
CREATE TIMESERIES root.vehicle.d0.s0 WITH DATATYPE=INT32, ENCODING=RLE
CREATE TIMESERIES root.vehicle.d0.s1 WITH DATATYPE=INT64, ENCODING=RLE
CREATE TIMESERIES root.vehicle.d0.s2 WITH DATATYPE=FLOAT, ENCODING=RLE
CREATE TIMESERIES root.vehicle.d0.s3 WITH DATATYPE=TEXT, ENCODING=PLAIN
CREATE TIMESERIES root.vehicle.d0.s4 WITH DATATYPE=BOOLEAN, ENCODING=PLAIN
CREATE TIMESERIES root.vehicle.d1.s0 WITH DATATYPE=INT32, ENCODING=RLEinsert
into root.vehicle.d0(timestamp,s0) values(1,101)
insert into root.vehicle.d0(timestamp,s0) values(2,198)
insert into root.vehicle.d0(timestamp,s0) values(100,99)
insert into root.vehicle.d0(timestamp,s0) values(101,99)
insert into root.vehicle.d0(timestamp,s0) values(102,80)
insert into root.vehicle.d0(timestamp,s0) values(103,99)
insert into root.vehicle.d0(timestamp,s0) values(104,90)
insert into root.vehicle.d0(timestamp,s0) values(105,99)
insert into root.vehicle.d0(timestamp,s0) values(106,99)
insert into root.vehicle.d0(timestamp,s0) values(2,1)
insert into root.vehicle.d0(timestamp,s0) values(50,1)
insert into root.vehicle.d0(timestamp,s0) values(1000,2)insert into
root.vehicle.d0(timestamp,s1) values(1,1101)
insert into root.vehicle.d0(timestamp,s1) values(2,198)
insert into root.vehicle.d0(timestamp,s1) values(100,199)
insert into root.vehicle.d0(timestamp,s1) values(101,199)
insert into root.vehicle.d0(timestamp,s1) values(102,180)
insert into root.vehicle.d0(timestamp,s1) values(103,199)
insert into root.vehicle.d0(timestamp,s1) values(104,190)
insert into root.vehicle.d0(timestamp,s1) values(105,199)
insert into root.vehicle.d0(timestamp,s1) values(2,4)
insert into root.vehicle.d0(timestamp,s1) values(50,5)
insert into root.vehicle.d0(timestamp,s1) values(1000,5)insert into
root.vehicle.d0(timestamp,s2) values(1000,5)
insert into root.vehicle.d0(timestamp,s2) values(2,2.22)
insert into root.vehicle.d0(timestamp,s2) values(3,3.33)
insert into root.vehicle.d0(timestamp,s2) values(4,4.44)
insert into root.vehicle.d0(timestamp,s2) values(102,10.00)
insert into root.vehicle.d0(timestamp,s2) values(105,11.11)
insert into root.vehicle.d0(timestamp,s2) values(1000,1000.11)insert into
root.vehicle.d0(timestamp,s3) values(60,'a')
insert into root.vehicle.d0(timestamp,s3) values(70,'b')
insert into root.vehicle.d0(timestamp,s3) values(80,'c')
insert into root.vehicle.d0(timestamp,s3) values(101,'d')
insert into root.vehicle.d0(timestamp,s3) values(102,'f')insert into
root.vehicle.d1(timestamp,s0) values(1,999)
insert into root.vehicle.d1(timestamp,s0) values(1000,888)insert into
root.vehicle.d0(timestamp,s1) values(2000-01-01T08:00:00+08:00, 100)
insert into root.vehicle.d0(timestamp,s3) values(2000-01-01T08:00:00+08:00,
'good')insert into root.vehicle.d0(timestamp,s4) values(100, false)
insert into root.vehicle.d0(timestamp,s4) values(100, true)
{code}
and then I query as follows:
{code:java}
IoTDB> select s0,s0,s0,s1 from root.vehicle.d0
+---+--+--+--+--+
|
Time|root.vehicle.d0.s0|root.vehicle.d0.s0|root.vehicle.d0.s0|root.vehicle.d0.s1|
+---+--+--+--+--+
| 1970-01-01T08:00:00.001+08:00| 101| 101|
101| 101|
| 1970-01-01T08:00:00.002+08:00| 1| 1|
1| 1|
| 1970-01-01T08:00:00.050+08:00| 1| 1|
1| 1|
| 1970-01-01T08:00:00.100+08:00|99|99|
99|99|
| 1970-01-01T08:00:00.101+08:00|99|99|
99|99|
| 1970-01-01T08:00:00.102+08:00|80|80|
80|80|
| 1970-01-01T08:00:00.103+08:00|99|99|
99|99|
| 1970-01-01T08:00:00.104+08:00|90|90|
90|90|
| 1970-01-01T08:00:00.105+08:00|99|99|
99|99|
| 1970-01-01T08:00:00.106+08:00|99|99|
99|99|
| 1970-01-01T08:00:01.000+08:00| 2| 2|
[jira] [Created] (IOTDB-237) client fetches all data in spite of maxPrintRowCount
Lei Rui created IOTDB-237: - Summary: client fetches all data in spite of maxPrintRowCount Key: IOTDB-237 URL: https://issues.apache.org/jira/browse/IOTDB-237 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-229) Inconsistent usage of Marker in TsFile
Lei Rui created IOTDB-229: - Summary: Inconsistent usage of Marker in TsFile Key: IOTDB-229 URL: https://issues.apache.org/jira/browse/IOTDB-229 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-223) TsFile skeleton viewer
Lei Rui created IOTDB-223: - Summary: TsFile skeleton viewer Key: IOTDB-223 URL: https://issues.apache.org/jira/browse/IOTDB-223 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (IOTDB-215) Make users aware of auxiliary tools such as print-tsfile-resource-files by fulfilling docs
Lei Rui created IOTDB-215: - Summary: Make users aware of auxiliary tools such as print-tsfile-resource-files by fulfilling docs Key: IOTDB-215 URL: https://issues.apache.org/jira/browse/IOTDB-215 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-214) TSExecuteStatementResp's ignoreTimeStamp is not set in the TSServiceImpl.executeQueryStatement
Lei Rui created IOTDB-214: - Summary: TSExecuteStatementResp's ignoreTimeStamp is not set in the TSServiceImpl.executeQueryStatement Key: IOTDB-214 URL: https://issues.apache.org/jira/browse/IOTDB-214 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-213) query optimization for repeated paths in SELECT clause
Lei Rui created IOTDB-213: - Summary: query optimization for repeated paths in SELECT clause Key: IOTDB-213 URL: https://issues.apache.org/jira/browse/IOTDB-213 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-212) Improve the IoTDB SQL Documentation
Lei Rui created IOTDB-212: - Summary: Improve the IoTDB SQL Documentation Key: IOTDB-212 URL: https://issues.apache.org/jira/browse/IOTDB-212 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-211) Start command too long on windows with big classpaths in start-server.bat
Lei Rui created IOTDB-211: - Summary: Start command too long on windows with big classpaths in start-server.bat Key: IOTDB-211 URL: https://issues.apache.org/jira/browse/IOTDB-211 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-204) spark-tsfile narrow table's new way to execute query
Lei Rui created IOTDB-204: - Summary: spark-tsfile narrow table's new way to execute query Key: IOTDB-204 URL: https://issues.apache.org/jira/browse/IOTDB-204 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-172) TsFileResource updateTime Bug
Lei Rui created IOTDB-172: - Summary: TsFileResource updateTime Bug Key: IOTDB-172 URL: https://issues.apache.org/jira/browse/IOTDB-172 Project: Apache IoTDB Issue Type: Bug Reporter: Lei Rui -- This message was sent by Atlassian Jira (v8.3.2#803003)
[jira] [Created] (IOTDB-153) LIMIT OFFSET clause is unreasonably slow when N and OffsetValue are rather small.
Lei Rui created IOTDB-153: - Summary: LIMIT OFFSET clause is unreasonably slow when N and OffsetValue are rather small. Key: IOTDB-153 URL: https://issues.apache.org/jira/browse/IOTDB-153 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IOTDB-150) Try some other web fonts
Lei Rui created IOTDB-150: - Summary: Try some other web fonts Key: IOTDB-150 URL: https://issues.apache.org/jira/browse/IOTDB-150 Project: Apache IoTDB Issue Type: Wish Reporter: Lei Rui -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Created] (IOTDB-102) CLONE - IoTDB failed to start client since the required jars are not in the correct lib directory
Lei Rui created IOTDB-102:
-
Summary: CLONE - IoTDB failed to start client since the required
jars are not in the correct lib directory
Key: IOTDB-102
URL: https://issues.apache.org/jira/browse/IOTDB-102
Project: Apache IoTDB
Issue Type: Bug
Reporter: Lei Rui
*Operating system version*: Windows 10 x64
*Reproduction of the problem*
When following the "Quick Start" instruction on IoTDB website
([https://iotdb.apache.org/#/Documents/Quick%20Start]), it would result in
"Cannot find the main class 'org.apache.iotdb.cli.client.Client' error". There
is the detailed operation below:
# Let $IOTDB_HOME = /workspace/incubator-iotdb/iotdb/iotdb/
Let $IOTDB_CLI_HOME = /workspace/incubator-iotdb/iotdb-cli/cli/
#
{code:java}
> pwd
/workspace/incubator-iotdb
> mvn clean package -pl iotdb -am -Dmaven.test.skip=true{code}
#
{code:java}
> $IOTDB_HOME\bin\start-server.bat{code}
#
{code:java}
> pwd
/workspace/incubator-iotdb
> mvn clean package -pl iotdb-cli -am -Dmaven.test.skip=true
> $IOTDB_CLI_HOME\bin\start-client.bat -h -p -u {code}
which would result in:
{code:java}
Cannot find the main class 'org.apache.iotdb.cli.client.Client'{code}
*Screenshots*
Follow the "Quick Start" instruction on the website.
!5.PNG!
!6.PNG!
!7.PNG!
!8.PNG!
!2.PNG!
!1.PNG!
!3.PNG!!4.PNG!
*Reason*
The reason of causing this error is the required jars to start IoTDB client are
not in the correct lib directory.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (IOTDB-64) TsFile Spark Connector
[ https://issues.apache.org/jira/browse/IOTDB-64?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei Rui closed IOTDB-64. Resolution: Fixed > TsFile Spark Connector > -- > > Key: IOTDB-64 > URL: https://issues.apache.org/jira/browse/IOTDB-64 > Project: Apache IoTDB > Issue Type: New Feature >Reporter: Lei Rui >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Old realization of TsFile-Spark-Connector is not compatible with the latest > TsFile since TsFile has been redesigned. > Besides, old version loads data from TsFile into Spark table in the form of > [time, delta_object, sensor1, sensor 2, ...], which will also be changed in > this update. The updated form is like [time, delta_object1.sensor1, > delta_object1.sensor2, delta_object2.sensor_1, ...] > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (IOTDB-92) The data locality principle used by Spark loses ground in the face of TsFile.
Lei Rui created IOTDB-92: Summary: The data locality principle used by Spark loses ground in the face of TsFile. Key: IOTDB-92 URL: https://issues.apache.org/jira/browse/IOTDB-92 Project: Apache IoTDB Issue Type: Improvement Reporter: Lei Rui In the development of TsFile-Spark-Connector, we discover the problem that the data locality principle used by Spark loses ground in the face of TsFile. We believe the problem is rooted in the storage structure design of TsFile. Our latest implementation of TsFile-Spark-Connector finds a way to guarantee the proper functionality despite the constraint. The resolvement of the data locality problem is left for future work. Below are the details. h1. 1. Spark Partition In Apache Spark, the data is stored in the form of RDDs and divided into partitions across various nodes. A partition is a logical chunk of a large distributed data set that helps parallelize distributed data processing. Spark works on data locality principle to minimize the network traffic for sending data between executors. https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-rdd-partitions.html https://techvidvan.com/tutorials/spark-partition/ h1. 2. TsFile Structure TsFile is a columnar storage file format designed for time series data, which supports efficient compression and query. Data in TsFile are organized by device-measurement hierarchy. As the figure below shows, the storage unit of a device is a chunk group and that of a measurement is a chunk. Measurements of a device are grouped together. Under this architecture, different partitions of Spark logically contain different sets of chunkgroups of TsFile. Now consider the query process of this TsFile on Spark. Supposing that we query “select * from root where d1.s6<2.5 and d2.s1>10”, a scheduled task of a partition has to deal with the whole data to get the right answer. However, this also means that it is nearly impossible to apply the data locality principle. h1. 3. Problem Now we can summarize two problems. The first problem is how to guarantee the correctness of the queried answer integrated from all the partition task without changing the current storage structure of TsFile. To solve this problem, we propose a solution by converting the space partition constraint to the time partition constraint while still requiring a single task to have access to the whole TsFile data. As shown in the figure below, the task of partition 1 is assigned the yellow marked time partition constraint; the task of partition 2 is assigned the green marked time partition constraint; the task of partition 3 is assigned empty time partition constraint because the former two tasks have completed the query. The second problem is more fundamental. That is, how we can adjust to enable some extent of data locality of TsFile when it is queried on Spark. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
