[GitHub] [incubator-iotdb] jixuan1989 commented on a change in pull request #311: modify antlr3 to support timeseries path in the number format

2019-07-26 Thread GitBox
jixuan1989 commented on a change in pull request #311: modify antlr3 to support 
timeseries path in the number format
URL: https://github.com/apache/incubator-iotdb/pull/311#discussion_r307954311
 
 

 ##
 File path: server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TSLexer.g
 ##
 @@ -146,25 +146,51 @@ StringLiteral
 )
 ;
 
-//2016-11-16T16:22:33+0800
-DATETIME
-: Digit+ (MINUS | DIVIDE | DOT) Digit+ (MINUS | DIVIDE | DOT) Digit+ ('T' 
| WS) Digit+ COLON Digit+ COLON Digit+ (DOT Digit+)? ((PLUS | MINUS) Digit+ 
COLON Digit+)?
-;
-
 NegativeInteger
 :
 '-' Digit+
 ;
 
-NonNegativeInteger
+PositiveInteger
+:
+'+' Digit+
+;
+
+NegativeFloat
+:
+'-' Digit+ DOT Digit+ (('e' | 'E') ('+'|'-')? Digit+)?
 
 Review comment:
   I tried Java:
   ```
 System.out.println(Float.valueOf(".3"));
 System.out.println(Float.valueOf("3."));
   ```
   Java works in these cases.
   
   I am not sure whether we need to support that...


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #311: modify antlr3 to support timeseries path in the number format

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #311: modify antlr3 to support 
timeseries path in the number format
URL: https://github.com/apache/incubator-iotdb/pull/311#discussion_r307950401
 
 

 ##
 File path: server/src/main/antlr3/org/apache/iotdb/db/sql/parse/TSLexer.g
 ##
 @@ -146,25 +146,51 @@ StringLiteral
 )
 ;
 
-//2016-11-16T16:22:33+0800
-DATETIME
-: Digit+ (MINUS | DIVIDE | DOT) Digit+ (MINUS | DIVIDE | DOT) Digit+ ('T' 
| WS) Digit+ COLON Digit+ COLON Digit+ (DOT Digit+)? ((PLUS | MINUS) Digit+ 
COLON Digit+)?
-;
-
 NegativeInteger
 :
 '-' Digit+
 ;
 
-NonNegativeInteger
+PositiveInteger
+:
+'+' Digit+
+;
+
+NegativeFloat
+:
+'-' Digit+ DOT Digit+ (('e' | 'E') ('+'|'-')? Digit+)?
 
 Review comment:
   If one of the integer part and the decimal part is zero, it may be omitted, 
like ".5" or "3.". Should we support that too?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin closed pull request #313: Directory name changed, but the document didn't change

2019-07-26 Thread GitBox
qiaojialin closed pull request #313: Directory name changed, but the document 
didn't change
URL: https://github.com/apache/incubator-iotdb/pull/313
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin commented on issue #313: Directory name changed, but the document didn't change

2019-07-26 Thread GitBox
qiaojialin commented on issue #313: Directory name changed, but the document 
didn't change
URL: https://github.com/apache/incubator-iotdb/pull/313#issuecomment-515646295
 
 
   this problem has been fixed in another pr, which is already merged, thanks!


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #316: Cherry pick rel0.8

2019-07-26 Thread GitBox
qiaojialin merged pull request #316: Cherry pick rel0.8
URL: https://github.com/apache/incubator-iotdb/pull/316
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin opened a new pull request #316: Cherry pick rel0.8

2019-07-26 Thread GitBox
qiaojialin opened a new pull request #316: Cherry pick rel0.8
URL: https://github.com/apache/incubator-iotdb/pull/316
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #315: Cherry pick rel0.8

2019-07-26 Thread GitBox
qiaojialin merged pull request #315: Cherry pick rel0.8
URL: https://github.com/apache/incubator-iotdb/pull/315
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin opened a new pull request #315: Cherry pick rel0.8

2019-07-26 Thread GitBox
qiaojialin opened a new pull request #315: Cherry pick rel0.8
URL: https://github.com/apache/incubator-iotdb/pull/315
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jixuan1989 merged pull request #314: fix readme bug

2019-07-26 Thread GitBox
jixuan1989 merged pull request #314: fix readme bug
URL: https://github.com/apache/incubator-iotdb/pull/314
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin opened a new pull request #314: fix readme bug

2019-07-26 Thread GitBox
qiaojialin opened a new pull request #314: fix readme bug
URL: https://github.com/apache/incubator-iotdb/pull/314
 
 
   fix bug in readme
   update license and notice files


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 edited a comment on issue #301: add -e option for iotdb client

2019-07-26 Thread GitBox
jt2594838 edited a comment on issue #301: add -e option for iotdb client
URL: https://github.com/apache/incubator-iotdb/pull/301#issuecomment-515643443
 
 
    


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on issue #301: add -e option for iotdb client

2019-07-26 Thread GitBox
jt2594838 commented on issue #301: add -e option for iotdb client
URL: https://github.com/apache/incubator-iotdb/pull/301#issuecomment-515643443
 
 
   +1


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] Genius-pig opened a new pull request #313: Directory name changed, but the document didn't change

2019-07-26 Thread GitBox
Genius-pig opened a new pull request #313: Directory name changed, but the 
document didn't change
URL: https://github.com/apache/incubator-iotdb/pull/313
 
 
   Directory name changed, but the document didn't change


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] Genius-pig closed pull request #306: Directory name changed, but document didn't change

2019-07-26 Thread GitBox
Genius-pig closed pull request #306: Directory name changed, but document 
didn't change
URL: https://github.com/apache/incubator-iotdb/pull/306
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] LeiRui commented on issue #311: modify antlr3 to support timeseries path in the number format

2019-07-26 Thread GitBox
LeiRui commented on issue #311: modify antlr3 to support timeseries path in the 
number format
URL: https://github.com/apache/incubator-iotdb/pull/311#issuecomment-515641225
 
 
   I prefer `rebase and merge` of this pr.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jixuan1989 commented on issue #312: use latest binary file to build the docker image

2019-07-26 Thread GitBox
jixuan1989 commented on issue #312: use latest binary file to build the docker 
image
URL: https://github.com/apache/incubator-iotdb/pull/312#issuecomment-515460588
 
 
   > Hey @jixuan1989 I would suggest to move this file to "src/main/docker" as 
this fits more to the maven convention. Otherwise it lloks good.
   
   Hi, I move the file to docker/src/main/.
   
   > Why is the Entrypoing commented out?
   
   It is because I think this docker image can be as a base image. We can use 
it to build a server image or a client image. Or, we can use it to run either a 
server instance or a client instance by:
   
   * `docker run -p 6667:6667 -v mydata:/iotdb/data -v mylogs:/iotdb/logs -d 
iotdb:base /iotdb/bin/start-server.sh`
   * `docker run -it iotdb:base /cli/bin/start-client.sh -h  -p 6667 -u 
root -pw root` 
   
   However, if we have chance to publish the image to DockerHub, we need open 
the Entrypoing, I think.
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jixuan1989 opened a new pull request #312: use latest binary file to build the docker image

2019-07-26 Thread GitBox
jixuan1989 opened a new pull request #312: use latest binary file to build the 
docker image
URL: https://github.com/apache/incubator-iotdb/pull/312
 
 
   Because the dockerfile depends on the compiled binary file of IoTDB, I 
modify the dockerfile to use the new binary file.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jixuan1989 commented on issue #300: Split the LICENSE and NOTICE as source-release version and binary-release version

2019-07-26 Thread GitBox
jixuan1989 commented on issue #300: Split the LICENSE and NOTICE as 
source-release version and binary-release version
URL: https://github.com/apache/incubator-iotdb/pull/300#issuecomment-515433649
 
 
   has been cherry-picked to master by #309 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jixuan1989 merged pull request #310: Final steps towards rel.

2019-07-26 Thread GitBox
jixuan1989 merged pull request #310: Final steps towards rel.
URL: https://github.com/apache/incubator-iotdb/pull/310
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer commented on issue #310: Final steps towards rel.

2019-07-26 Thread GitBox
JulianFeinauer commented on issue #310: Final steps towards rel.
URL: https://github.com/apache/incubator-iotdb/pull/310#issuecomment-515430030
 
 
   @jixuan1989 you have to use `mvn package -Papache-release` to enable these 
steps.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jixuan1989 commented on issue #310: Final steps towards rel.

2019-07-26 Thread GitBox
jixuan1989 commented on issue #310: Final steps towards rel.
URL: https://github.com/apache/incubator-iotdb/pull/310#issuecomment-515429820
 
 
   Hi, I tried this branch but after running `mvn package`, I do not get the 
.zip file... 


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer commented on a change in pull request #310: Final steps towards rel.

2019-07-26 Thread GitBox
JulianFeinauer commented on a change in pull request #310: Final steps towards 
rel.
URL: https://github.com/apache/incubator-iotdb/pull/310#discussion_r307714852
 
 

 ##
 File path: pom.xml
 ##
 @@ -584,5 +583,78 @@
 
 
 
+
+
+apache-release
+
+
+
+org.apache.maven.plugins
+maven-assembly-plugin
+
+
+source-release-assembly
+package
+
+single
+
+
+
+
apache-iotdb-${project.version}-incubating
 
 Review comment:
   yeah.. therefore I changed the name of the module back to "iotdb-parent" 
which is more standard : )


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jixuan1989 commented on a change in pull request #310: Final steps towards rel.

2019-07-26 Thread GitBox
jixuan1989 commented on a change in pull request #310: Final steps towards rel.
URL: https://github.com/apache/incubator-iotdb/pull/310#discussion_r307714334
 
 

 ##
 File path: pom.xml
 ##
 @@ -584,5 +583,78 @@
 
 
 
+
+
+apache-release
+
+
+
+org.apache.maven.plugins
+maven-assembly-plugin
+
+
+source-release-assembly
+package
+
+single
+
+
+
+
apache-iotdb-${project.version}-incubating
 
 Review comment:
   Oh, now I get what Justin means... just add "incubating" here is fine..


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #309: Cherry pick rel0.8

2019-07-26 Thread GitBox
qiaojialin merged pull request #309: Cherry pick rel0.8
URL: https://github.com/apache/incubator-iotdb/pull/309
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] samperson1997 commented on issue #306: Directory name changed, but document didn't change

2019-07-26 Thread GitBox
samperson1997 commented on issue #306: Directory name changed, but document 
didn't change
URL: https://github.com/apache/incubator-iotdb/pull/306#issuecomment-515419874
 
 
   Hi, there are some others' commits and changed files mixed in this PR during 
your conflict solving. You can simply reset and stage the changes in this PR, 
and use git pull --rebase xxx command to pull and rebase on the latest codes. 
Then by unstaging and force pushing, there will only be your own changes in 
this PR :-)


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] LeiRui opened a new pull request #311: modify antlr3 to support timeseries path in the number format

2019-07-26 Thread GitBox
LeiRui opened a new pull request #311: modify antlr3 to support timeseries path 
in the number format
URL: https://github.com/apache/incubator-iotdb/pull/311
 
 
   To support series path in number format, we have the following logic in 
TSParser.g already:
   ```
   nodeName
   : identifier
   | STAR
   ;
   
   identifier
   :
   Identifier | integer
   ;
   ```
   
   Then the main challenge left here is to teach antlr3 to recognize `integer` 
in the series path, which is easy to be confused with and recognized as 
`Float`, `DATETIME` partly because of the "Longest match wins first" rule of 
Antlr3.
   
   For example,

   |  integer   |   float|  
datetime  | seriesPath in the number 
format |
   
||||--|
   | +12 / -12 / 12 | +12.233 / -12.233 / +12.3332e-1 / -12.3332e+2  / 12.3332 
/ 12.3332e-1 | 2017-11-1T00:15:00+08:00 / 2017.11.1 00:15:00+08:00 | 
8955.1227.223|
   
   The solution is to do a more elaborate division and some are directly 
recognized in TSLexer and some are later combined in TSParser: 
   
   In TSLexer, we have:
   - PositiveInteger, e.g., `+12`
   - NegativeInteger, e.g., `-12`
   - PositiveFloat, e.g., `+12.233`, `+12.3332e-1 `
   - NegativeFloat, e.g., `-12.233`, `-12.3332e-1 `
   NOTE that the existence of '+' and '-' help ease the match rule. For those 
without signs, things get tricky as follows.
   - UnsignedInteger, e.g., `12`
   - DATETIME, e.g., `2017.11.1 00:15:00+08:00`. 
   NOTE that here an critical trick is used (see code). If the first lexer rule 
of DATETIME is not matched, then UnsignedInteger is recognized instead.
   - DoubleInScientificNotationSuffix, e.g., `.E+10`. 
   NOTE that here the same critical trick is used (see code). If the first rule 
is not matched, then DOT is recognized instead.
   
   In TSParser, we have:
   - nonNegativeInteger (which is needed by LIMIT clause)
   - integer
   - floatValue
   NOTE that "float" shoud not be used as a name as it is reserved.
   NOTE that the old FLOAT in the TSLexer is removed and floatValue in TSParser 
is used instead. It is because we want to support series path in the number 
format, **which can not be separated from the float number in the lexer level**.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin commented on issue #306: Directory name changed, but document didn't change

2019-07-26 Thread GitBox
qiaojialin commented on issue #306: Directory name changed, but document didn't 
change
URL: https://github.com/apache/incubator-iotdb/pull/306#issuecomment-515413980
 
 
   besides, before we release 0.8.0, all changes in documents should go to 
rel/0.8


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin commented on issue #306: Directory name changed, but document didn't change

2019-07-26 Thread GitBox
qiaojialin commented on issue #306: Directory name changed, but document didn't 
change
URL: https://github.com/apache/incubator-iotdb/pull/306#issuecomment-515413801
 
 
   the conflicts are too many... You'd better checkout a branch from rel/0.8 
and resubmit a pr


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #308: Update dockerfile to only compile server and client

2019-07-26 Thread GitBox
qiaojialin merged pull request #308: Update dockerfile to only compile server 
and client
URL: https://github.com/apache/incubator-iotdb/pull/308
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #307: Added a note on spark-tsfile documentation to indicate the potential …

2019-07-26 Thread GitBox
qiaojialin merged pull request #307: Added a note on spark-tsfile documentation 
to indicate the potential …
URL: https://github.com/apache/incubator-iotdb/pull/307
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer opened a new pull request #310: Final steps towards rel.

2019-07-26 Thread GitBox
JulianFeinauer opened a new pull request #310: Final steps towards rel.
URL: https://github.com/apache/incubator-iotdb/pull/310
 
 
   Removed "build" module and moved this to parent pom.
   Renamed parent pom without incubating.
   Added incubating artefact names in assembly plugin.
   Added SHA512 for all artefacts.
   
   Big props to @chrisdutz for his support!
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307677023
 
 

 ##
 File path: server/src/main/java/org/apache/iotdb/db/service/IoTDB.java
 ##
 @@ -97,6 +98,7 @@ private void setUp() throws StartupException {
 registerManager.register(Measurement.INSTANCE);
 registerManager.register(SyncServerManager.getInstance());
 registerManager.register(TVListAllocator.getInstance());
+registerManager.register(MergeManager.getINSTANCE());
 
 Review comment:
   It's okay since RecoverMergeTasks are not submitted to the MergeManager, 
they are done synchronously instead. Imagine, if the timed merge task is 
triggered before the StorageEngine is set up, things will become a mess.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin opened a new pull request #309: Cherry pick rel0.8

2019-07-26 Thread GitBox
qiaojialin opened a new pull request #309: Cherry pick rel0.8
URL: https://github.com/apache/incubator-iotdb/pull/309
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307674702
 
 

 ##
 File path: server/src/main/java/org/apache/iotdb/db/service/IoTDB.java
 ##
 @@ -97,6 +98,7 @@ private void setUp() throws StartupException {
 registerManager.register(Measurement.INSTANCE);
 registerManager.register(SyncServerManager.getInstance());
 registerManager.register(TVListAllocator.getInstance());
+registerManager.register(MergeManager.getINSTANCE());
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307673734
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
 
 Review comment:
   If you are not okay with that, just turn on full merge option from the 
beginning, otherwise, you just are making the previous merges do some useless 
work. We may add further compactions in the future, but it is not the concern 
here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] samperson1997 removed a comment on issue #308: Update dockerfile to only compile server and client

2019-07-26 Thread GitBox
samperson1997 removed a comment on issue #308: Update dockerfile to only 
compile server and client
URL: https://github.com/apache/incubator-iotdb/pull/308#issuecomment-515392570
 
 
   @samperson1997


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] samperson1997 commented on issue #308: Update dockerfile to only compile server and client

2019-07-26 Thread GitBox
samperson1997 commented on issue #308: Update dockerfile to only compile server 
and client
URL: https://github.com/apache/incubator-iotdb/pull/308#issuecomment-515392570
 
 
   @samperson1997


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307673734
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
 
 Review comment:
   If you are not okay with that, just turn on full merge option from the 
beginning. We may add further compactions in the future, but it is not the 
concern here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307677023
 
 

 ##
 File path: server/src/main/java/org/apache/iotdb/db/service/IoTDB.java
 ##
 @@ -97,6 +98,7 @@ private void setUp() throws StartupException {
 registerManager.register(Measurement.INSTANCE);
 registerManager.register(SyncServerManager.getInstance());
 registerManager.register(TVListAllocator.getInstance());
+registerManager.register(MergeManager.getINSTANCE());
 
 Review comment:
   Actually, it's okay since RecoverMergeTasks are not submitted to the 
MergeManager, they are done synchronously instead.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] samperson1997 commented on issue #305: let the dockerfile build from iotdb v0.8

2019-07-26 Thread GitBox
samperson1997 commented on issue #305: let the dockerfile build from iotdb v0.8
URL: https://github.com/apache/incubator-iotdb/pull/305#issuecomment-515391410
 
 
   Spark-tsfile module is avoided to be compiled by [PR 
#308](https://github.com/apache/incubator-iotdb/pull/308).


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307675040
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeManager.java
 ##
 @@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.manage;
+
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.merge.task.MergeTask;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.ServiceType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeManager provides a ThreadPool to queue and run all merge tasks to 
restrain the total
+ * resources occupied by merge and manages a Timer to periodically issue a 
global merge.
+ */
+public class MergeManager implements IService {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeManager.class);
+  private static final MergeManager INSTANCE = new MergeManager();
+
+  private AtomicInteger threadCnt = new AtomicInteger();
+  private ThreadPoolExecutor mergeTaskPool;
+  private ScheduledExecutorService timedMergeThreadPool;
+
+  private MergeManager() {
+  }
 
 Review comment:
   replied previously


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] samperson1997 opened a new pull request #308: Update dockerfile to only compile server and client

2019-07-26 Thread GitBox
samperson1997 opened a new pull request #308: Update dockerfile to only compile 
server and client
URL: https://github.com/apache/incubator-iotdb/pull/308
 
 
   Update dockerfile to only compile server and client during building.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307675040
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeManager.java
 ##
 @@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.manage;
+
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.merge.task.MergeTask;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.ServiceType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeManager provides a ThreadPool to queue and run all merge tasks to 
restrain the total
+ * resources occupied by merge and manages a Timer to periodically issue a 
global merge.
+ */
+public class MergeManager implements IService {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeManager.class);
+  private static final MergeManager INSTANCE = new MergeManager();
+
+  private AtomicInteger threadCnt = new AtomicInteger();
+  private ThreadPoolExecutor mergeTaskPool;
+  private ScheduledExecutorService timedMergeThreadPool;
+
+  private MergeManager() {
+  }
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307674976
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/manage/MergeManager.java
 ##
 @@ -0,0 +1,110 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.manage;
+
+import java.util.concurrent.Executors;
+import java.util.concurrent.ScheduledExecutorService;
+import java.util.concurrent.ThreadPoolExecutor;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import org.apache.iotdb.db.conf.IoTDBDescriptor;
+import org.apache.iotdb.db.engine.StorageEngine;
+import org.apache.iotdb.db.engine.merge.task.MergeTask;
+import org.apache.iotdb.db.exception.StorageEngineException;
+import org.apache.iotdb.db.service.IService;
+import org.apache.iotdb.db.service.ServiceType;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeManager provides a ThreadPool to queue and run all merge tasks to 
restrain the total
+ * resources occupied by merge and manages a Timer to periodically issue a 
global merge.
+ */
+public class MergeManager implements IService {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeManager.class);
+  private static final MergeManager INSTANCE = new MergeManager();
+
+  private AtomicInteger threadCnt = new AtomicInteger();
+  private ThreadPoolExecutor mergeTaskPool;
+  private ScheduledExecutorService timedMergeThreadPool;
+
+  private MergeManager() {
+  }
+
+  public static MergeManager getINSTANCE() {
+return INSTANCE;
+  }
+
+  public void submit(MergeTask mergeTask) {
+mergeTaskPool.submit(mergeTask);
+  }
+
+  @Override
+  public void start() {
+if (mergeTaskPool == null) {
+  int threadNum = 
IoTDBDescriptor.getInstance().getConfig().getMergeConcurrentThreads();
+  if (threadNum <= 0) {
+threadNum = 1;
+  }
+  mergeTaskPool =
+  (ThreadPoolExecutor) Executors.newFixedThreadPool(threadNum,
+  r -> new Thread(r, "MergeThread-" + 
threadCnt.getAndIncrement()));
+  long mergeInterval = 
IoTDBDescriptor.getInstance().getConfig().getMergeIntervalSec();
+  if (mergeInterval > 0) {
+timedMergeThreadPool = Executors.newSingleThreadScheduledExecutor( r 
-> new Thread(r,
+"TimedMergeThread"));
+timedMergeThreadPool.scheduleAtFixedRate(this::flushAll, mergeInterval,
+mergeInterval, TimeUnit.SECONDS);
+  }
+  logger.info("MergeManager started");
+}
+  }
+
+  @Override
+  public void stop() {
+if (mergeTaskPool != null) {
+  if (timedMergeThreadPool != null) {
+timedMergeThreadPool.shutdownNow();
+timedMergeThreadPool = null;
+  }
+  mergeTaskPool.shutdownNow();
+  logger.info("Waiting for task pool to shut down");
+  while (!mergeTaskPool.isShutdown()) {
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307674702
 
 

 ##
 File path: server/src/main/java/org/apache/iotdb/db/service/IoTDB.java
 ##
 @@ -97,6 +98,7 @@ private void setUp() throws StartupException {
 registerManager.register(Measurement.INSTANCE);
 registerManager.register(SyncServerManager.getInstance());
 registerManager.register(TVListAllocator.getInstance());
+registerManager.register(MergeManager.getINSTANCE());
 
 Review comment:
   fixed


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of merge

2019-07-26 Thread GitBox
jt2594838 commented on a change in pull request #258: [IOTDB-143]Development of 
merge
URL: https://github.com/apache/incubator-iotdb/pull/258#discussion_r307673734
 
 

 ##
 File path: 
server/src/main/java/org/apache/iotdb/db/engine/merge/task/MergeFileTask.java
 ##
 @@ -0,0 +1,230 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.engine.merge.task;
+
+import java.io.IOException;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import org.apache.commons.io.FileUtils;
+import org.apache.iotdb.db.engine.merge.manage.MergeContext;
+import org.apache.iotdb.db.engine.merge.manage.MergeResource;
+import org.apache.iotdb.db.engine.merge.recover.MergeLogger;
+import org.apache.iotdb.db.engine.storagegroup.TsFileResource;
+import org.apache.iotdb.tsfile.exception.write.TsFileNotCompleteException;
+import org.apache.iotdb.tsfile.file.metadata.ChunkGroupMetaData;
+import org.apache.iotdb.tsfile.file.metadata.ChunkMetaData;
+import org.apache.iotdb.tsfile.read.TsFileSequenceReader;
+import org.apache.iotdb.tsfile.read.common.Chunk;
+import org.apache.iotdb.tsfile.read.common.Path;
+import org.apache.iotdb.tsfile.write.schema.FileSchema;
+import org.apache.iotdb.tsfile.write.writer.ForceAppendTsFileWriter;
+import org.apache.iotdb.tsfile.write.writer.RestorableTsFileIOWriter;
+import org.apache.iotdb.tsfile.write.writer.TsFileIOWriter;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * MergeFileTask merges the merge temporary files with the seqFiles, either 
move the merged
+ * chunks in the temp files into the seqFiles or move the unmerged chunks into 
the merge temp
+ * files, depending on which one is the majority.
+ */
+class MergeFileTask {
+
+  private static final Logger logger = 
LoggerFactory.getLogger(MergeFileTask.class);
+
+  private String taskName;
+  private MergeContext context;
+  private MergeLogger mergeLogger;
+  private MergeResource resource;
+  private List unmergedFiles;
+
+  MergeFileTask(String taskName, MergeContext context, MergeLogger mergeLogger,
+  MergeResource resource, List unmergedSeqFiles) {
+this.taskName = taskName;
+this.context = context;
+this.mergeLogger = mergeLogger;
+this.resource = resource;
+this.unmergedFiles = unmergedSeqFiles;
+  }
+
+  void mergeFiles() throws IOException {
+// decide whether to write the unmerged chunks to the merge files or to 
move the merged chunks
+// back to the origin seqFile's
+if (logger.isInfoEnabled()) {
+  logger.info("{} starts to merge {} files", taskName, 
unmergedFiles.size());
+}
+long startTime = System.currentTimeMillis();
+int cnt = 0;
+for (TsFileResource seqFile : unmergedFiles) {
+  int mergedChunkNum = context.getMergedChunkCnt().getOrDefault(seqFile, 
0);
+  int unmergedChunkNum = 
context.getUnmergedChunkCnt().getOrDefault(seqFile, 0);
+  if (mergedChunkNum >= unmergedChunkNum) {
+// move the unmerged data to the new file
+if (logger.isInfoEnabled()) {
+  logger.info("{} moving unmerged data of {} to the merged file, {} 
merged chunks, {} "
+  + "unmerged chunks", taskName, seqFile.getFile().getName(), 
mergedChunkNum, unmergedChunkNum);
+}
+moveUnmergedToNew(seqFile);
+  } else {
+// move the merged data to the old file
+if (logger.isInfoEnabled()) {
 
 Review comment:
   If you are not okay with that, just turn on full merge option from the 
beginning. You may add further compactions in the future, but it is not the 
concern here.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] mikip65 opened a new pull request #307: Added a note on spark-tsfile documentation to indicate the potential …

2019-07-26 Thread GitBox
mikip65 opened a new pull request #307: Added a note on spark-tsfile 
documentation to indicate the potential …
URL: https://github.com/apache/incubator-iotdb/pull/307
 
 
   …incompatiblity between openjdk and scala


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer commented on issue #300: Split the LICENSE and NOTICE as source-release version and binary-release version

2019-07-26 Thread GitBox
JulianFeinauer commented on issue #300: Split the LICENSE and NOTICE as 
source-release version and binary-release version
URL: https://github.com/apache/incubator-iotdb/pull/300#issuecomment-515379861
 
 
   +1


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 merged pull request #300: Split the LICENSE and NOTICE as source-release version and binary-release version

2019-07-26 Thread GitBox
jt2594838 merged pull request #300: Split the LICENSE and NOTICE as 
source-release version and binary-release version
URL: https://github.com/apache/incubator-iotdb/pull/300
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] JulianFeinauer commented on issue #300: Split the LICENSE and NOTICE as source-release version and binary-release version

2019-07-26 Thread GitBox
JulianFeinauer commented on issue #300: Split the LICENSE and NOTICE as 
source-release version and binary-release version
URL: https://github.com/apache/incubator-iotdb/pull/300#issuecomment-515379621
 
 
   I would merge this now if there are no concerns left?


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] jt2594838 commented on issue #300: Split the LICENSE and NOTICE as source-release version and binary-release version

2019-07-26 Thread GitBox
jt2594838 commented on issue #300: Split the LICENSE and NOTICE as 
source-release version and binary-release version
URL: https://github.com/apache/incubator-iotdb/pull/300#issuecomment-515379301
 
 
   +1


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin merged pull request #305: let the dockerfile build from iotdb v0.8

2019-07-26 Thread GitBox
qiaojialin merged pull request #305: let the dockerfile build from iotdb v0.8
URL: https://github.com/apache/incubator-iotdb/pull/305
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] Genius-pig opened a new pull request #306: Directory name changed, but document didn't change

2019-07-26 Thread GitBox
Genius-pig opened a new pull request #306: Directory name changed, but document 
didn't change
URL: https://github.com/apache/incubator-iotdb/pull/306
 
 
   Directory name changed, but document didn't change


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services


[GitHub] [incubator-iotdb] qiaojialin commented on issue #303: transform old form to new form and use a parameter to control

2019-07-26 Thread GitBox
qiaojialin commented on issue #303: transform old form to new form and use a 
parameter to control
URL: https://github.com/apache/incubator-iotdb/pull/303#issuecomment-515348571
 
 
   > As for the performance consideration. Given that tsfile is read by path, 
if we want get new table form, we need to split one row of data and build the 
iterator. I'm not sure whether this step can be parallelized. In current 
implementation, I just use union function, which means all operations can be in 
one stage without shuffle.
   
   I mean when users want to use the new table form, there will be no wide row 
which contains all time series at all. Therefore, we do not need to split a 
wide row into several narrow rows.
   
   The following is an example: suppose the file schema has "s1, s2, s3"
   
   In DefaultSource.inferSchema(), we return the "timestamp, deviceId, s1, s2, 
s3" to spark to generate the new table form.
   
   In DefaultSource.buildReader(), the required schema is "timestamp, deviceId, 
s1, s2, s3", suppose the PartitionedFile has one chunk group (device1, s1, s2, 
s3). Then, we could build a tsfile reader to query three time series : 
device1.s1, device1.s2, device1.s3 and directly construct the result to an 
InternalRow and return to Spark.
   
   This step can be parallelized. Each executor in spark will read a partition 
of one tsfile. These executors could work independently.


This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


With regards,
Apache Git Services