LeiRui commented on a change in pull request #455: [IOTDB-251]improve
TSQueryDataSet structure in RPC
URL: https://github.com/apache/incubator-iotdb/pull/455#discussion_r337007825
##########
File path:
server/src/main/java/org/apache/iotdb/db/utils/QueryDataSetUtils.java
##########
@@ -55,66 +55,86 @@ public static TSQueryDataSet
convertQueryDataSetByFetchSize(QueryDataSet queryDa
public static TSQueryDataSet convertQueryDataSetByFetchSize(QueryDataSet
queryDataSet,
int fetchSize, WatermarkEncoder watermarkEncoder) throws IOException {
+ List<TSDataType> dataTypes = queryDataSet.getDataTypes();
+ int columnNum = dataTypes.size();
TSQueryDataSet tsQueryDataSet = new TSQueryDataSet();
- tsQueryDataSet.setRecords(new ArrayList<>());
+ int columnNumWithTime = columnNum + 1;
+ DataOutputStream[] dataOutputStreams = new
DataOutputStream[columnNumWithTime];
+ ByteArrayOutputStream[] byteArrayOutputStreams = new
ByteArrayOutputStream[columnNumWithTime];
+ for (int i = 0; i < columnNumWithTime; i++) {
+ byteArrayOutputStreams[i] = new ByteArrayOutputStream();
+ dataOutputStreams[i] = new DataOutputStream(byteArrayOutputStreams[i]);
+ }
+
+ int rowCount = 0;
+ int valueOccupation = 0;
for (int i = 0; i < fetchSize; i++) {
if (queryDataSet.hasNext()) {
+ rowCount++;
RowRecord rowRecord = queryDataSet.next();
if (watermarkEncoder != null) {
rowRecord = watermarkEncoder.encodeRecord(rowRecord);
}
- tsQueryDataSet.getRecords().add(convertToTSRecord(rowRecord));
+ // use columnOutput to write byte array
+ dataOutputStreams[0].writeLong(rowRecord.getTimestamp());
+ List<Field> fields = rowRecord.getFields();
+ for (int k = 0; k < fields.size(); k++) {
+ Field field = fields.get(k);
+ DataOutputStream dataOutputStream = dataOutputStreams[k + 1]; // DO
NOT FORGET +1
+ if (field.getDataType() == null) {
+ dataOutputStream.writeBoolean(true); // is_empty true
+ } else {
+ dataOutputStream.writeBoolean(false); // is_empty false
Review comment:
Hi, yes, this is scheduled as a possible future improvement as for the inner
structure of TSQueryDataSet's value buffer. And more experiments are needed.
But for now, the core aim of this pr, which is to remove record-wise
structure in TSQueryDataSet, is hit.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
