[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-415220804 @arina-ielchiieva @sohami thanks for the detail catch, done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-414913778 @arina-ielchiieva done This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-414051224 Thanks for your effort to review this PR @amansinha100 @sohami . It has been rebased onto the master and squashed into one commit. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-413782958 @sohami tests have passed by removing the Kind.Integer from the OptionValue. Still keep the `IntegerValidator` to validate the number which should be a integer range value. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-413099526 @sohami @parthchandra thanks for the help. I have addressed the comments according to your branch. Btw, I suggest the SV2's `batchActualRecordCount` be renamed as `originalRecordCount`. The `recordCount` member is the truly final actual record count. Otherwise ,this naming will make others confused. If you agree with it, I will update that. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-412817095 @sohami Since we move the filter logic from `ScanBatch` into the `RuntimeFilterRecordBatch` , I refactor the `RuntimeFilterManager` to move the JPPD judgement logic into `RuntimeFilterVisitor`. These part of codes have fewer changes than before. Your SV2 optimization is more general than mine. I don't know how to operate to rebase your own branch into mine. Maybe you can guide me or I fire another JIRA to refactor the implementation to yours one after this PR and yours were merged. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-412321059 @sohami this commit has added a `RuntimeFilterRecordBath` to replace the related work of the `ScanBatch`. Please review, thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-411604543 ScanBatch’s discussion is on JIRA,please see that @sohami. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-411444653 @sohami I have changed the logic to generate SV2 in the ScanBatch. It is comparatively easy now. Initially, I worried about the performance of the `RemovingRecordBatch`'s `setupNewSchema()` while generating different output schema of `ScanBatch` by applying `RuntimeFilter` with an assumption implementation of the `RemovingRecodBatch`'s `setupNewSchema()` to generate dynamic codes.But actually `RemovingRecodBatch`'s `setupNewSchema()` does not generate dynamic codes. So the changed implementation has reduced some complexity to judge whether to output a schema with SV2 or NONE. I am not sure I fully understand your proposed strategy , maybe a implementation like adding `RemovingRecodBatch` on a node . But the whole design is to connect the `Scan` node and `HashJoin` one. That changing will make more work to do. So if you have no doubt about the current changed implementation , let's it go. Btw, others you mentioned has been addressed. Please review them, tks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-409220807 @amansinha100 The following JIRAs: DRILL-6573 ,DRILL-6572 would solve what we identified before . @sohami 's comments have been addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-408674169 @amansinha100 please review the changes made according to your advices. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-408599354 To the cleanup point, I currently think there's no problem. Once the query was canceled, the foreman or the drillbit will run out , the in-flight runtime filter will failed due to the wire termination of one side. The off-heap memory occupied by the bloom filter will be released by the promise of using `SendingAccountor` to send out `RuntimeFilter` through DataTunnel. If we find any potential issues later , we can fixed them at the following JIRAs. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-407627190 @amansinha100 thanks for your valuable review. Since being on vacation , others will be commented and updated later. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-406601421 ping @amansinha100 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-403284219 @amansinha100 RuntimeFilterManager has changed to support the left deep tree case which you mentioned at the JIRA. Also please see the JRIA reply and review the updates, Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-401686350 @amansinha100 The scan node's memory copy logic has removed. Thanks for the knowledge of `SelectionVectorPrelVisitor.addSelectionRemoverWhereNecessary`. I appreciate what you payed out. Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] weijietong commented on issue #1334: DRILL-6385: Support JPPD feature
weijietong commented on issue #1334: DRILL-6385: Support JPPD feature URL: https://github.com/apache/drill/pull/1334#issuecomment-401230814 ping @amansinha100 @parthchandra or someone else would be pleased to review this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
