[jira] [Work logged] (HIVE-24065) Bloom filters can be cached after deserialization in VectorInBloomFilterColDynamicValue
[ https://issues.apache.org/jira/browse/HIVE-24065?focusedWorklogId=476059=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476059 ] ASF GitHub Bot logged work on HIVE-24065: - Author: ASF GitHub Bot Created on: 29/Aug/20 06:13 Start Date: 29/Aug/20 06:13 Worklog Time Spent: 10m Work Description: abstractdog commented on pull request #1423: URL: https://github.com/apache/hive/pull/1423#issuecomment-683243881 pushed to master This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 476059) Time Spent: 1h 20m (was: 1h 10m) > Bloom filters can be cached after deserialization in > VectorInBloomFilterColDynamicValue > --- > > Key: HIVE-24065 > URL: https://issues.apache.org/jira/browse/HIVE-24065 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: image-2020-08-05-10-05-25-080.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > Same bloom filter is loaded multiple times across tasks. It would be good to > check if we can optimise this, to avoid deserializing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24065) Bloom filters can be cached after deserialization in VectorInBloomFilterColDynamicValue
[ https://issues.apache.org/jira/browse/HIVE-24065?focusedWorklogId=476060=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-476060 ] ASF GitHub Bot logged work on HIVE-24065: - Author: ASF GitHub Bot Created on: 29/Aug/20 06:13 Start Date: 29/Aug/20 06:13 Worklog Time Spent: 10m Work Description: abstractdog closed pull request #1423: URL: https://github.com/apache/hive/pull/1423 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 476060) Time Spent: 1.5h (was: 1h 20m) > Bloom filters can be cached after deserialization in > VectorInBloomFilterColDynamicValue > --- > > Key: HIVE-24065 > URL: https://issues.apache.org/jira/browse/HIVE-24065 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0 > > Attachments: image-2020-08-05-10-05-25-080.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > Same bloom filter is loaded multiple times across tasks. It would be good to > check if we can optimise this, to avoid deserializing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24065) Bloom filters can be cached after deserialization in VectorInBloomFilterColDynamicValue
[ https://issues.apache.org/jira/browse/HIVE-24065?focusedWorklogId=474788=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-474788 ] ASF GitHub Bot logged work on HIVE-24065: - Author: ASF GitHub Bot Created on: 26/Aug/20 12:31 Start Date: 26/Aug/20 12:31 Worklog Time Spent: 10m Work Description: abstractdog commented on pull request #1423: URL: https://github.com/apache/hive/pull/1423#issuecomment-680849960 could you please take a look @rbalamohan ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 474788) Time Spent: 1h 10m (was: 1h) > Bloom filters can be cached after deserialization in > VectorInBloomFilterColDynamicValue > --- > > Key: HIVE-24065 > URL: https://issues.apache.org/jira/browse/HIVE-24065 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Attachments: image-2020-08-05-10-05-25-080.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Same bloom filter is loaded multiple times across tasks. It would be good to > check if we can optimise this, to avoid deserializing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24065) Bloom filters can be cached after deserialization in VectorInBloomFilterColDynamicValue
[
https://issues.apache.org/jira/browse/HIVE-24065?focusedWorklogId=474213=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-474213
]
ASF GitHub Bot logged work on HIVE-24065:
-
Author: ASF GitHub Bot
Created on: 25/Aug/20 08:56
Start Date: 25/Aug/20 08:56
Worklog Time Spent: 10m
Work Description: abstractdog commented on a change in pull request #1423:
URL: https://github.com/apache/hive/pull/1423#discussion_r476289475
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java
##
@@ -100,26 +103,39 @@ public void init(Configuration conf) {
default:
throw new IllegalStateException("Unsupported type " + colVectorType);
}
+
+String queryId = HiveConf.getVar(conf, HiveConf.ConfVars.HIVEQUERYID);
+runtimeCache = ObjectCacheFactory.getCache(conf, queryId, false, true);
}
- private void initValue() {
-InputStream in = null;
+ private void initValue() {
try {
- Object val = bloomFilterDynamicValue.getValue();
- if (val != null) {
-BinaryObjectInspector boi = (BinaryObjectInspector)
bloomFilterDynamicValue.getObjectInspector();
-byte[] bytes = boi.getPrimitiveJavaObject(val);
-in = new NonSyncByteArrayInputStream(bytes);
-bloomFilter = BloomKFilter.deserialize(in);
- } else {
-bloomFilter = null;
- }
- initialized = true;
-} catch (Exception err) {
- throw new RuntimeException(err);
-} finally {
- IOUtils.closeStream(in);
Review comment:
added it back and force pushed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 474213)
Time Spent: 1h (was: 50m)
> Bloom filters can be cached after deserialization in
> VectorInBloomFilterColDynamicValue
> ---
>
> Key: HIVE-24065
> URL: https://issues.apache.org/jira/browse/HIVE-24065
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-08-05-10-05-25-080.png
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Same bloom filter is loaded multiple times across tasks. It would be good to
> check if we can optimise this, to avoid deserializing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24065) Bloom filters can be cached after deserialization in VectorInBloomFilterColDynamicValue
[
https://issues.apache.org/jira/browse/HIVE-24065?focusedWorklogId=474201=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-474201
]
ASF GitHub Bot logged work on HIVE-24065:
-
Author: ASF GitHub Bot
Created on: 25/Aug/20 08:26
Start Date: 25/Aug/20 08:26
Worklog Time Spent: 10m
Work Description: abstractdog commented on a change in pull request #1423:
URL: https://github.com/apache/hive/pull/1423#discussion_r476270406
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java
##
@@ -100,26 +103,39 @@ public void init(Configuration conf) {
default:
throw new IllegalStateException("Unsupported type " + colVectorType);
}
+
+String queryId = HiveConf.getVar(conf, HiveConf.ConfVars.HIVEQUERYID);
+runtimeCache = ObjectCacheFactory.getCache(conf, queryId, false, true);
}
- private void initValue() {
-InputStream in = null;
+ private void initValue() {
try {
- Object val = bloomFilterDynamicValue.getValue();
- if (val != null) {
-BinaryObjectInspector boi = (BinaryObjectInspector)
bloomFilterDynamicValue.getObjectInspector();
-byte[] bytes = boi.getPrimitiveJavaObject(val);
-in = new NonSyncByteArrayInputStream(bytes);
-bloomFilter = BloomKFilter.deserialize(in);
- } else {
-bloomFilter = null;
- }
- initialized = true;
-} catch (Exception err) {
- throw new RuntimeException(err);
-} finally {
- IOUtils.closeStream(in);
Review comment:
good catch, I think it's needed, or at least I removed it accidentally
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 474201)
Time Spent: 50m (was: 40m)
> Bloom filters can be cached after deserialization in
> VectorInBloomFilterColDynamicValue
> ---
>
> Key: HIVE-24065
> URL: https://issues.apache.org/jira/browse/HIVE-24065
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-08-05-10-05-25-080.png
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> Same bloom filter is loaded multiple times across tasks. It would be good to
> check if we can optimise this, to avoid deserializing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24065) Bloom filters can be cached after deserialization in VectorInBloomFilterColDynamicValue
[
https://issues.apache.org/jira/browse/HIVE-24065?focusedWorklogId=474196=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-474196
]
ASF GitHub Bot logged work on HIVE-24065:
-
Author: ASF GitHub Bot
Created on: 25/Aug/20 08:19
Start Date: 25/Aug/20 08:19
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1423:
URL: https://github.com/apache/hive/pull/1423#discussion_r476265302
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java
##
@@ -100,26 +103,39 @@ public void init(Configuration conf) {
default:
throw new IllegalStateException("Unsupported type " + colVectorType);
}
+
+String queryId = HiveConf.getVar(conf, HiveConf.ConfVars.HIVEQUERYID);
+runtimeCache = ObjectCacheFactory.getCache(conf, queryId, false, true);
}
- private void initValue() {
-InputStream in = null;
+ private void initValue() {
try {
- Object val = bloomFilterDynamicValue.getValue();
- if (val != null) {
-BinaryObjectInspector boi = (BinaryObjectInspector)
bloomFilterDynamicValue.getObjectInspector();
-byte[] bytes = boi.getPrimitiveJavaObject(val);
-in = new NonSyncByteArrayInputStream(bytes);
-bloomFilter = BloomKFilter.deserialize(in);
- } else {
-bloomFilter = null;
- }
- initialized = true;
-} catch (Exception err) {
- throw new RuntimeException(err);
-} finally {
- IOUtils.closeStream(in);
Review comment:
no...for bytearrayinputstream its not needed
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 474196)
Time Spent: 40m (was: 0.5h)
> Bloom filters can be cached after deserialization in
> VectorInBloomFilterColDynamicValue
> ---
>
> Key: HIVE-24065
> URL: https://issues.apache.org/jira/browse/HIVE-24065
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-08-05-10-05-25-080.png
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Same bloom filter is loaded multiple times across tasks. It would be good to
> check if we can optimise this, to avoid deserializing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24065) Bloom filters can be cached after deserialization in VectorInBloomFilterColDynamicValue
[
https://issues.apache.org/jira/browse/HIVE-24065?focusedWorklogId=474192=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-474192
]
ASF GitHub Bot logged work on HIVE-24065:
-
Author: ASF GitHub Bot
Created on: 25/Aug/20 08:17
Start Date: 25/Aug/20 08:17
Worklog Time Spent: 10m
Work Description: kgyrtkirk commented on a change in pull request #1423:
URL: https://github.com/apache/hive/pull/1423#discussion_r476264312
##
File path:
ql/src/java/org/apache/hadoop/hive/ql/exec/vector/expressions/VectorInBloomFilterColDynamicValue.java
##
@@ -100,26 +103,39 @@ public void init(Configuration conf) {
default:
throw new IllegalStateException("Unsupported type " + colVectorType);
}
+
+String queryId = HiveConf.getVar(conf, HiveConf.ConfVars.HIVEQUERYID);
+runtimeCache = ObjectCacheFactory.getCache(conf, queryId, false, true);
}
- private void initValue() {
-InputStream in = null;
+ private void initValue() {
try {
- Object val = bloomFilterDynamicValue.getValue();
- if (val != null) {
-BinaryObjectInspector boi = (BinaryObjectInspector)
bloomFilterDynamicValue.getObjectInspector();
-byte[] bytes = boi.getPrimitiveJavaObject(val);
-in = new NonSyncByteArrayInputStream(bytes);
-bloomFilter = BloomKFilter.deserialize(in);
- } else {
-bloomFilter = null;
- }
- initialized = true;
-} catch (Exception err) {
- throw new RuntimeException(err);
-} finally {
- IOUtils.closeStream(in);
Review comment:
I don't see this close in the new implementation...isn't that needed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 474192)
Time Spent: 0.5h (was: 20m)
> Bloom filters can be cached after deserialization in
> VectorInBloomFilterColDynamicValue
> ---
>
> Key: HIVE-24065
> URL: https://issues.apache.org/jira/browse/HIVE-24065
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2020-08-05-10-05-25-080.png
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> Same bloom filter is loaded multiple times across tasks. It would be good to
> check if we can optimise this, to avoid deserializing.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (HIVE-24065) Bloom filters can be cached after deserialization in VectorInBloomFilterColDynamicValue
[ https://issues.apache.org/jira/browse/HIVE-24065?focusedWorklogId=474174=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-474174 ] ASF GitHub Bot logged work on HIVE-24065: - Author: ASF GitHub Bot Created on: 25/Aug/20 07:44 Start Date: 25/Aug/20 07:44 Worklog Time Spent: 10m Work Description: abstractdog commented on pull request #1423: URL: https://github.com/apache/hive/pull/1423#issuecomment-679861897 @rbalamohan : could you please take a look? simple patch, tested on cluster This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 474174) Time Spent: 20m (was: 10m) > Bloom filters can be cached after deserialization in > VectorInBloomFilterColDynamicValue > --- > > Key: HIVE-24065 > URL: https://issues.apache.org/jira/browse/HIVE-24065 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Labels: pull-request-available > Attachments: image-2020-08-05-10-05-25-080.png > > Time Spent: 20m > Remaining Estimate: 0h > > Same bloom filter is loaded multiple times across tasks. It would be good to > check if we can optimise this, to avoid deserializing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (HIVE-24065) Bloom filters can be cached after deserialization in VectorInBloomFilterColDynamicValue
[ https://issues.apache.org/jira/browse/HIVE-24065?focusedWorklogId=473920=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-473920 ] ASF GitHub Bot logged work on HIVE-24065: - Author: ASF GitHub Bot Created on: 24/Aug/20 14:49 Start Date: 24/Aug/20 14:49 Worklog Time Spent: 10m Work Description: abstractdog opened a new pull request #1423: URL: https://github.com/apache/hive/pull/1423 Change-Id: I311f131c03392618cc2dac186e7e53a48ede1eb4 ### What changes were proposed in this pull request? As the title suggests, expensive bloom filter deserialization can be eliminated by caching the bloom filters. This way, only 1 filter instance per daemon (or container in container mode) will be present. ### Why are the changes needed? Performance improvement. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Tested on cluster. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 473920) Remaining Estimate: 0h Time Spent: 10m > Bloom filters can be cached after deserialization in > VectorInBloomFilterColDynamicValue > --- > > Key: HIVE-24065 > URL: https://issues.apache.org/jira/browse/HIVE-24065 > Project: Hive > Issue Type: Improvement >Reporter: László Bodor >Assignee: László Bodor >Priority: Major > Attachments: image-2020-08-05-10-05-25-080.png > > Time Spent: 10m > Remaining Estimate: 0h > > Same bloom filter is loaded multiple times across tasks. It would be good to > check if we can optimise this, to avoid deserializing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
