[jira] [Assigned] (ARROW-5533) [Plasma] Plasma client should be thread-safe
[ https://issues.apache.org/jira/browse/ARROW-5533?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu reassigned ARROW-5533: Assignee: Zhijun Fu > [Plasma] Plasma client should be thread-safe > > > Key: ARROW-5533 > URL: https://issues.apache.org/jira/browse/ARROW-5533 > Project: Apache Arrow > Issue Type: Improvement > Components: C++ - Plasma >Reporter: Zhijun Fu >Assignee: Zhijun Fu >Priority: Major > > [https://github.com/ray-project/ray/pull/4922#discussion_r291422782] > Make plasma client thread safe, so it's guaranteed when the PlasmaBuffer for > an object destructs, the call to plasma_client.Release(object_id) is properly > protected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ARROW-5142) [CI] Fix conda calls in AppVeyor scripts
[
https://issues.apache.org/jira/browse/ARROW-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijun Fu reassigned ARROW-5142:
Assignee: (was: Zhijun Fu)
> [CI] Fix conda calls in AppVeyor scripts
>
>
> Key: ARROW-5142
> URL: https://issues.apache.org/jira/browse/ARROW-5142
> Project: Apache Arrow
> Issue Type: Bug
> Components: Continuous Integration
>Affects Versions: 0.13.0
>Reporter: Antoine Pitrou
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.14.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Our {{.bat}} files for AppVeyor CI use "{{conda}}" to execute Conda commands,
> unfortunately those exit the {{.bat}} script immediately after returning. We
> need to invoke Conda using "{{call conda}}" instead. However, doing so
> exposes a bug in the wheel tests that went unnoticed.
> See https://github.com/apache/arrow/pull/4015
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-5533) [Plasma] Plasma client should be thread-safe
Zhijun Fu created ARROW-5533: Summary: [Plasma] Plasma client should be thread-safe Key: ARROW-5533 URL: https://issues.apache.org/jira/browse/ARROW-5533 Project: Apache Arrow Issue Type: Improvement Components: C++ - Plasma Reporter: Zhijun Fu [https://github.com/ray-project/ray/pull/4922#discussion_r291422782] Make plasma client thread safe, so it's guaranteed when the PlasmaBuffer for an object destructs, the call to plasma_client.Release(object_id) is properly protected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (ARROW-5142) [CI] Fix conda calls in AppVeyor scripts
[
https://issues.apache.org/jira/browse/ARROW-5142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijun Fu reassigned ARROW-5142:
Assignee: Zhijun Fu
> [CI] Fix conda calls in AppVeyor scripts
>
>
> Key: ARROW-5142
> URL: https://issues.apache.org/jira/browse/ARROW-5142
> Project: Apache Arrow
> Issue Type: Bug
> Components: Continuous Integration
>Affects Versions: 0.13.0
>Reporter: Antoine Pitrou
>Assignee: Zhijun Fu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.14.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Our {{.bat}} files for AppVeyor CI use "{{conda}}" to execute Conda commands,
> unfortunately those exit the {{.bat}} script immediately after returning. We
> need to invoke Conda using "{{call conda}}" instead. However, doing so
> exposes a bug in the wheel tests that went unnoticed.
> See https://github.com/apache/arrow/pull/4015
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-5402) [Plasma] Pin objects in plasma store
Zhijun Fu created ARROW-5402:
Summary: [Plasma] Pin objects in plasma store
Key: ARROW-5402
URL: https://issues.apache.org/jira/browse/ARROW-5402
Project: Apache Arrow
Issue Type: Improvement
Reporter: Zhijun Fu
Assignee: Zhijun Fu
[https://github.com/apache/arrow/issues/4368]
Sometimes we want to "pin" an object in plasma store - we don't want this
object to be deleted even though there's nobody that's currently referencing
it. In this case, we can specify a flag when creating the object so that it
won't be deleted by LRU cache when its refcnt drops to 0, and can only be
deleted by an explicit {{Delete()}} call.
Currently, we found that an actor FO problem. The actor creation task depends
on a plasma object put by user. After the the actor running for a long time,
the object will be deleted by plasma LRU. Then, an Actor FO happens, the
creation task cannot find the object put by user, so the FO is hanging forever.
Would this make sense to you?
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (ARROW-4711) [Plasma] enhance plasma client interfaces to work with multiple objects
Zhijun Fu created ARROW-4711: Summary: [Plasma] enhance plasma client interfaces to work with multiple objects Key: ARROW-4711 URL: https://issues.apache.org/jira/browse/ARROW-4711 Project: Apache Arrow Issue Type: Improvement Components: C++ - Plasma Reporter: Zhijun Fu Right now the Delete() interface in plasma client supports deleting multiple objects in a single shot, so that we can save IPCs (inter-processing communication) between plasma clients and plasma store. This reduces latency for plasma clients, and also improve the actual throughput for plasma store. I made a simple prototype for changing Release() function as well, when batching release 10 objects in a single IPC, it only takes about 1/6 of the time compared with using 10 separate IPCs. Also from profiling, processing IPCs takes a lot of CPU cycles in plasma store currently, as UNIX domain socket processing needs to go through kernel, thus batching multiple IPCs into a single one should greatly improve the plasma store performance as well. This change mostly applies to Release(), Seal(), and Create() interfaces. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4418) [Plasma] replace event loop with boost::asio for plasma store
[ https://issues.apache.org/jira/browse/ARROW-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757902#comment-16757902 ] Zhijun Fu commented on ARROW-4418: -- [~suquark] yes I think that's probably a good idea. It should be simpler than that as plasma store only deals with local UNIX domain socket. > [Plasma] replace event loop with boost::asio for plasma store > - > > Key: ARROW-4418 > URL: https://issues.apache.org/jira/browse/ARROW-4418 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Original text: > It would be nice to move plasma store from current event loop to boost::asio > to modernize the code, and more importantly to benefit from the > functionalities provided by asio, which I think also provides opportunities > for performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4418) [Plasma] replace event loop with boost::asio for plasma store
[ https://issues.apache.org/jira/browse/ARROW-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756803#comment-16756803 ] Zhijun Fu commented on ARROW-4418: -- [~robertnishihara], yes totally agreed that the move from event loop to asio should be a straightforward change. The multithread optimization can be evaluated after that, and see if it can provide non-trivial performance improvements. > [Plasma] replace event loop with boost::asio for plasma store > - > > Key: ARROW-4418 > URL: https://issues.apache.org/jira/browse/ARROW-4418 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Original text: > It would be nice to move plasma store from current event loop to boost::asio > to modernize the code, and more importantly to benefit from the > functionalities provided by asio, which I think also provides opportunities > for performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (ARROW-4418) [Plasma] replace event loop with boost::asio for plasma store
[ https://issues.apache.org/jira/browse/ARROW-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755804#comment-16755804 ] Zhijun Fu edited comment on ARROW-4418 at 1/30/19 8:12 AM: --- In addition to above benefits that Robert mentioned, asio also provides opportunities for performance improvements, by providing io service, thread pool .etc. In our internal testing, which uses 10+ actors on a single machine, I found 50% of plasma store CPU are spent on receiving messages from plasma clients, using UNIX domain socket. I'm thinking that one way to improve perf is like this: * Use a pool of threads to receive messages from clients. To ensure correct behavior, we can bind a boost::strand to a single client, so that all the messages from a given client arrives in order. As this part is CPU consumings, using multiple threads is going to help. * After this, the messages are posted into io service of main thread, which calls ProcessMessages for each of them in order. * After this, post the replies to a pool of threads, again use boost::strand for each plasma client to ensure correct order. I'm thinking this would probably help on cases where there are multiple workers using plasma store on the same machine, which should be very common. And it seems implementing this would be hard without asio functionalities. was (Author: zhijunfu): In addition to above benefits that Robert mentioned, asio also provides opportunities for performance improvements, by providing io service, thread pool .etc. In our internal testing, which uses 10+ actors on a single machine, I found 50% of plasma store CPU are spent on receiving messages from plasma clients, using UNIX domain socket. I'm thinking that one way to improve perf is like this: * Use a pool of threads to receive messages from clients. To ensure correct behavior, we can bind a boost::strand to a single client, so that all the messages from a given client arrives in order. As this part is CPU consumings, using multiple threads is going to help. * After this, the messages are posted into io service of main thread, which calls ProcessMessages for each of them in order. * After this, post the replies to a pool of threads, again use boost::strand for each plasma client to ensure correct order. I'm thinking this would probably help on cases where there are multiple workers using plasma store on the same machine, which should be very common. And it seems implementing this would be hard without asio functionalities. Thoughts? > [Plasma] replace event loop with boost::asio for plasma store > - > > Key: ARROW-4418 > URL: https://issues.apache.org/jira/browse/ARROW-4418 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Original text: > It would be nice to move plasma store from current event loop to boost::asio > to modernize the code, and more importantly to benefit from the > functionalities provided by asio, which I think also provides opportunities > for performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-4418) [Plasma] replace event loop with boost::asio for plasma store
[ https://issues.apache.org/jira/browse/ARROW-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755804#comment-16755804 ] Zhijun Fu commented on ARROW-4418: -- In addition to above benefits that Robert mentioned, asio also provides opportunities for performance improvements, by providing io service, thread pool .etc. In our internal testing, which uses 10+ actors on a single machine, I found 50% of plasma store CPU are spent on receiving messages from plasma clients, using UNIX domain socket. I'm thinking that one way to improve perf is like this: * Use a pool of threads to receive messages from clients. To ensure correct behavior, we can bind a boost::strand to a single client, so that all the messages from a given client arrives in order. As this part is CPU consumings, using multiple threads is going to help. * After this, the messages are posted into io service of main thread, which calls ProcessMessages for each of them in order. * After this, post the replies to a pool of threads, again use boost::strand for each plasma client to ensure correct order. I'm thinking this would probably help on cases where there are multiple workers using plasma store on the same machine, which should be very common. And it seems implementing this would be hard without asio functionalities. Thoughts? > [Plasma] replace event loop with boost::asio for plasma store > - > > Key: ARROW-4418 > URL: https://issues.apache.org/jira/browse/ARROW-4418 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > Original text: > It would be nice to move plasma store from current event loop to boost::asio > to modernize the code, and more importantly to benefit from the > functionalities provided by asio, which I think also provides opportunities > for performance improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-4015) [Plasma] remove legacy interfaces for plasma manager
Zhijun Fu created ARROW-4015: Summary: [Plasma] remove legacy interfaces for plasma manager Key: ARROW-4015 URL: https://issues.apache.org/jira/browse/ARROW-4015 Project: Apache Arrow Issue Type: Improvement Components: Plasma (C++) Reporter: Zhijun Fu Assignee: Zhijun Fu [https://github.com/apache/arrow/issues/3154] In legacy ray, interacting with remote plasma stores is done via plasma manager, which is part of ray, and plasma has a few interfaces to support it - namely Fetch() and Wait(). Currently the legacy ray code has already been removed, and the new raylet uses object manager to interface with remote machine, and these legacy plasma interfaces are no longer used. I think we could remove these legacy interfaces to cleanup code and avoid confusion. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2551) [Plasma] Improve notification logic
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16504230#comment-16504230 ] Zhijun Fu commented on ARROW-2551: -- [~ovidiumarcu] Sorry for the late reply. Good to know that it fixed your issue:) For the order of notifications, there are two cases: 1) When a client subscribes to plasma store, the store will push notifications for all existing objects to the new client, in this case there's *no specific order* among these notifications - the store simply loop through all the objects from an unordered_map and push notification for each of them. 2) When a new object is sealed, a notification is pushed to all the clients that have subscribed for notifications. In this case, the object notifications are sent in the order of their *sealed* time, instead of creation time. > [Plasma] Improve notification logic > --- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 2.5h > Remaining Estimate: 0h > > There are a few issues in current Plasma notification code: > * When a client subscribes to Plasma, the store pushes notifications about > existing objects to ALL subscribers, while it should only push to the new > subscriber. > * And in the above scenario, it should only push "sealed" objects to the new > subscriber, while currently it pushes all objects regardless of the state. > * When a client disconnects, it will no longer be able to receive > notifications, thus the NotificationQueue for the client should be removed > from global map. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2597) [Plasma] remove UniqueIDHasher
Zhijun Fu created ARROW-2597: Summary: [Plasma] remove UniqueIDHasher Key: ARROW-2597 URL: https://issues.apache.org/jira/browse/ARROW-2597 Project: Apache Arrow Issue Type: Improvement Components: Plasma (C++) Reporter: Zhijun Fu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-2551) [Plasma] Improve notification logic
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu updated ARROW-2551: - Description: There are a few issues in current Plasma notification code: * When a client subscribes to Plasma, the store pushes notifications about existing objects to ALL subscribers, while it should only push to the new subscriber. * And in the above scenario, it should only push "sealed" objects to the new subscriber, while currently it pushes all objects regardless of the state. * When a client disconnects, it will no longer be able to receive notifications, thus the NotificationQueue for the client should be removed from global map. was:There are two issues in current Plasma notification code > [Plasma] Improve notification logic > --- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > There are a few issues in current Plasma notification code: > * When a client subscribes to Plasma, the store pushes notifications about > existing objects to ALL subscribers, while it should only push to the new > subscriber. > * And in the above scenario, it should only push "sealed" objects to the new > subscriber, while currently it pushes all objects regardless of the state. > * When a client disconnects, it will no longer be able to receive > notifications, thus the NotificationQueue for the client should be removed > from global map. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-2551) [Plasma] Improve notification logic
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu updated ARROW-2551: - Summary: [Plasma] Improve notification logic (was: [Plasma] Improve notification related code) > [Plasma] Improve notification logic > --- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > There are two issues in current Plasma notification code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-2551) [Plasma] Improve notification related code
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu updated ARROW-2551: - Description: There are two issues in current Plasma notification code > [Plasma] Improve notification related code > -- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > There are two issues in current Plasma notification code -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-2556) [Plasma] plasma pushes notifications to all subscribers when a new client subscribes
[ https://issues.apache.org/jira/browse/ARROW-2556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu closed ARROW-2556. Resolution: Duplicate see ARROW-2551 > [Plasma] plasma pushes notifications to all subscribers when a new client > subscribes > > > Key: ARROW-2556 > URL: https://issues.apache.org/jira/browse/ARROW-2556 > Project: Apache Arrow > Issue Type: Bug > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > When a new client subscribes to plasma store, plasma would push notifications > about existing objects to the new subscriber - while the current code pushes > notifications to ALL subscribers in this case which is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-2559) [Plasma] delete object notification queue for a client when it disconnects with plasma
[ https://issues.apache.org/jira/browse/ARROW-2559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu closed ARROW-2559. Resolution: Duplicate see ARROW-2551 > [Plasma] delete object notification queue for a client when it disconnects > with plasma > -- > > Key: ARROW-2559 > URL: https://issues.apache.org/jira/browse/ARROW-2559 > Project: Apache Arrow > Issue Type: Improvement >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (ARROW-2551) [Plasma] Improve notification related code
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu reopened ARROW-2551: -- > [Plasma] Improve notification related code > -- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2558) [Plasma] avoid walk through all the objects when a client disconnects
Zhijun Fu created ARROW-2558: Summary: [Plasma] avoid walk through all the objects when a client disconnects Key: ARROW-2558 URL: https://issues.apache.org/jira/browse/ARROW-2558 Project: Apache Arrow Issue Type: Improvement Components: Plasma (C++) Reporter: Zhijun Fu Currently plasma stores list-of-clients in ObjectTableEntry, which is used to track which clients are using a given object, this serves for two purposes: - If an object is in use. - If the client trying to abort an object is the one who created it. A problem with list-of-clients approach is that when a client disconnects, we need to walk through all the objects and remove the client pointer from the list for each object. Instead, we could add a reference count in ObjectTableEntry, and store list-of-object-ids in client structure. This could both goals that the original approach is targeting, while when a client disconnects, it just walk through its object-ids and dereference each ObjectTableEntry, there's no need to walk through all objects. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2556) [Plasma] plasma pushes notifications to all subscribers when a new client subscribes
Zhijun Fu created ARROW-2556: Summary: [Plasma] plasma pushes notifications to all subscribers when a new client subscribes Key: ARROW-2556 URL: https://issues.apache.org/jira/browse/ARROW-2556 Project: Apache Arrow Issue Type: Bug Components: Plasma (C++) Reporter: Zhijun Fu When a new client subscribes to plasma store, plasma would push notifications about existing objects to the new subscriber - while the current code pushes notifications to ALL subscribers in this case which is unnecessary. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-2551) [Plasma] Improve notification related code
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu updated ARROW-2551: - Description: (was: https://github.com/apache/arrow/pull/2011 This change addresses two issues in plasma notification code: # Currently plasma store uses recv_fd to receive the notification fd from client. As mentioned in comments from the code, this could cause plasma store to hang if client doesn't call send_fd. This particular change puts the notification fd in SubscriptionRequest to solve the problem. # When a client subscribes to plasma store, the store pushes notifications of existing objects to ALL subscribers - while in this case it just needs to push to the new subscriber. This change adds a new interface to push to a particular client.) > [Plasma] Improve notification related code > -- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (ARROW-2551) [Plasma] Improve notification related code
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu updated ARROW-2551: - Comment: was deleted (was: PR is here: https://github.com/apache/arrow/pull/2011) > [Plasma] Improve notification related code > -- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > https://github.com/apache/arrow/pull/2011 > This change addresses two issues in plasma notification code: > # Currently plasma store uses recv_fd to receive the notification fd from > client. As mentioned in comments from the code, this could cause plasma store > to hang if client doesn't call send_fd. This particular change puts the > notification fd in SubscriptionRequest to solve the problem. > # When a client subscribes to plasma store, the store pushes notifications > of existing objects to ALL subscribers - while in this case it just needs to > push to the new subscriber. This change adds a new interface to push to a > particular client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (ARROW-2551) [Plasma] Improve notification related code
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu closed ARROW-2551. Resolution: Invalid > [Plasma] Improve notification related code > -- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (ARROW-2551) [Plasma] Improve notification related code
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhijun Fu updated ARROW-2551: - Description: https://github.com/apache/arrow/pull/2011 This change addresses two issues in plasma notification code: # Currently plasma store uses recv_fd to receive the notification fd from client. As mentioned in comments from the code, this could cause plasma store to hang if client doesn't call send_fd. This particular change puts the notification fd in SubscriptionRequest to solve the problem. # When a client subscribes to plasma store, the store pushes notifications of existing objects to ALL subscribers - while in this case it just needs to push to the new subscriber. This change adds a new interface to push to a particular client. was: This change addresses two issues in plasma notification code: # Currently plasma store uses recv_fd to receive the notification fd from client. As mentioned in comments from the code, this could cause plasma store to hang if client doesn't call send_fd. This particular change puts the notification fd in SubscriptionRequest to solve the problem. # When a client subscribes to plasma store, the store pushes notifications of existing objects to ALL subscribers - while in this case it just needs to push to the new subscriber. This change adds a new interface to push to a particular client. > [Plasma] Improve notification related code > -- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > https://github.com/apache/arrow/pull/2011 > This change addresses two issues in plasma notification code: > # Currently plasma store uses recv_fd to receive the notification fd from > client. As mentioned in comments from the code, this could cause plasma store > to hang if client doesn't call send_fd. This particular change puts the > notification fd in SubscriptionRequest to solve the problem. > # When a client subscribes to plasma store, the store pushes notifications > of existing objects to ALL subscribers - while in this case it just needs to > push to the new subscriber. This change adds a new interface to push to a > particular client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2551) [Plasma] Improve notification related code
[ https://issues.apache.org/jira/browse/ARROW-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467057#comment-16467057 ] Zhijun Fu commented on ARROW-2551: -- PR is here: https://github.com/apache/arrow/pull/2011 > [Plasma] Improve notification related code > -- > > Key: ARROW-2551 > URL: https://issues.apache.org/jira/browse/ARROW-2551 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This change addresses two issues in plasma notification code: > # Currently plasma store uses recv_fd to receive the notification fd from > client. As mentioned in comments from the code, this could cause plasma store > to hang if client doesn't call send_fd. This particular change puts the > notification fd in SubscriptionRequest to solve the problem. > # When a client subscribes to plasma store, the store pushes notifications > of existing objects to ALL subscribers - while in this case it just needs to > push to the new subscriber. This change adds a new interface to push to a > particular client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2551) [Plasma] Improve notification related code
Zhijun Fu created ARROW-2551: Summary: [Plasma] Improve notification related code Key: ARROW-2551 URL: https://issues.apache.org/jira/browse/ARROW-2551 Project: Apache Arrow Issue Type: Improvement Components: Plasma (C++) Reporter: Zhijun Fu This change addresses two issues in plasma notification code: # Currently plasma store uses recv_fd to receive the notification fd from client. As mentioned in comments from the code, this could cause plasma store to hang if client doesn't call send_fd. This particular change puts the notification fd in SubscriptionRequest to solve the problem. # When a client subscribes to plasma store, the store pushes notifications of existing objects to ALL subscribers - while in this case it just needs to push to the new subscriber. This change adds a new interface to push to a particular client. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2540) [Plasma] add constructor/destructor to make sure dlfree is called automatically
[ https://issues.apache.org/jira/browse/ARROW-2540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463558#comment-16463558 ] Zhijun Fu commented on ARROW-2540: -- PR is here: https://github.com/apache/arrow/pull/1996 > [Plasma] add constructor/destructor to make sure dlfree is called > automatically > --- > > Key: ARROW-2540 > URL: https://issues.apache.org/jira/browse/ARROW-2540 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > Add constructor & destructor to ObjectTableEntry structure to make sure > dlfree() is called for the pointer field when the object gets destructed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2540) [Plasma] add constructor/destructor to make sure dlfree is called automatically
Zhijun Fu created ARROW-2540: Summary: [Plasma] add constructor/destructor to make sure dlfree is called automatically Key: ARROW-2540 URL: https://issues.apache.org/jira/browse/ARROW-2540 Project: Apache Arrow Issue Type: Improvement Components: Plasma (C++) Reporter: Zhijun Fu Add constructor & destructor to ObjectTableEntry structure to make sure dlfree() is called for the pointer field when the object gets destructed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-2539) [Plasma] Use unique_ptr instead of raw pointer
[ https://issues.apache.org/jira/browse/ARROW-2539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16463537#comment-16463537 ] Zhijun Fu commented on ARROW-2539: -- I've created an PR for this: [https://github.com/apache/arrow/pull/1993] > [Plasma] Use unique_ptr instead of raw pointer > -- > > Key: ARROW-2539 > URL: https://issues.apache.org/jira/browse/ARROW-2539 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Zhijun Fu >Priority: Minor > > There are some places in Plasma where explicit new & delete are used, > forgetting to delete can cause memory leak. Use unique_ptr instead when > possible so that memory gets deleted automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (ARROW-2539) [Plasma] Use unique_ptr instead of raw pointer
Zhijun Fu created ARROW-2539: Summary: [Plasma] Use unique_ptr instead of raw pointer Key: ARROW-2539 URL: https://issues.apache.org/jira/browse/ARROW-2539 Project: Apache Arrow Issue Type: Improvement Components: Plasma (C++) Reporter: Zhijun Fu There are some places in Plasma where explicit new & delete are used, forgetting to delete can cause memory leak. Use unique_ptr instead when possible so that memory gets deleted automatically. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
