[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[ https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595231#comment-16595231 ] Wes McKinney commented on ARROW-1744: - We can definitely generate the changelog. ARROW-3132 > [Plasma] Provide TensorFlow operator to read tensors from plasma > > > Key: ARROW-1744 > URL: https://issues.apache.org/jira/browse/ARROW-1744 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Philipp Moritz >Assignee: Philipp Moritz >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > Time Spent: 17h 10m > Remaining Estimate: 0h > > see https://www.tensorflow.org/extend/adding_an_op -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[ https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595192#comment-16595192 ] Brian Hulette commented on ARROW-1744: -- [~wesmckinn] not sure if you saw this - I think the issues I mentioned above were incorrectly marked with fix version JS-0.4.0 rather than 0.10.0. I've fixed them in Jira now, but I don't think they made it into the CHANGELOG.md as a result. Is there anything we can/should do about that now? > [Plasma] Provide TensorFlow operator to read tensors from plasma > > > Key: ARROW-1744 > URL: https://issues.apache.org/jira/browse/ARROW-1744 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Philipp Moritz >Assignee: Philipp Moritz >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > Time Spent: 17h 10m > Remaining Estimate: 0h > > see https://www.tensorflow.org/extend/adding_an_op -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[ https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583295#comment-16583295 ] Brian Hulette commented on ARROW-1744: -- I think the same thing happened in ARROW-2940, ARROW-2451, ARROW-2437, ARROW-2458, and ARROW-2397 - I went ahead and updated them all. It looks like these mistakes prevented them from being added to CHANGELOG.md for v0.10.0 > [Plasma] Provide TensorFlow operator to read tensors from plasma > > > Key: ARROW-1744 > URL: https://issues.apache.org/jira/browse/ARROW-1744 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Philipp Moritz >Assignee: Philipp Moritz >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > Time Spent: 17h 10m > Remaining Estimate: 0h > > see https://www.tensorflow.org/extend/adding_an_op -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[ https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583285#comment-16583285 ] Brian Hulette commented on ARROW-1744: -- It looks like this was actually merged for 0.10 (and certainly not JS-0.4.0) - is it too late to update the fix version? > [Plasma] Provide TensorFlow operator to read tensors from plasma > > > Key: ARROW-1744 > URL: https://issues.apache.org/jira/browse/ARROW-1744 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Philipp Moritz >Assignee: Philipp Moritz >Priority: Major > Labels: pull-request-available > Fix For: JS-0.4.0 > > Time Spent: 17h 10m > Remaining Estimate: 0h > > see https://www.tensorflow.org/extend/adding_an_op -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[ https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16538719#comment-16538719 ] Wes McKinney commented on ARROW-1744: - I moved this to 0.11. If it gets merged before 0.10, that's fine, but since Ray (where this would be most immediately used) does not use Arrow's versioned releases they aren't blocking on getting this into 0.10 > [Plasma] Provide TensorFlow operator to read tensors from plasma > > > Key: ARROW-1744 > URL: https://issues.apache.org/jira/browse/ARROW-1744 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Philipp Moritz >Assignee: Philipp Moritz >Priority: Major > Labels: pull-request-available > Fix For: 0.11.0 > > Time Spent: 13h 40m > Remaining Estimate: 0h > > see https://www.tensorflow.org/extend/adding_an_op -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[ https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16383630#comment-16383630 ] ASF GitHub Bot commented on ARROW-1744: --- xhochy commented on issue #1379: ARROW-1744: [WIP] Add plasma tensorflow op URL: https://github.com/apache/arrow/pull/1379#issuecomment-369934686 @pcmoritz Any progress on this topic here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Plasma] Provide TensorFlow operator to read tensors from plasma > > > Key: ARROW-1744 > URL: https://issues.apache.org/jira/browse/ARROW-1744 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Philipp Moritz >Assignee: Philipp Moritz >Priority: Major > Labels: pull-request-available > Fix For: 0.10.0 > > > see https://www.tensorflow.org/extend/adding_an_op -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[
https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287530#comment-16287530
]
ASF GitHub Bot commented on ARROW-1744:
---
holdenk commented on a change in pull request #1379: ARROW-1744: [WIP] Add
plasma tensorflow op
URL: https://github.com/apache/arrow/pull/1379#discussion_r156349402
##
File path: cpp/src/plasma/tf/test.py
##
@@ -0,0 +1,46 @@
+import numpy as np
+import pyarrow as pa
+import pyarrow.plasma as plasma
+import tensorflow as tf
+
+import time
+
+zero_out_module = tf.load_op_library('./plasma_op.so')
+
+client = plasma.connect("/tmp/plasma", "", 64)
Review comment:
Minor comment, but maybe do a mktemp type thing here instead? I know some
folks run tests in parallel on the same machine.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Plasma] Provide TensorFlow operator to read tensors from plasma
>
>
> Key: ARROW-1744
> URL: https://issues.apache.org/jira/browse/ARROW-1744
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Plasma (C++)
>Reporter: Philipp Moritz
>Assignee: Philipp Moritz
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> see https://www.tensorflow.org/extend/adding_an_op
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[
https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16287531#comment-16287531
]
ASF GitHub Bot commented on ARROW-1744:
---
holdenk commented on a change in pull request #1379: ARROW-1744: [WIP] Add
plasma tensorflow op
URL: https://github.com/apache/arrow/pull/1379#discussion_r156349604
##
File path: cpp/src/plasma/tf/plasma_op.cc
##
@@ -0,0 +1,70 @@
+#include "tensorflow/core/framework/op.h"
+#include "tensorflow/core/framework/op_kernel.h"
+#include "tensorflow/core/framework/shape_inference.h"
+
+#include "arrow/io/memory.h"
+#include "arrow/ipc/reader.h"
+#include "arrow/tensor.h"
+#include "plasma/client.h"
+
+using namespace tensorflow;
+
+REGISTER_OP("PlasmaData")
+.Input("object_id: string")
+.Output("output: float32")
+.Attr("socket: string");
+
+// TODO(pcm): Make this zero-copy if possible
+
+class PlasmaDataOp : public OpKernel {
+ public:
+ explicit PlasmaDataOp(OpKernelConstruction* context) : OpKernel(context) {
+std::cout << "called constructor" << std::endl;
+std::string socket;
+OP_REQUIRES_OK(context, context->GetAttr("socket", ));
+// Connect to plasma
+ARROW_CHECK_OK(client_.Connect(socket, "", PLASMA_DEFAULT_RELEASE_DELAY));
+std::cout << "constructor finished" << std::endl;
+ }
+
+ void Compute(OpKernelContext* context) override {
+// Grab the input tensor
+const Tensor& input_tensor = context->input(0);
+auto input = input_tensor.flat();
+
+// Get the object
+plasma::ObjectID object_id = plasma::ObjectID::from_binary(input(0));
+plasma::ObjectBuffer object_buffer;
+ARROW_CHECK_OK(client_.Get(_id, 1, -1, _buffer));
+
+// Get the tensor
+std::shared_ptr result;
+arrow::io::BufferReader reader(object_buffer.data,
object_buffer.data_size);
+int64_t offset;
+ARROW_CHECK_OK(reader.Tell());
+ARROW_CHECK_OK(arrow::ipc::ReadTensor(0, , ));
+
+std::cout << "shape is" << result->shape()[0] << " , " <<
result->shape()[1]
+ << std::endl;
+
+// Create an output tensor
+TensorShape shape(result->shape());
+Tensor* output_tensor = NULL;
+OP_REQUIRES_OK(context, context->allocate_output(0, shape,
_tensor));
+auto output_flat = output_tensor->flat();
+
+// Set all but the first element of the output tensor to 0.
Review comment:
Is this comment perhaps about an older version of the code?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> [Plasma] Provide TensorFlow operator to read tensors from plasma
>
>
> Key: ARROW-1744
> URL: https://issues.apache.org/jira/browse/ARROW-1744
> Project: Apache Arrow
> Issue Type: Improvement
> Components: Plasma (C++)
>Reporter: Philipp Moritz
>Assignee: Philipp Moritz
> Labels: pull-request-available
> Fix For: 0.9.0
>
>
> see https://www.tensorflow.org/extend/adding_an_op
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[ https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274910#comment-16274910 ] Philipp Moritz commented on ARROW-1744: --- [~holdenk] I submitted a WIP version of what I have so far; it works but needs some profiling. Unfortunately I won't have time to work on it before next week, feel free to try it out and let me know what you think/keep developing it in the meantime. > [Plasma] Provide TensorFlow operator to read tensors from plasma > > > Key: ARROW-1744 > URL: https://issues.apache.org/jira/browse/ARROW-1744 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Philipp Moritz >Assignee: Philipp Moritz > Labels: pull-request-available > > see https://www.tensorflow.org/extend/adding_an_op -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[ https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274905#comment-16274905 ] ASF GitHub Bot commented on ARROW-1744: --- pcmoritz opened a new pull request #1379: ARROW-1744: [WIP] Add plasma tensorflow op URL: https://github.com/apache/arrow/pull/1379 Before this can be merged, it needs some profiling and comparison to other ways of loading data in TensorFlow. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > [Plasma] Provide TensorFlow operator to read tensors from plasma > > > Key: ARROW-1744 > URL: https://issues.apache.org/jira/browse/ARROW-1744 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Philipp Moritz >Assignee: Philipp Moritz > Labels: pull-request-available > > see https://www.tensorflow.org/extend/adding_an_op -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (ARROW-1744) [Plasma] Provide TensorFlow operator to read tensors from plasma
[ https://issues.apache.org/jira/browse/ARROW-1744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16274849#comment-16274849 ] holdenk commented on ARROW-1744: Are you working on this [~pcmoritz]? > [Plasma] Provide TensorFlow operator to read tensors from plasma > > > Key: ARROW-1744 > URL: https://issues.apache.org/jira/browse/ARROW-1744 > Project: Apache Arrow > Issue Type: Improvement > Components: Plasma (C++) >Reporter: Philipp Moritz >Assignee: Philipp Moritz > > see https://www.tensorflow.org/extend/adding_an_op -- This message was sent by Atlassian JIRA (v6.4.14#64029)
