Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/5580
I've addressed @zentol last comments and rebased to fix conflicts. Will
merge after travis gives green.
---
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/5580
@zentol I've rebased this PR. Feel free to review whenever you find a while.
---
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/5580
Thanks @zentol for a review.
You are right not supporting it in old cluster mode would be a regression.
As I've first discussed it, there were some doubts if it could work in old mode
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5580
@zentol What do you think we still need now? I think this makes things
easier for users. And we also need this for work on the Beam Flink Runner.
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5580
I think that's a very good solution in the end that even simplifies things.
What do you think, @zentol ?
---
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/5580
@aljoscha In the end I've implemented uploading directories as zip
archives, as @zentol suggested. End-to-end python tests passes. Let me know
what you both think.
---
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5580
Ok, to unblock this, I would suggest to add the new functionality under a
different name, support only adding a single file (and also verify that the
path passed in is a single file). @dawidwys
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/5580
I am not suggesting dropping uploading directories, but rather gradually
switch to blob server. Leave the possibility to distribute files via DFS (and
leave Python using it) and only for 1.6.0 drop
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5580
Not supporting directories would be a regression regardless of whether the
Python API needs it or not.
We either have to add a new method with different behavior, or zip
directories
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/5580
@aljoscha Yes that's right. The latest commit does not work because Python
API expects complete directories to be uploaded. I think it is quite important
cause this way the uploaded Python code can
Github user aljoscha commented on the issue:
https://github.com/apache/flink/pull/5580
So the current state of this PR doesn't work with the Python API because it
uploads complete directories? Is that really used/needed by the Python API?
I would like to have this feature in
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/5580
@zentol I've tried changing the behaviour of `registerCachedFile` to always
distribute files via BlobServer, but I've stumbled across problem with python
API. Right now whole directories can be
Github user ifndef-SleePy commented on the issue:
https://github.com/apache/flink/pull/5580
@zentol All right, got your point. That's a problem indeed.
---
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/5580
@zentol I agree it will need to be changed in the end (though wouldn't say
rewritten completely) after the submission process is completed.
I think though this change is important from the
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5580
I'm aware that the `RestClusterClient` currently submits jars directly to
the blob service. If this PR would've been opened half a year ago I would've
welcomed it with open arms.
My point
Github user ifndef-SleePy commented on the issue:
https://github.com/apache/flink/pull/5580
If some day the behavior of `RestClusterClient` changes. Maybe uploading
the jars via REST API. We could just upload the user-defined files via REST API
either. That's not a big problem,
Github user ifndef-SleePy commented on the issue:
https://github.com/apache/flink/pull/5580
Hi @zentol
In Flip-6 the client communicates with cluster via REST API, that's true.
However this not include blobs. Currently in `RestClusterClient.submit()`
method, the client uses
Github user zentol commented on the issue:
https://github.com/apache/flink/pull/5580
As far as i know one of the end-goals of FLIP-6 is to have all
client-cluster communication go through the REST API. This implies that the
client cannot submit things directly to the blob service.
Github user dawidwys commented on the issue:
https://github.com/apache/flink/pull/5580
R: @aljoscha
---
19 matches
Mail list logo
