Thanks Michael,
syntax = "proto3";
service FileServer {
rpc upload(stream Chunk) returns (Reply) {}
rpc download(Request) returns (stream Chunk) {}
}
message Chunk {
bytes buffer = 1;
}
message Request {
string name = 1;
}
message Reply {
int32 length = 1;
and my client.py
def get_file_chunks(filename):
with open(filename, 'rb') as f:
while True:
piece = f.read(CHUNK_SIZE);
if len(piece) == 0:
return
yield chunk_pb2.Chunk(buffer=piece)
def save_chunks_to_file(chunks, filename):
with open(filename, 'wb') as f:
for chunk in chunks:
f.write(chunk.buffer)
class FileClient:
def __init__(self, address):
channel = grpc.insecure_channel(address)
self.stub = chunk_pb2_grpc.FileServerStub(channel)
def upload(self, in_file_name):
chunks_generator = get_file_chunks(in_file_name)
response = self.stub.upload(chunks_generator)
assert response.length == os.path.getsize(in_file_name)
def download(self, target_name, out_file_name):
response = self.stub.download(chunk_pb2.Request(name=target_name))
save_chunks_to_file(response, out_file_name)
if __name__ == '__main__':
client = FileClient('localhost:8888')
# demo for file uploading
in_file_name = sys.argv[1]
client.upload(in_file_name)
On Tuesday, April 28, 2020 at 12:13:47 AM UTC+5:30, Michael Webster wrote:
>
> Do you have any snippets you can share from your code? I use python-grpc
> in another project for transferring files and I've never had an issue with
> files > 2gb, with 1mb chunks (local network operations).
>
> On Monday, April 27, 2020 at 7:08:50 PM UTC+1, VigneshDhanraj G wrote:
>>
>> Thanks Mya,
>>
>> There is no any error on the server and i am getting error if i upload
>> more than 2GB file, i am constanly uploading chunk on the storge.
>>
>> On Monday, April 27, 2020 at 8:42:50 PM UTC+5:30, Mya Pitzeruse wrote:
>>>
>>> Is there an associated error on the server? The error message suggests a
>>> server error with an index out of bounds exception: 2^31. Only way I could
>>> see something like that happening is if the file was being buffered on the
>>> server side in memory.
>>>
>>> On Mon, Apr 27, 2020 at 6:56 AM VigneshDhanraj G <[email protected]>
>>> wrote:
>>>
>>>> Hi Team,
>>>>
>>>> I am not able to upload large files more than 2GB even after chunk. I
>>>> have found the file is uploaded still exeception raised? Please help me to
>>>> understand the issue here and way to fix this problem.
>>>>
>>>> Traceback (most recent call last):
>>>> File "demo_client.py", line 12, in <module>
>>>> client.upload(in_file_name)
>>>> File
>>>> "/home/vigneshdhanraj/Project/grpc-upload/grpc-file-transfer/src/lib.py",
>>>> line 34, in upload
>>>> response = self.stub.upload(chunks_generator)
>>>> File
>>>> "/home/vigneshdhanraj/Project/grpc-upload/myenv/lib/python3.6/site-packages/grpc/_channel.py",
>>>>
>>>> line 1011, in __call__
>>>> return _end_unary_response_blocking(state, call, False, None)
>>>> File
>>>> "/home/vigneshdhanraj/Project/grpc-upload/myenv/lib/python3.6/site-packages/grpc/_channel.py",
>>>>
>>>> line 729, in _end_unary_response_blocking
>>>> raise _InactiveRpcError(state)
>>>> grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that
>>>> terminated with:
>>>> status = StatusCode.UNKNOWN
>>>> details = "*Exception calling application: Value out of range:
>>>> 2147483648*"
>>>> debug_error_string =
>>>> "{"created":"@1587649701.194973268","description":"Error received from
>>>> peer
>>>> ipv6:[::1]:8888","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Exception
>>>>
>>>> calling application: Value out of range: 2147483648","grpc_status":2}"
>>>>
>>>> Regards,
>>>> VigneshDhanraj G
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "grpc.io" group.
>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To view this discussion on the web visit
>>>> https://groups.google.com/d/msgid/grpc-io/cbc945c2-6d3a-4d2b-a52f-2a6b59a9dfba%40googlegroups.com
>>>>
>>>> <https://groups.google.com/d/msgid/grpc-io/cbc945c2-6d3a-4d2b-a52f-2a6b59a9dfba%40googlegroups.com?utm_medium=email&utm_source=footer>
>>>> .
>>>>
>>>
>>>
>>> --
>>>
>>> Mya Pitzeruse
>>>
>>> Principal Software Engineer - Service Infrastructure
>>>
>>> Gender Pronouns: She, Her, Hers
>>>
>>> [email protected]
>>>
>>>
>>> Indeed - We help people get jobs.
>>>
>>> Indeed.com <http://www.indeed.com/>
>>>
>>> Facebook <http://www.facebook.com/indeed> | Twitter
>>> <http://www.twitter.com/indeed> | Instagram
>>> <http://www.instagram.com/indeedworks>
>>>
>>
--
You received this message because you are subscribed to the Google Groups
"grpc.io" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To view this discussion on the web visit
https://groups.google.com/d/msgid/grpc-io/e3f6c515-71cd-40ca-bafb-b409788cb1fb%40googlegroups.com.
