joddiy commented on issue #807: URL: https://github.com/apache/singa/issues/807#issuecomment-708222390
> @joddiy Thanks for your patch PR at #808, another failure problem occurs(as you mentioned above): > > ``` > $ cd examples/onnx > $ python3 training/train.py --model vgg16 --data cifar10 --bs 1 > ``` > > the full log is: > > ``` > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream:: SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 553512191 > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream:: SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 553512191 > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream:: SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 553512191 > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream:: SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 553514489 > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:537] Reading dangerously large protocol message. If the message turns out to be larger than 2147483647 bytes, parsing will be halted for security reasons. To increase the limit (or to disable these warnings), see CodedInputStream:: SetTotalBytesLimit() in google/protobuf/io/coded_stream.h. > [libprotobuf WARNING google/protobuf/io/coded_stream.cc:78] The total number of bytes read was 553514491 > ^M 0%| | 0/50000 [00:00<?, ?it/s]WARNING: Logging before InitGoogleLogging() is written to STDERR > F1014 15:29:10.526221 23807 cuda_gpu.cc:207] Check failed: error == cudaSuccess (700 vs. 0) an illegal memory access was encountered > *** Check failure stack trace: *** > ``` > > env: > singa version: 3100(the latest build from the source code of master branch) > Python version: 3.5.2 > ONNX version: 1.5.0 > > > @lijiansong @pinpom > > it should be fixed at this PR: #808 I found this issue yesterday, however, this is caused by GPU memory, @dcslin has any idea? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
