Re: Django broken pipe error

[justin walters]

On Mon, Jan 2, 2017 at 6:14 AM,  wrote:

>
> Thanks a lot Justin,
>
> The problem was solved when I employed standard Framework methods for
> creation of new database object:
>
> in JS:
> var trendModel = new App.TrendModel();
> trendModel.set("phrase", search_phrase);
> trendModel.set("from_time", time_from);
> trendModel.set(...
> trendModel.save(...
>
> in PY:
> def create(self, request):
> ...
>
> I've also created extra template and additional View. PageView for some
> unclear reason didn't support creation of new object - this event just
> disappeared.
>
> Regards,
>
> Roman
> --
> https://mail.python.org/mailman/listinfo/python-list
>

Glad to hear that you solved the problem!
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Django broken pipe error

[dr . roman . graf]

On Monday, December 12, 2016 at 6:38:39 PM UTC+1, justin walters wrote:
> On Mon, Dec 12, 2016 at 7:27 AM, roma  wrote:
> 
> > Thanks Justin,
> >
> > I believe, the whole database story has no influence on the broken pipe
> > error. I've commented out the whole block and leave only return line:
> > return HttpResponse(res, content_type="text/plain; charset=utf-8")
> > The error is still present. And I have no influence on that.
> >
> > I call python from js client:
> >
> > var newTrendReport = new App.TrendReport();
> > newTrendReport.set('search_phrase', search_phrase);
> > newTrendReport.set('time_from', time_from);
> > newTrendReport.set('time_to', time_to);
> > newTrendReport.set('time_scale', time_scale);
> > newTrendReport.set('category', category);
> > newTrendReport.startExport(
> >
> > function(response){
> > console.log("Successfully calculated trend report.");
> > App.trendPage = new App.TrendPageView();
> > App.trendPage.render(response);
> > },
> > );
> >
> > go throw:
> >
> > App.TrendReport = Backbone.Model.extend({
> > urlRoot: "/api/trend_reports/",
> > defaults: {
> > search_phrase: "",
> > time_from: "",
> > time_to: "",
> > time_scale: "",
> > category: ""
> > },
> >
> > startExportSuffix: "/export_report/",
> >
> > startExport: function( successCallback, errorCallback ) {
> > console.log("start trend calculation");
> > var that = this;
> > var ajaxUrl = this.startExportSuffix;
> > var options = {
> > method: "POST",
> > data: this.attributes,
> > contentType: "application/json;charset=UTF-8",
> > dataType: "json",
> >
> > error: errorCallback,
> > success: successCallback
> > };
> > console.log("start trend export sync");
> > App.ajax(ajaxUrl, options);
> > }
> >
> > });
> >
> > and come in export_report method.
> >
> > My urls.py:
> >
> > url(r'^export_report', ensure_csrf_cookie(views.export_report),
> > name="export_report"),
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
> 
> I'm not super familiar with the way backbone does http requests, but
> something seems off about the startExport function.
> It seems to me that you are sending a post request to "/export_report/"
> which is an endpoint that I'm guessing is nested
> in an include from "/api/trend_reports/". However, it looks like the error
> you're getting above says you aren't sending
> the request to "https://root.com/api/trend_reports/export_report/;.
> Instead, you are sending the request to "https://root.com/export_report/; .
> Though, I'm also not sure that's the case because that would normally throw
> a 404.
> 
> I also noticed that you set content type to 'application/json' in your js,
> but the view function returns a 'text/plain' content type. Is
> There a reason for this?
> 
> The data key in your js http function is set to the attributes variable
> which, as far as I can tell, does not exist.
> 
> There's a lot going on here, but I think you can probably narrow it down to
> something in your backbone code or the way
> backbone handles http requests as the error you are getting is caused by
> the client prematurely closing the socket.
> It's possible backbone will stop reading the response since it isn't the
> same content-type as the request.

Thanks a lot Justin,

The problem was solved when I employed standard Framework methods for creation 
of new database object:

in JS:
var trendModel = new App.TrendModel();
trendModel.set("phrase", search_phrase);
trendModel.set("from_time", time_from);
trendModel.set(...
trendModel.save(...

in PY:
def create(self, request):
...

I've also created extra template and additional View. PageView for some unclear 
reason didn't support creation of new object - this event just disappeared.

Regards,

Roman
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Django broken pipe error

[justin walters]

On Mon, Dec 12, 2016 at 7:27 AM, roma  wrote:

> Thanks Justin,
>
> I believe, the whole database story has no influence on the broken pipe
> error. I've commented out the whole block and leave only return line:
> return HttpResponse(res, content_type="text/plain; charset=utf-8")
> The error is still present. And I have no influence on that.
>
> I call python from js client:
>
> var newTrendReport = new App.TrendReport();
> newTrendReport.set('search_phrase', search_phrase);
> newTrendReport.set('time_from', time_from);
> newTrendReport.set('time_to', time_to);
> newTrendReport.set('time_scale', time_scale);
> newTrendReport.set('category', category);
> newTrendReport.startExport(
>
> function(response){
> console.log("Successfully calculated trend report.");
> App.trendPage = new App.TrendPageView();
> App.trendPage.render(response);
> },
> );
>
> go throw:
>
> App.TrendReport = Backbone.Model.extend({
> urlRoot: "/api/trend_reports/",
> defaults: {
> search_phrase: "",
> time_from: "",
> time_to: "",
> time_scale: "",
> category: ""
> },
>
> startExportSuffix: "/export_report/",
>
> startExport: function( successCallback, errorCallback ) {
> console.log("start trend calculation");
> var that = this;
> var ajaxUrl = this.startExportSuffix;
> var options = {
> method: "POST",
> data: this.attributes,
> contentType: "application/json;charset=UTF-8",
> dataType: "json",
>
> error: errorCallback,
> success: successCallback
> };
> console.log("start trend export sync");
> App.ajax(ajaxUrl, options);
> }
>
> });
>
> and come in export_report method.
>
> My urls.py:
>
> url(r'^export_report', ensure_csrf_cookie(views.export_report),
> name="export_report"),
> --
> https://mail.python.org/mailman/listinfo/python-list
>

I'm not super familiar with the way backbone does http requests, but
something seems off about the startExport function.
It seems to me that you are sending a post request to "/export_report/"
which is an endpoint that I'm guessing is nested
in an include from "/api/trend_reports/". However, it looks like the error
you're getting above says you aren't sending
the request to "https://root.com/api/trend_reports/export_report/;.
Instead, you are sending the request to "https://root.com/export_report/; .
Though, I'm also not sure that's the case because that would normally throw
a 404.

I also noticed that you set content type to 'application/json' in your js,
but the view function returns a 'text/plain' content type. Is
There a reason for this?

The data key in your js http function is set to the attributes variable
which, as far as I can tell, does not exist.

There's a lot going on here, but I think you can probably narrow it down to
something in your backbone code or the way
backbone handles http requests as the error you are getting is caused by
the client prematurely closing the socket.
It's possible backbone will stop reading the response since it isn't the
same content-type as the request.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Django broken pipe error

[roma]

On Wednesday, December 7, 2016 at 6:15:41 PM UTC+1, justin walters wrote:
> On Wed, Dec 7, 2016 at 1:08 AM,  wrote:
> 
> > Thank you Justin,
> >
> > I'm on the dev server and should present results in this way.
> >
> > Yes, I use manage.py runserver --insecure to start the server (from
> > PyCharm).
> >
> > My views.py call:
> >
> > @detail_route(methods=['post'], permission_classes=[permissions.AllowAny])
> > def export_report(request):
> > body_unicode = request.body.decode('utf-8')
> > body_str = body_unicode.encode('ascii','ignore')
> > attr_list = body_str.split('&')
> > attr_dict = {}
> > if (len(attr_list) > 0):
> > for attr in attr_list:
> >...
> >key = key_value_pair[0]
> >attr_dict[key] = key_value_pair[1]
> > trend = trends.calculate_trend(
> > attr_dict['search_phrase']
> > , attr_dict['time_from']
> > , attr_dict['time_to']
> > , attr_dict['time_scale']
> > )
> > attr_dict['trend'] = trend
> > res = str(json.dumps(attr_dict))
> > return HttpResponse(res, content_type="text/plain; charset=utf-8")
> >
> > and trend calculation in trends.py with database calls:
> >
> > def calculate_trend(query_phrase, time_from, time_to, time_scale):
> > # check in database if trend already exists
> > try:
> > db_trend = Trend.objects.get(pk=query_phrase)
> > if db_trend.from_time.strftime("%Y-%m-%d") == time_from \
> > and db_trend.to_time.strftime("%Y-%m-%d") == time_to \
> > and db_trend.granularity == time_scale:
> > logger.info("trend already exists.")
> > existing_trend_dict = ast.literal_eval(db_trend.content)
> > return existing_trend_dict
> > except Trend.DoesNotExist:
> > logger.info("It is a new trend search.")
> > trend_dict = {}
> > start_time = pd.Timestamp(value[0])
> > end_time = pd.Timestamp(value[-1])
> > freq = ... get frequency using pandas lib
> > trend_dict[key] = freq
> > json_trend_content = trend_dict_to_sorted_json_str(trend_dict)
> > trend = Trend(
> > phrase=query_phrase,
> > content=json_trend_content,
> > from_time=time_from,
> > to_time=time_to,
> > granularity=time_scale,
> > )
> > if trend is not None:
> > try:
> > db_trend = Trend.objects.get(pk=query_phrase)
> > db_trend.delete()
> > logger.info("delete old trend: %s. " % trend)
> > except Trend.DoesNotExist:
> > logger.info("create trend: %s. " % trend)
> > trend.save()
> > return trend_dict
> >
> > Thank you in advance!
> >
> > Roman
> > --
> > https://mail.python.org/mailman/listinfo/python-list
> >
> 
> 
> It looks like you can probably get rid of the try/except block at the end
> of the calculate_trend
> method as any existing Trend object will have already been caught in the
> first try/except block.
> 
> >From what I'm reading here:
> http://stackoverflow.com/questions/11866792/how-to-prevent-errno-32-broken-pipe
> ,
> this issue can be caused by the client closing the connection before
> sendall() finishes writing. Can you estimate
> how long it takes for a request to this endpoint takes to resolve? If it's
> a long time(maybe due to the pandas call?),
> your browser/client may be timing out. It could be because it takes a while
> for the Db to find an existing Trend object
> as well.
> 
> I can't give you any advice on how to fix it exactly, but I can tell you
> what the problem is: The client is closing the
> connection before socket.sendall() has finished writing to the socket. My
> guess is that the calculate_trend() method
> takes a long time to complete and the client is timing out.

Thanks Justin,

I believe, the whole database story has no influence on the broken pipe error. 
I've commented out the whole block and leave only return line:
return HttpResponse(res, content_type="text/plain; charset=utf-8") 
The error is still present. And I have no influence on that.

I call python from js client:

var newTrendReport = new App.TrendReport();
newTrendReport.set('search_phrase', search_phrase);
newTrendReport.set('time_from', time_from);
newTrendReport.set('time_to', time_to);
newTrendReport.set('time_scale', time_scale);
newTrendReport.set('category', category);
newTrendReport.startExport(

function(response){
console.log("Successfully calculated trend report.");
App.trendPage = new App.TrendPageView();
App.trendPage.render(response);
},
);

go throw:

App.TrendReport = Backbone.Model.extend({
urlRoot: "/api/trend_reports/",
defaults: {
search_phrase: "",
time_from: "",
time_to: "",
time_scale: "",
category: ""
},

startExportSuffix: 

Re: Django broken pipe error

[justin walters]

On Wed, Dec 7, 2016 at 1:08 AM,  wrote:

> Thank you Justin,
>
> I'm on the dev server and should present results in this way.
>
> Yes, I use manage.py runserver --insecure to start the server (from
> PyCharm).
>
> My views.py call:
>
> @detail_route(methods=['post'], permission_classes=[permissions.AllowAny])
> def export_report(request):
> body_unicode = request.body.decode('utf-8')
> body_str = body_unicode.encode('ascii','ignore')
> attr_list = body_str.split('&')
> attr_dict = {}
> if (len(attr_list) > 0):
> for attr in attr_list:
>...
>key = key_value_pair[0]
>attr_dict[key] = key_value_pair[1]
> trend = trends.calculate_trend(
> attr_dict['search_phrase']
> , attr_dict['time_from']
> , attr_dict['time_to']
> , attr_dict['time_scale']
> )
> attr_dict['trend'] = trend
> res = str(json.dumps(attr_dict))
> return HttpResponse(res, content_type="text/plain; charset=utf-8")
>
> and trend calculation in trends.py with database calls:
>
> def calculate_trend(query_phrase, time_from, time_to, time_scale):
> # check in database if trend already exists
> try:
> db_trend = Trend.objects.get(pk=query_phrase)
> if db_trend.from_time.strftime("%Y-%m-%d") == time_from \
> and db_trend.to_time.strftime("%Y-%m-%d") == time_to \
> and db_trend.granularity == time_scale:
> logger.info("trend already exists.")
> existing_trend_dict = ast.literal_eval(db_trend.content)
> return existing_trend_dict
> except Trend.DoesNotExist:
> logger.info("It is a new trend search.")
> trend_dict = {}
> start_time = pd.Timestamp(value[0])
> end_time = pd.Timestamp(value[-1])
> freq = ... get frequency using pandas lib
> trend_dict[key] = freq
> json_trend_content = trend_dict_to_sorted_json_str(trend_dict)
> trend = Trend(
> phrase=query_phrase,
> content=json_trend_content,
> from_time=time_from,
> to_time=time_to,
> granularity=time_scale,
> )
> if trend is not None:
> try:
> db_trend = Trend.objects.get(pk=query_phrase)
> db_trend.delete()
> logger.info("delete old trend: %s. " % trend)
> except Trend.DoesNotExist:
> logger.info("create trend: %s. " % trend)
> trend.save()
> return trend_dict
>
> Thank you in advance!
>
> Roman
> --
> https://mail.python.org/mailman/listinfo/python-list
>


It looks like you can probably get rid of the try/except block at the end
of the calculate_trend
method as any existing Trend object will have already been caught in the
first try/except block.

>From what I'm reading here:
http://stackoverflow.com/questions/11866792/how-to-prevent-errno-32-broken-pipe
,
this issue can be caused by the client closing the connection before
sendall() finishes writing. Can you estimate
how long it takes for a request to this endpoint takes to resolve? If it's
a long time(maybe due to the pandas call?),
your browser/client may be timing out. It could be because it takes a while
for the Db to find an existing Trend object
as well.

I can't give you any advice on how to fix it exactly, but I can tell you
what the problem is: The client is closing the
connection before socket.sendall() has finished writing to the socket. My
guess is that the calculate_trend() method
takes a long time to complete and the client is timing out.
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Django broken pipe error

[dr . roman . graf]

On Tuesday, December 6, 2016 at 3:22:13 PM UTC+1, dr.rom...@gmail.com wrote:
> Hi,
> 
> I'm facing strange Django broken pipe error (Python 2.7 on Ubuntu) that 
> apparently is a not fixed Django bug. Does anybody now how to fix it? I've 
> been searching a lot and didn't find any solution.
> 
> This error happens very irregularly by Post request in Django. Sometimes it 
> works sometimes not: 
> 
> [06/Dec/2016 13:33:57] "POST /export_report/ HTTP/1.1" 500 59
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/SocketServer.py", line 593, in 
> process_request_thread
> self.finish_request(request, client_address)
>   File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
> self.RequestHandlerClass(request, client_address, self)
>   File 
> "/home/ait/.virtualenvs/env1/local/lib/python2.7/site-packages/django/core/servers/basehttp.py",
>  line 129, in __init__
> super(WSGIRequestHandler, self).__init__(*args, **kwargs)
>   File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
> self.finish()
>   File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
> self.wfile.close()
>   File "/usr/lib/python2.7/socket.py", line 279, in close
> self.flush()
>   File "/usr/lib/python2.7/socket.py", line 303, in flush
> self._sock.sendall(view[write_offset:write_offset+buffer_size])
> error: [Errno 32] Broken pipe
> 
> Exception happened during processing of request from ('127.0.0.1', 38224)
> 
> It is also described in: 
> https://www.reddit.com/r/joinmarket/comments/4atqrm/is_this_exception_normal_exception_happened/
> 
> On https://bugs.python.org/issue14574 is stated that this error should 
> already be fixed but apparently not.
> 
> Best regards,
> 
> Roman

Thank you Justin,

I'm on the dev server and should present results in this way. 

Yes, I use manage.py runserver --insecure to start the server (from PyCharm).

My views.py call:

@detail_route(methods=['post'], permission_classes=[permissions.AllowAny])
def export_report(request):
body_unicode = request.body.decode('utf-8')
body_str = body_unicode.encode('ascii','ignore')
attr_list = body_str.split('&')
attr_dict = {}
if (len(attr_list) > 0):
for attr in attr_list:
   ...
   key = key_value_pair[0]
   attr_dict[key] = key_value_pair[1]
trend = trends.calculate_trend(
attr_dict['search_phrase']
, attr_dict['time_from']
, attr_dict['time_to']
, attr_dict['time_scale']
)
attr_dict['trend'] = trend
res = str(json.dumps(attr_dict))
return HttpResponse(res, content_type="text/plain; charset=utf-8")

and trend calculation in trends.py with database calls:

def calculate_trend(query_phrase, time_from, time_to, time_scale):
# check in database if trend already exists
try:
db_trend = Trend.objects.get(pk=query_phrase)
if db_trend.from_time.strftime("%Y-%m-%d") == time_from \
and db_trend.to_time.strftime("%Y-%m-%d") == time_to \
and db_trend.granularity == time_scale:
logger.info("trend already exists.")
existing_trend_dict = ast.literal_eval(db_trend.content)
return existing_trend_dict
except Trend.DoesNotExist:
logger.info("It is a new trend search.")
trend_dict = {}
start_time = pd.Timestamp(value[0])
end_time = pd.Timestamp(value[-1])
freq = ... get frequency using pandas lib
trend_dict[key] = freq
json_trend_content = trend_dict_to_sorted_json_str(trend_dict)
trend = Trend(
phrase=query_phrase,
content=json_trend_content,
from_time=time_from,
to_time=time_to,
granularity=time_scale,
)
if trend is not None:
try:
db_trend = Trend.objects.get(pk=query_phrase)
db_trend.delete()
logger.info("delete old trend: %s. " % trend)
except Trend.DoesNotExist:
logger.info("create trend: %s. " % trend)
trend.save()
return trend_dict

Thank you in advance!

Roman
-- 
https://mail.python.org/mailman/listinfo/python-list

Re: Django broken pipe error

[justin walters]

On Tue, Dec 6, 2016 at 6:21 AM,  wrote:

> Hi,
>
> I'm facing strange Django broken pipe error (Python 2.7 on Ubuntu) that
> apparently is a not fixed Django bug. Does anybody now how to fix it? I've
> been searching a lot and didn't find any solution.
>
> This error happens very irregularly by Post request in Django. Sometimes
> it works sometimes not:
>
> [06/Dec/2016 13:33:57] "POST /export_report/ HTTP/1.1" 500 59
> Traceback (most recent call last):
>   File "/usr/lib/python2.7/SocketServer.py", line 593, in
> process_request_thread
> self.finish_request(request, client_address)
>   File "/usr/lib/python2.7/SocketServer.py", line 334, in finish_request
> self.RequestHandlerClass(request, client_address, self)
>   File "/home/ait/.virtualenvs/env1/local/lib/python2.7/site-
> packages/django/core/servers/basehttp.py", line 129, in __init__
> super(WSGIRequestHandler, self).__init__(*args, **kwargs)
>   File "/usr/lib/python2.7/SocketServer.py", line 651, in __init__
> self.finish()
>   File "/usr/lib/python2.7/SocketServer.py", line 710, in finish
> self.wfile.close()
>   File "/usr/lib/python2.7/socket.py", line 279, in close
> self.flush()
>   File "/usr/lib/python2.7/socket.py", line 303, in flush
> self._sock.sendall(view[write_offset:write_offset+buffer_size])
> error: [Errno 32] Broken pipe
> 
> Exception happened during processing of request from ('127.0.0.1', 38224)
>
> It is also described in: https://www.reddit.com/r/
> joinmarket/comments/4atqrm/is_this_exception_normal_exception_happened/
>
> On https://bugs.python.org/issue14574 is stated that this error should
> already be fixed but apparently not.
>
> Best regards,
>
> Roman


Hi Roman,

Is this happening on the dev server or a production server? Did you use the
command ./manage.py runserver to start the server?

Can you please provide the code for the view class/function that is
throwing the error? I believe that this issue
may be caused by a call to the database not being structured correctly.
-- 
https://mail.python.org/mailman/listinfo/python-list