Re: [Catalyst] Problems with Encoding support in Catalyst.
On Sat, Jan 7, 2017 at 9:55 PM, Bill Moseley wrote:
>
> # Let it be set to undef
> if (my $wanted = shift) {
> $encoding = Encode::find_encoding($wanted)
> or Carp::croak( qq/Unknown encoding '$wanted'/ );
> *binmode(STDERR, ':encoding(' . $encoding->name . ')');*
>
>
> The problem with that is it seems to turn off autoflush on STDERR. In my
> case that is causing intermixing of lines in the log files.
>
> If I turn on STDERR->autoflush(1) then the logging works again and log
> lines are not mangled.
>
> But, I'm not sure we should apply a layer to STDERR like that. For
> example, I log to stderr with JSON and that is already encoded.
>
Just in case it isn't clear that is a problem:
use strict;
use warnings;
use Encode;
use JSON;
binmode( STDERR, ':utf8' );
my $json = encode_json( { chars => decode_utf8( '雨下降' ) } );
print STDERR "$json\n";
Results in:
{"chars":"é›¨ä¸‹é™ "}
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Problems with Encoding support in Catalyst.
I just finally removed my custom encoding code to use the built-in code
Catalyst now provides to decode input data and encode output.
But, I came across two issues that I wanted to get some feedback on.
First one is simple. This code will die if $value is already decoded. Any
reason not to test it?
sub _handle_param_unicode_decoding {
my ( $self, $value, $check ) = @_;
return unless defined $value; # not in love with just ignoring undefs
- jnap
return $value if blessed($value); #don't decode when the value is an
object.
+return $value if Encode::is_utf8($value); # already decoded?
This next one is what I have questions about. This is also in Catalyst.pm
in the encoding() method:
# Let it be set to undef
if (my $wanted = shift) {
$encoding = Encode::find_encoding($wanted)
or Carp::croak( qq/Unknown encoding '$wanted'/ );
*binmode(STDERR, ':encoding(' . $encoding->name . ')');*
The problem with that is it seems to turn off autoflush on STDERR. In my
case that is causing intermixing of lines in the log files.
If I turn on STDERR->autoflush(1) then the logging works again and log
lines are not mangled.
But, I'm not sure we should apply a layer to STDERR like that. For
example, I log to stderr with JSON and that is already encoded.
Is there a compelling reason to binmode STDERR there?
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] From Development to Production.
On Wed, Mar 9, 2016 at 7:35 AM, Dermot wrote: I want to caution you, in the nicest possible way. Writing software > requires a number of skills and a lot of research and learning. You can't > avoid the latter. What may seem like a lot of unnecessary aggravation > (testing and version control) have come about because it's no fun fixing > problems after the event. > Very true. Andrew, there's an overwhelming amount of reading on this topic. If you think there's *any* chance that your app will grow in usage and developers over time then it's well worth planning ahead. Automate *everything* from the start. It's critical to make deployments fast, repeatable, and dead simple. Probably should also include "scalable" there as a good buzzword. Here's a few random links to get you thinking: http://12factor.net/ https://zachholman.com/posts/deploying-software https://guides.github.com/introduction/flow/ (we do a modified version of this) The bottom line is you want to be spending your energy on the application functionality and not on deploying and running the application -- which seems to consume time exponentially if not careful. To be honest we have not really found a perfect way of managing Perl dependencies. I think the standard Perl module versioning system falls apart when there's lots of developers, many "apps" involved, and a version control system. Our single app turned into many Catalyst apps that (with the goal of) have "thin" controllers and many models. Those models were originally separate Dist::Zilla Perl distributions in our local DarkPAN. As those were updated with new version numbers the apps were also updated to depended on those. This breaks down when using Git and heavy use of branches because versions are no longer linear -- instead small features and fixes are merged individually. We have swung quite far the other way now and moving toward separate repos for each "app" and even copying the same module into multiple repos to promote decoupling (and the cost of code duplication). We are also using Carton to lock down versions and bundle dependencies. We have also put Perlbrew into Git with mixed results.I'm not a big fan of placing build artifacts into Git, but having the entire app in a single repo does have benefits. We would like to get to a CI pipeline that generates a Docker container and use that for all non-dev deployments. You never know when you will need to deploy a million containers <https://www.hashicorp.com/c1m.html>... -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Plugin attributes and Moose
I wanted to note something (before the week gets started and time vanishes)
about how Catalyst is behaving, and see if it's expected.
I asked about this on the Moose list, so sorry if you already saw this.
A Catalyst app extends Catalyst::Component, which has a BUILDARGS sub to
merge in class configuration when creating instances of components.
This is how configuration can set initial values on attributes in Models,
Views, Controllers, and the application class itself.
If a (non-role) Moose-based plugin is loaded (which is common) it's added
to the app's inheritance like this:
$meta->superclasses($plugin, $meta->superclasses);
Which can be thought of like this:
@App::ISA = ( qw/ Catalyst::Plugin::Foo Catalyst / );
The result then is that BUILDARGS in Catalyst::Component is no longer
called, and then attributes in the App class are no longer populated from
the config.
So, the behavior changes depending on what plugins are brought in.
Another odd issue I came against is that MooseX::Emulate::Class::Accessor::Fast
causes odd behavior of Moose attributes. This role is widely used in
Catalyst.
In the code below note how the "foo" attribute has init_arg => undef to
prevent it from being initialized. With the the role not only is it
initialized, but with a value that isn't an "Int".
package Foo;
use Moose;
with 'MooseX::Emulate::Class::Accessor::Fast';
has foo => (
is => 'ro',
isa => 'Int', # for error
init_arg => undef,
);
package main;
use strict;
use warnings;
my $foo = Foo->new( { foo => 'bar' } );
use Data::Dumper;
print Dumper $foo->foo;
Generates 'bar', which is not an Int.
$VAR1 = 'bar';
Comment out "with 'MooseX::Emulate::Class::Accessor::Fast';" and it behaves
as expected. Also comment out "init_arg" and Moose will complain about
the string not begin an Int.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Setting config after setup has been run is not allowed.
I'm confused about this rather stern note in Catalyst.pm.
Is this the following wrapper related to the text in the NOTE?
B you MUST NOT call C<< $self->config >> or C<< __PACKAGE__->config
>>
as a way of reading config within your code, as this B give you
the
correctly merged config back. You B take the config values supplied to
the constructor and use those instead.
=cut
around config => sub {
my $orig = shift;
my $c = shift;
croak('Setting config after setup has been run is not allowed.')
if ( @_ and $c->setup_finished );
$c->$orig(@_);
};
I understand the NOTE for Model/View/Controllers where the component's
__PACKAGE__->config is merged in with the application config for that
component. Calling $self->config won't be the same thing passed to the
component's constructor.
But, what's the issue with calling $c->config( foo => 123 ) at runtime?
Or even $c->config( \%new_config )?
Note that wrapper only applies to calling config on the app class, and does
not apply to Model/View/Controllers.
Is there some other issues I'm missing? What problem is that wrapper
trying to prevent?
Thanks,
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Url Encoded UTF8 parameters
BTW -- I wonder about the Catalyst behavior here.
On Sat, Aug 1, 2015 at 10:36 PM, Bill Moseley wrote:
>
>
> On Sat, Aug 1, 2015 at 6:31 AM, Stefan wrote:
>
>> Hi,
>>
>> if a URL parameter contains a Unicode character (e.g.
>> www.example.com/?param=%D6lso%DF which stands for param=Ölsoße), the
>> parameter is not correctly parsed as Unicode.
>>
>
One note here -- data over the wire must be encoded into octets. So, all
Unicode characters must be encoded and then decoded when received. (You
can't send "Unicode characters".) UTF-8 is used now (for obvious
reasons). http://tools.ietf.org/html/rfc3986.
You are specifying %D6 -- although the Unicode characters is U+00D6, the
UTF-8 octet sequence is 0xC3 0x96. See:
http://www.fileformat.info/info/unicode/char/00D6/index.htm
Unless otherwise instructed, Catalyst uses UTF-8
<https://github.com/perl-catalyst/catalyst-runtime/blob/master/lib/Catalyst/Engine.pm#L579>
as the encoding for decoding query parameters -- query parameters are
decoded from UTF-8 octets to Perl characters.
As your example showed, if you use invalid UTF-8 sequences then
Encode::decode() as used by Catalyst will replace those with the U+FFFD
substitution character
<http://www.fileformat.info/info/unicode/char/fffd/index.htm> "�".
This may or may not be what you want. Personally, I think it's not
correct to silently modify user input. You intended to pass "Ölsoße" but
ended up with "�lso�e" -- is that really the data you would want to
process/store for the request? Seems unlikely.
If "param" is suppose to be passed as textual, UTF-8-encoded octets, and it
isn't, then maybe returning a 400 is a better way of handling that. That
probably would have helped you see what is wrong in this case.
i.e. use "eval { decode( $default_query_encoding, $str, FB_CROAK |
LEAVE_SRC ); }" to catch invalid data and return to the client the "$str"
that failed and why.
Of course, it is also possible that you have some query parameters that you
want decoded as UTF-8 and some that might represent something else (a raw
sequence of bytes), and want more manual control. In that case
$c->config->{do_not_decode_query} could be used to bypass the decoding.
But then, you must manually decode() yourself.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Url Encoded UTF8 parameters
On Sat, Aug 1, 2015 at 6:31 AM, Stefan wrote:
> Hi,
>
> if a URL parameter contains a Unicode character (e.g.
> www.example.com/?param=%D6lso%DF which stands for param=Ölsoße), the
> parameter is not correctly parsed as Unicode.
>
4. This outputs for the example url: localhost:3000/?param=%D6lso%DF:
>
> [debug] $VAR1 = {
>
> 'param' => "\x{fffd}lso\x{fffd}e"
>
> };
>
> [debug] $VAR1 = '\x{d6}lso\x{df}e';
>
>
>
>
>
> As you can see, the first output only contains one equal character:
> \x{fffd} which is obviously not the same as it should be: \x{d6}lso\x{df}e
>
\x{fffd} is the unicode replacement character used by Encode to replace
invalid UTF-8 sequences you are passing in.
Try this instead in your browser:
?param=Ölsoße
And then print $c->request->parameters->{param} -- and if you check
Encode::is_utf8( $param ) it should be true, too, indicating the param was
decoded correctly into characters.
Or if you prefer:
perl -le 'use URI::Escape; print uri_escape( "Ölsoße" )'
%C3%96lso%C3%9Fe
so,
?param=%C3%96lso%C3%9Fe
but most likely the browser will turn it back into ?param=Ölsoße
If you really want to say you are using utf8 constant strings (i.e. "use
utf8;"):
$ perl -le 'use URI::Escape; use Encode; use utf8; use Encode; print
uri_escape( encode_utf8( "Ölsoße" ) )'
%C3%96lso%C3%9Fe
or
$ perl -le 'use URI::Escape; use Encode; use utf8; use Encode; print
uri_escape_utf8( "Ölsoße" )'
%C3%96lso%C3%9Fe
All the same thing.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Configure psgi.input as optional?
I created a pull for checking the return for the psgi.input file:
https://github.com/perl-catalyst/catalyst-runtime/pull/100
I created a ticket for HTTP::Body:
https://rt.cpan.org/Ticket/Display.html?id=105021
I'll create a separate pull for a config setting to make creating the
psgi.input file optional when I have a bit of time.
On Fri, Jun 5, 2015 at 4:38 PM, Lasse Makholm wrote:
>
>
> On Fri, Jun 5, 2015 at 8:26 PM, Bill Moseley wrote:
>
>> Hi,
>>
>> Our app handles a lot of uploads, often quite large uploads.As we
>> know, uploads (and any non-parsed body) gets written to temp files via
>> HTTP::Body.
>>
>> Now, Catalyst::Request also writes every body to a temp file (via
>> Stream::Buffered). So, depending on the body size, can take up to twice
>> the space in temp files as the content length.
>>
>> If we are not going to use psgi.input and just use Catalyst's
>> (HTTP::Body's) temp files could we make the creation of psgi.input optional?
>>
>> I have a few other concerns about the current code.
>>
>> prepare_body in Catalyst::Request does not check the return code when it
>> writes to the psgi.input file. HTTP::Body also fails to check return
>> codes when it writes. This means if the partition where temp files are
>> created fills then Catalyst request will ignore this and continue as if
>> there's no problem. I trust the risk here is recognized.
>>
>> I think the fix in Catalyst::Request is pretty simple by checking return
>> values:
>>
>> # Check for definedness as you could read '0'
>> while ( defined ( my $chunk = $self->read() ) ) {
>> $self->prepare_body_chunk($chunk);
>>
>> next unless $stream_buffer;
>> $stream_buffer->print($chunk)
>> || die sprintf "Failed to write %d bytes to psgi.input file:
>> $!", length( $chunk );
>> }
>>
>> HTTP::Body needs similar changes.
>>
>
> On a related note, things like NFS mounted file systems tend to not
> guarantee that data is safely written to disk until all write() calls AND
> the final close() have completed successfully.
>
> Code that checks that write() succeeds but then fails to do the same for
> close() is still broken.
>
> Thanks for looking into this. Our app handles lots of large uploads too
> and I would love to see this fixed in HTTP::Body.
>
> /L
>
>
>>
>> See any problem with making the psgi.input file optional? And any
>> reason not to throw an exception when writing to the temp file fails?
>>
>> BTW -- Plack::Handler::Apache2 sets psgi.input to the Apache request
>> object $r -- it would be handy to preserve this object for later use.
>>
>>
>> --
>> Bill Moseley
>> mose...@hank.org
>>
>> ___
>> List: Catalyst@lists.scsys.co.uk
>> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
>> Searchable archive:
>> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
>> Dev site: http://dev.catalyst.perl.org/
>>
>>
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Configure psgi.input as optional?
Hi,
Our app handles a lot of uploads, often quite large uploads.As we know,
uploads (and any non-parsed body) gets written to temp files via HTTP::Body.
Now, Catalyst::Request also writes every body to a temp file (via
Stream::Buffered). So, depending on the body size, can take up to twice
the space in temp files as the content length.
If we are not going to use psgi.input and just use Catalyst's
(HTTP::Body's) temp files could we make the creation of psgi.input optional?
I have a few other concerns about the current code.
prepare_body in Catalyst::Request does not check the return code when it
writes to the psgi.input file. HTTP::Body also fails to check return
codes when it writes. This means if the partition where temp files are
created fills then Catalyst request will ignore this and continue as if
there's no problem. I trust the risk here is recognized.
I think the fix in Catalyst::Request is pretty simple by checking return
values:
# Check for definedness as you could read '0'
while ( defined ( my $chunk = $self->read() ) ) {
$self->prepare_body_chunk($chunk);
next unless $stream_buffer;
$stream_buffer->print($chunk)
|| die sprintf "Failed to write %d bytes to psgi.input file:
$!", length( $chunk );
}
HTTP::Body needs similar changes.
See any problem with making the psgi.input file optional? And any reason
not to throw an exception when writing to the temp file fails?
BTW -- Plack::Handler::Apache2 sets psgi.input to the Apache request object
$r -- it would be handy to preserve this object for later use.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Catalyst::Request::Upload->filename is not decoded.
All my upload forms have accept-charset="utf-8".We expect that uploaded
filenames could have wide-characters.
The problem I hit was ->basename does this:
$ perl -le 'use Catalyst::Request::Upload; my $upload =
Catalyst::Request::Upload->new( { filename => q[документ обучения.pdf] } );
print $upload->basename;'
_.pdf
That's pretty mangled.
The problem is that $upload->filename is not decoded so the substitution is
working on octets not characters.
sub _build_basename {
my $self = shift;
my $basename = $self->filename;
$basename =~ s|\\|/|g;
$basename = ( File::Spec::Unix->splitpath($basename) )[2];
$basename =~ s|[^\w\.-]+|_|g;
return $basename;
}
Obviously, we want \w to work on characters, not encoded octets. Decoding
the filename should be done -- it's character data.
Does it make sense to do it in Engine's prepare_uploads?
For example:
my $u = Catalyst::Request::Upload->new(
size => $upload->{size},
type => scalar $headers->content_type,
headers => $headers,
tempname => $upload->{tempname},
filename =>
*$c->_handle_unicode_decoding($upload->{filename})*,
);
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Catalyst Unicode
On Fri, Jan 31, 2014 at 3:58 AM, Will Crawford wrote: > > If the string has been decoded *from* UTF-8 to Perl's internal > representation, it's *not* going to be marked as UTF8 internally; it > *shouldn't* be. It's no longer a "UTF8" string but a "Unicode" string, > complete with wide characters. If anything, the internal "UTF8" flag > means "this string needs decoding" rather than "has been decoded". > $ perl -le 'use Encode; my $chars = decode_utf8( "bytes" ); print Encode::is_utf8( $chars ) ? "Is flagged utf8\n" : "not flagged\n"; use Devel::Peek; Dump($chars)' Is flagged utf8 SV = PV(0x7fb8c10023f0) at 0x7fb8c102b6a8 REFCNT = 1 FLAGS = (PADMY,POK,pPOK,UTF8) PV = 0x7fb8c0e01170 "bytes"\0 [UTF8 "bytes"] CUR = 5 LEN = 16 Everything is encoded. The flag tells Perl that its internal representation is encoded as utf8 so knows to work with it as utf8 characters (e.g. length() is length of chars, matching works on chars, etc.) $ perl -le 'use Encode; my $chars = decode( 'latin1', "bytes" ); print Encode::is_utf8( $chars ) ? "Is flagged utf8\n" : "not flagged\n"; use Devel::Peek; Dump($chars)' Is flagged utf8 -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Implementing Webhooks.
I'm running Catalyst under mod_perl2 (currently, subject to change). Some requests trigger a callback to a user-provided URL -- a webhook. Obviously, it's best to do those asynchronously and not in-process. Don't want a web request (or really an Apache process) hanging while waiting on an external web server to respond. A queue is probably the best approach, but there's also some advantage of having the Catalyst app make the webhook request -- specifically because the Catalyst app has the context of the request and has application logging available. How would you implement this? How would that change if you wanted more than just "fire-and-forget"? For example, if you wanted to provide some kind of retry interval for failed callbacks. The external server might be temporarily down for maintenance. Thanks, -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Changing format of date field
On Tue, Jan 7, 2014 at 1:41 AM, Adam Witney wrote: > [% user_time( foo.event_start, '_DT_TIMESTAMP_WITH_ZONE' ) | html %] > > Hi Bill, > > Thanks for your email. I agree that the template seems like a good place > for this, but to give a little extra detail. > > I have a form that is used for both object create and editing. When I edit > an object I have this in my controller: > > $c->stash(formdata => $object); > > and in the template: > > > > But when creating a new object, if the form validation fails I pass the > form data back to the form like so: > > $c->stash( formdata => $c->request->params ); > Have you looked at HTML::FormHandler? I would assume it addresses this. I use something similar and the concept is that you have a field object that knows how to render either from the existing object or from user provided input. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Changing format of date field
On Mon, Jan 6, 2014 at 7:18 AM, Adam Witney wrote:
> Hi,
>
> I have a date field in a DBIx::Class Result class using
> InflateColumn::DateTime. When I pass this through a Catalyst/TT
> application, the date is presented in -MM-DD format.
>
> I can modify this to DD/MM/ in my TT template with object.dmy('/'),
> but I would prefer to change the default DateTime stringification, however
> I am not quite sure where or how to do this?
>
> Any ideas greatly appreciated.
>
Seem like formatting in the template is the right place -- may want to have
different formats in different places in your app. Then you might also
think about how best to localize.
One idea is to localize a set of names for a set of formats:
[% dt_fmt = c.localize( '_DT_DATE_ONLY' ); foo.some_dt_object.strftime(
dt_fmt ) | html %]
I have also used a function that does the above, but also clones and sets
the time zone and locale based on the user's preferences:
[% user_time( foo.event_start, '_DT_TIMESTAMP_WITH_ZONE' ) | html %]
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Plack::Middleware::ContentLength problem
On Fri, Dec 20, 2013 at 8:34 PM, neil.lunn wrote: > > My article today actually (http://www.catalystframework.org/calendar/2013/21), > even though I'm actually talking here about the above case. > Just a note on the Advent article. Thanks for writing that up. It's a well-written article and clearly explains the issue I was facing and the fix you provided to me. One thing I really like about the Advent articles is that I learn new ways to do things. For example, I wasn't aware of namespace::sweep and never thought about overriding the -X filetests. I just set the content length manually now in the Controller along with the body. I'm was also very happy to see you building this into a model at the end. I sometimes wonder if that is not stressed enough when learning Catalyst. I see a lot of code written into controllers at work that should really be models. I will pass the link to the Advent article around. In my specific situation for this problem I already had a model. The gzipped files are stored on a distributed file system and I already had a model class for fetching files. I just extended that to handle these gzipped files.But, I think your solution is a bit more elegant and, well, more correct because it can be used more widely. Thanks, ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Anyone have experience with locales?
I'm using the Catalyst plugin to support utf8 encoded input and output, I have my database driver set to read in utf8, and I have Template Toolkit configured for utf8 templates. I want to use available locale to render numbers. I'm using Number::Format in a number of places and it works with POSIX's setlocale() function. What I'm not clear on is if setlocale() plays nicely with everything above. Anyone here have experience with this? http://perldoc.perl.org/perllocale.html#Unicode-and-UTF-8 says: It is strongly recommended that when combining Unicode and locale (starting in v5.16), you use 1. use <http://perldoc.perl.org/functions/use.html> locale ':not_characters'; When this form of the pragma is used, only the non-character portions of locales are used by Perl, for exampleLC_NUMERIC . Perl assumes that you have translated all the characters it is to operate on into Unicode (actually the platform's native character set (ASCII or EBCDIC) plus Unicode). For data in files, this can conveniently be done by also specifying... For one thing I'm not using 5.16 yet - and I'm not clear what means for pre 5.16. Should I only use setlocale( &LC_NUMERIC, $locale ), for example? That is, using can setlocale( &LC_ALL, $locale ) cause problems with my existing character handling? -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Bump Catalyst::Model::Adaptor
On Sun, Dec 15, 2013 at 6:09 PM, neil.lunn wrote:
>
>
> But I prefer the mangle_arguments particulary for classes that do not use
> a Moose like contructor.
> The change from the old code seems to be deliberate in passing in extra
> arguments to the model constructor, so it seems best to override these
> hooks where needed rather than patch the base.
>
It's a bit confusing, though. The docs for "args" say this:
This is a hashref of arguments to pass to the constructor of class. It is
optional, of course. If you omit it, nothing is passed to the constructor
(as opposed to {}, an empty hashref).
But maybe there was a use case where having the extra config (in addition
to config in "args") passed to the constructor was necessary.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Bump Catalyst::Model::Adaptor
On Sun, Dec 15, 2013 at 4:49 PM, neil.lunn wrote:
>
> Isn't this just a case of adding to your model class:
>
Yes, that's good. Thanks.
>
> sub mangle_arguments {
> my ( $self, $args ) = @_;
> return $args->{args};
> }
>
> Thus overriding the default with what you want to do for your constructor.
> This clears the error for me.
>
>
> In this case it is Cache::Memcached::libmemcached complaining.
>
> I think there's a suggested patch in one of these. Is there someone
> that can review and maybe push out a new version?
>
> https://rt.cpan.org/Public/Bug/Display.html?id=67078
> https://rt.cpan.org/Public/Bug/Display.html?id=78663
>
>
>
> --
> Bill Moseley
> mose...@hank.org
>
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
>
>
> --
><http://www.avast.com/>
>
> This email is free from viruses and malware because avast!
> Antivirus<http://www.avast.com/>protection is active.
>
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Bump Catalyst::Model::Adaptor
I'm a big fan of Catalyst::Model::Adaptor.It helps maintain good separation of concerns and code resue. But, there's an annoying issue where it sends its entire config to the model's constructor instead of just what is in "args". Some code will complain when it finds unknown constructor arguments. For example I'm seeing lots of these: Unrecognised options: args catalyst_component_name class at /var/lib/jenkins/perl5/perlbrew/perls/jenkins/lib/site_perl/5.14.4/Catalyst/Model/Adaptor/Base.pm line 27. In this case it is Cache::Memcached::libmemcached complaining. I think there's a suggested patch in one of these. Is there someone that can review and maybe push out a new version? https://rt.cpan.org/Public/Bug/Display.html?id=67078 https://rt.cpan.org/Public/Bug/Display.html?id=78663 -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Re: Content Disposition filename
On Tue, Nov 19, 2013 at 10:32 AM, Bill Moseley wrote:
> Anyone aware of a good, portable way in Perl to encode the filename in a
> Content-Disposition header? I would like to support UTF8 filenames, but
> support in browsers is unclear (if not changing).
>
> Is this complexity something that the Catalyst framework should handle?
> It's one of those areas where it's easy to get wrong (I can see many
> different approaches in our own code).
>
> http://greenbytes.de/tech/tc2231/
>
>
> http://stackoverflow.com/questions/93551/how-to-encode-the-filename-parameter-of-content-disposition-header-in-http
>
I have no idea what the client can accept or what its OS uses as a
path-separator, and I don't want to go down the client-sniffing path,
anyway.
I have a user-supplied character string that I want to use as the filename,
which I have to assume can contain any unicode character since it's
user-supplied data.
>From my limited tests it seems most modern browsers are supporting the
"filename*" extension. Each browser does some special handling (like
replacing the path-separator, or adding a file extension based on
content-type if no file extension is in the filename).
All I want to do is make valid HTTP headers and let the client decide how
to handle it, but also provide a usable filename (not just underscores, for
example).
So, all I'm after is to make this valid markup:
$c->res->header( content_disposition =>
qq[attachment; filename="$ascii_file"; filename*=UTF-8''$utf8_file]
);
The filename* is easy, I'm finding:
my $utf8_file = uri_escape( Encode::encode( 'UTF-8' => $filename ) );
But the $ascii_file is a bit more work. Percent-encoding doesn't work.
So, have to do a bit of filtering.
See any easier/cleaner/more-correct approach? When I see this much code I
tend to think it's the wrong approach.
# Convert to ASCII using underscore as replacement
my $ascii_file = Encode::encode( ascii => $filename, sub { '_' } );
# Remove quotes as we want to use quoted form of "filename" and preserve
whitespace.
$ascii_file =~ s/"/_/g;
# Replace non-printable characters with underscore, and collapse dups
$ascii_file =~ s/[^[:print:]]/_/g;
$ascii_file =~ s/_{2,}/_/g;
# Split off the extension so can check length of filename w/o extension.
# Of course, $ext could end up as dot + underscore.
my ( $base, $ext ) = split /(\.\w+)$/, $ascii_file;
# Use default filename if we don't have more than three "meaningful"
characters.
# very subjective.
$base = 'your_file' unless ( () = $base =~ /[A-Za-z0-9]/g ) > 3;
# Stuff the extension back on.
$ascii_file = $base;
$ascii_file .= $ext if defined $ext;
Again, "filename*" support is good, and I'm not trying to prevent buggy
clients from doing something stupid (e.g. filename=/etc/passwd), but want
to provide a reasonable fallback to "filename".
Perhaps the simple solution is to always use "filename=your_file" and hope
most clients use the filename* extension.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Setting file handle as the response body generates warnings.
On Wed, Nov 20, 2013 at 11:32 PM, neil.lunn wrote: > > This approach may work for you is the compressed data is actually in a > scalar and not too large. And not too small. YMMV. > > my $z = read_file "product.json.gz"; > > my $io = IO::Scalar->new( \$z ); > $io->seek( -4, 2 ); > $io->read( my $buf, 4); > > my $uncompressed_size = unpack( 'V', $buf ); > This indeed does work in my tests. Thanks for all the help, Neil. I really appreciate the time you spent on this. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Setting file handle as the response body generates warnings.
On Wed, Nov 20, 2013 at 3:45 PM, neil.lunn wrote:
> 2.3.1.2. Compliance
>
>
> A compliant compressor must produce files with correct ID1,
> ID2, CM, CRC32, and ISIZE, but may set all the other fields in
> the fixed-length part of the header to default values (255 for
> OS, 0 for all others). The compressor must set all reserved
> bits to zero.
>
>
>
Seems noncompliance may be rampant.
Anyway, sounds like Catalyst isn't quite supporting this kind of file
handle as expected. John, is there anything you would want me to try?
$ perl -MIO::Uncompress::Gunzip -le 'use Data::Dumper; print Dumper
+IO::Uncompress::Gunzip->new( "Catalyst-Runtime-5.90051.tar.gz"
)->getHeaderInfo'
$VAR1 = {
'Time' => 1383843952,
'Flags' => 8,
'TextFlag' => 0,
'MethodID' => 8,
'ExtraField' => [],
'CommentFlag' => 0,
'Type' => 'rfc1952',
'NameFlag' => 1,
'ExtraFlags' => 2,
'HeaderCRC' => undef,
'isMinimalHeader' => 0,
'MethodName' => 'Deflated',
'ExtraFlag' => 0,
'HeaderLength' => 39,
'ExtraFieldRaw' => undef,
'Comment' => undef,
'OsName' => 'Unix',
'FingerprintLength' => 2,
'HeaderCRCFlag' => 0,
'OsID' => 3,
'TrailerLength' => 8,
'Name' => 'Catalyst-Runtime-5.90051.tar',
'Header' => p�{RCatalyst-Runtime-5.90051.tar'
};
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Setting file handle as the response body generates warnings.
On Wed, Nov 20, 2013 at 7:37 AM, Bill Moseley wrote:
>
> On Wed, Nov 20, 2013 at 4:08 AM, neil.lunn wrote:
>
>> my $length = $body->getHeaderInfo
>
>
> Well, that's helpful. Thanks. Completely missed that in the docs.
>
Well, except for the actual files I have no ISIZE:
$VAR1 = {
'Time' => 0,
'Flags' => 0,
'TextFlag' => 0,
'MethodID' => 8,
'ExtraField' => [],
'CommentFlag' => 0,
'Type' => 'rfc1952',
'NameFlag' => 0,
'ExtraFlags' => 0,
'HeaderCRC' => undef,
'isMinimalHeader' => 0,
'MethodName' => 'Deflated',
'ExtraFlag' => 0,
'HeaderLength' => 10,
'ExtraFieldRaw' => undef,
'Comment' => undef,
'OsName' => 'Unix',
'FingerprintLength' => 2,
'HeaderCRCFlag' => 0,
'OsID' => 3,
'TrailerLength' => 8,
'Name' => undef,
'Header' => ''
};
And even if I gzip a file (e.g. $ gzip foo.txt) then the resulting
foo.txt.gz has 'ISIZE' => 0,
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Setting file handle as the response body generates warnings.
On Wed, Nov 20, 2013 at 4:08 AM, neil.lunn wrote: > my $length = $body->getHeaderInfo Well, that's helpful. Thanks. Completely missed that in the docs. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Setting file handle as the response body generates warnings.
On Tue, Nov 19, 2013 at 6:34 PM, John Napiorkowski wrote:
>
> This looks familiar, but I thought we fixed this... what version of
> Catalyst are you using here?
>
I just updated:
$ perl -MCatalyst -le 'print Catalyst->VERSION'
5.90051
I can debug more in the morning -- anything specific I should look at?
It's pretty easy to reproduce:
package Zip::Controller::Root;
...
use IO::Uncompress::Gunzip;
use IO::Compress::Gzip qw/ gzip $GzipError /;
sub uncompress : Local {
my ( $self, $c ) = @_;
my $uncompressed = "This is some text that can be compressed.\n" x 5;
my $compressed;
gzip( \$uncompressed, \$compressed ) || die $GzipError;
$c->res->body( IO::Uncompress::Gunzip->new( \$compressed ) );
return;
}
$ script/zip_test.pl /uncompress
-s on unopened filehandle GEN2 at
/home/bill/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Catalyst.pm
line 1948, line 1000.
[warn] Serving filehandle without a content-length
This is some text that can be compressed.
This is some text that can be compressed.
This is some text that can be compressed.
This is some text that can be compressed.
This is some text that can be compressed.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Setting file handle as the response body generates warnings.
Speaking of downloads, I have a gzipped file. Well, it's a file in memory. If I see Accept-Encoding that includes "gzip" it's easy. I simply return the content and set Content-Encoding: gzip and set the content length. But, if the client does not accept gzip I uncompress it using IO::Uncompress::Gunzip. IO::Uncompress::Gunzip looks like a IO::File handle. So, I can do: $c->res->body( IO::Uncompress::Gunzip->new( \$gzipped_data ) ); Apache will add the header "Transfer-Encoding: chunked" because there's no Content-Length header. This works, but Catalyst in finalize_headers issues two warnings: -s on unopened filehandle GEN10 at /home/bill/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Catalyst.pm line 1893. [warn] Serving filehandle without a content-length Are those warning a problem with how Catalyst is handling this, or something wrong with how IO::Uncompress::Gunzip is working? The uncompressed file could be quite large, which is why I'd prefer to not uncompress it in memory. I suppose I could uncompress to /tmp and then serve the file from there. Of course, not using Catalyst to serve large files is perhaps another solution. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Content Disposition filename
Anyone aware of a good, portable way in Perl to encode the filename in a Content-Disposition header? I would like to support UTF8 filenames, but support in browsers is unclear (if not changing). Is this complexity something that the Catalyst framework should handle? It's one of those areas where it's easy to get wrong (I can see many different approaches in our own code). http://greenbytes.de/tech/tc2231/ http://stackoverflow.com/questions/93551/how-to-encode-the-filename-parameter-of-content-disposition-header-in-http Thanks, -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: Have exceeded the maximum number of attempts (1000) to open temp file/dir
On Mon, Nov 4, 2013 at 4:20 PM, neil.lunn wrote:
>
> I'd be thinking along the lines of mod_perl is evil. From a quick google
> of "mod_perl srand" there seem to be some similar cases. And a where to
> call srand in this post:
> http://blogs.perl.org/users/brian_phillips/2010/06/when-rand-isnt-random.html
>
> Give it a try.
>
Thanks. That indeed was part of the problem. Other part was a leak.
First, our code to manage database replication was causing a leak of $c.
That's why I could not replicate in dev where I don't normally run with a
replicated database.
The result was that a temp file would be created but not destroyed at the
end of the request. It would be destroyed at the start of the *next*
request.
So, the temp files would be deleted, but just not at the right time. The
one exception was the last request -- when Apache killed off a child
process then that final temp file would be left behind.
Here's where mod_perl comes in.
If I understand Perl correctly, the first time rand() is called a new seed
is generated. So, if you fork a bunch of children and each calls rand()
each will get a new seed. But, if you explicitly call srand() once in
the parent before forking then all the children get the same seed, and all
the children will generate the same random sequence.
When using Starman or even the forking test server this is not a problem as
I believe they both call srand() for each child. With Apache the children
end up with the same seed which likely means the parent process called
rand() or srand().
Over time, /tmp would fill, and fill with a pre-defined set of filenames
due to the common seed eventually the first 1000 names would get used up.
My fix is simply this in the app base class:
my $srand;
before handle_request => sub { srand() unless $srand++ };
The question if this is something Catalyst should handle.
As for the leak, we use the same replication code in different apps -- so
need a way for the specific app to work with this.
The broken code was in an ACCEPT_CONTEXT that included:
$replication_object->callback( sub { $self->callback_method( $c ) } );
I'm not thrilled by $c getting passed here, but the callback need quite a
bit from $c.
The "fix" that seems to work is simply this:
weaken( $c );
$replication_object->callback( sub { $self->callback_method( $c ) } );
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Re: Have exceeded the maximum number of attempts (1000) to open temp file/dir
On Fri, Oct 25, 2013 at 6:51 AM, Bill Moseley wrote:
>
> [ERROR] "Caught exception in engine "Error in tempfile() using
> /tmp/XX: Have exceeded the maximum number of attempts (1000) to
> open temp file/dir
>
I don't really see how this can be a Catalyst issue, but I can't reproduce
it outside of Catalyst -- and outside of our production environment.
Can anyone think of anything else that might be going on here?
The template has 10 "X" that are replaced by I think 63 random ascii
characters. 63^10 is a huge number of random strings. File::Temp only
loops when sysopen returns EEXISTS -- that is, when sysopen fails AND the
error is that the file already exists.
Sure, there's 50 web processes but the odds of them all being in exact
lock-step with calling rand() is unlikely. And even if they started out
that way if two processes opened the exact same name at the same time one
process would just try the next random name and be done.
I have something like 26K files in /tmp, so nothing compared to 63^10.
And each web server is only seeing about 10 request/sec.
It's just not making sense.
Again, I'm unable to replicate the problem with a simple test script that
is designed to clash.
I fork 50 (or more) child processes to replicate the web server processes
and then in each one I do this:
# Wait until top of the second so each child procss starts about
the same time.
my $t = time(); # Time::HiRes
sleep( int( $t ) + 1 - $t );
for ( 1 .. 500 ) {
my $fh = File::Temp->new(
TEMPLATE => 'bill_X',
DIR => '/tmp',
);
}
And never see any contention.
>
> The File::Temp docs say:
>
> If you are forking many processes in parallel that are all creating
>> temporary files, you may need to reset the random number seed using
>> srand(EXPR) in each child else all the children will attempt to walk
>> through the same set of random file names and may well cause
>> themselves to give up if they exceed the number of retry attempts.
>
>
> We are running under mod_perl. Could it be as simple as the procs all
> were in sync? I'm just surprised this has not happened before. Is there
> another explanation?
>
> Where would you suggest to call srand()?
>
>
> Another problem, and one I've
> commented<https://rt.cpan.org/Public/Bug/Display.html?id=84004>on before, is
> that HTTP::Body doesn't use File::Temp's unlink feature and
> depends on Catalyst cleaning up. This results in orphaned files left on
> temp disk.
>
>
>
>
>
> --
> Bill Moseley
> mose...@hank.org
>
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: Response traits.
On Fri, Nov 1, 2013 at 8:31 AM, John Napiorkowski wrote: > I think a patch that made sure strings being set to location conformed to > the expected standard would be very welcomed! > > on the other hand I thought this was also caught by > > https://metacpan.org/pod/Plack::Middleware::Lint > Yes, you might be right about that. That's probably the right (final) place to catch this. From the framework point of view I'm not quite clear what $res->redirect should accept. That is, if it contains \n should it get precent-encoded? Maybe it should (have) only accept a URI object? And only a fully-qualified URI.And what if redirect is passed wide characters? Maybe it's legit to pass wide characters encoded in some format. > > > as well? In any case Catalyst response could reject any attempts to set > ->location with incorrect values. > > The only thing is that people could still probably get around it by > setting headers directly. I think for now we'll say if you do that we have > to assume you know what you are doing! > One would hope. Experience often shows that manually doing it often is an indicator of not knowing what one is doing. The framework's benefit is so you don't screw things up. > > John > > > On Thursday, October 31, 2013 7:03 PM, Bill Moseley > wrote: > > > > On Thu, Oct 31, 2013 at 2:34 PM, John Napiorkowski wrote: > > I'm currently recommending people take advantage of native PSGI support in > the newer Catalyst and use Middleware for when you need to munge and or > alter the response (if its being done globally). The interface is more > straightforward. > > > Do you think that Catalyst::Response should validate the location provided > to redirect()? > > The issue that came up was a newline was ending up in the location > provided which resulted in a split > response<http://en.wikipedia.org/wiki/HTTP_response_splitting>. >I was thinking of doing something like: > > $self->location( URI->new( $location )->as_string ); > > But with perhaps a bit more error handling. > > > > > > > > johnn > > >On Thursday, October 31, 2013 11:33 AM, Bill Moseley > wrote: > > On Thu, Oct 31, 2013 at 12:51 AM, Aristotle Pagaltzis wrote: > > CatalystX::RoleApplicator > > > Thanks. That was what I was looking for. Just missed it when looking. > > > -- > Bill Moseley > mose...@hank.org > > ___ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > > > > _______ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > > > > > > -- > Bill Moseley > mose...@hank.org > > > -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: Response traits.
On Thu, Oct 31, 2013 at 2:34 PM, John Napiorkowski wrote: > I'm currently recommending people take advantage of native PSGI support in > the newer Catalyst and use Middleware for when you need to munge and or > alter the response (if its being done globally). The interface is more > straightforward. > Do you think that Catalyst::Response should validate the location provided to redirect()? The issue that came up was a newline was ending up in the location provided which resulted in a split response<http://en.wikipedia.org/wiki/HTTP_response_splitting>. I was thinking of doing something like: $self->location( URI->new( $location )->as_string ); But with perhaps a bit more error handling. > > johnn > > >On Thursday, October 31, 2013 11:33 AM, Bill Moseley > wrote: > > On Thu, Oct 31, 2013 at 12:51 AM, Aristotle Pagaltzis wrote: > > CatalystX::RoleApplicator > > > Thanks. That was what I was looking for. Just missed it when looking. > > > -- > Bill Moseley > mose...@hank.org > > ___ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > > > > ___ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > > -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Have exceeded the maximum number of attempts (1000) to open temp file/dir
On Thu, Oct 31, 2013 at 2:44 PM, John Napiorkowski wrote:
>
> am calling ->cleanup(1) when we create the HTTP::Body. is that not enough
> to cleanup tmp files ?
>
I haven't look at this in a while, but I think it's described here:
https://rt.cpan.org/Public/Bug/Display.html?id=84004
HTTP::Body assumes $self->{upload} exists before deleting, and that might
not be created yet.
I have my own version for handling 'multipart/form-data' that sets UNLINK
=> 1.
Now, the application/octet-stream handling is another issue. There
HTTP::Body uses the default File::Temp (e.g. UNLINK => 1), but I'm still
finding a large number of those files left around.
In my dev environment I have not been able to make it leave files on /tmp.
On production I can run watch 'ls /tmp | wc -l' and see the counts
increase and decrease so I know files are being deleted, but every once in
a while a file gets left behind. I don't see segfaults in the logs, and
I've tested with Apache's MaxRequestPerChild low (so recycling child
processes often) and not seeing that leave files behind.
I'm going to update our copy of HTTP::Body and put the process ID in the
temp file template to essentially namespace and use cron to keep /tmp
cleaner. But, I still have yet to figure out why those are left behind.
With UNLINK => 1 they should not be left there. File::Temp doesn't appear
to check the return value from unlink.
They come and go but some stick around:
$ for i in $(seq 10); do ls /tmp | wc -l; sleep 2; done
23861
23865
23863
23864
23862
23862
23865
23865
23864
23866
$ ls -lt /tmp | head -2
total 95492
-rw--- 1 tii-rest tii-rest 14 Oct 31 16:40 Nudjp9WDNy
$ ls -lt /tmp | tail -2
-rw--- 1 tii-rest tii-rest 16 Oct 28 13:36 NWwxOhwhRW
-rw--- 1 tii-rest tii-rest 16 Oct 28 13:35 Ll1Ze0TNPL
>
> regarding the tmp file thing, wow I have no idea, but I hope you find out
> and report it to us!
>
> Johnn
>
>
>On Friday, October 25, 2013 8:53 AM, Bill Moseley
> wrote:
> I have an API where requests can include JSON. HTTP::Body saves those
> off to temp files.
>
> Yesterday got a very large number of errors:
>
> [ERROR] "Caught exception in engine "Error in tempfile() using
> /tmp/XX: Have exceeded the maximum number of attempts (1000) to
> open temp file/dir
>
> The File::Temp docs say:
>
> If you are forking many processes in parallel that are all creating
> temporary files, you may need to reset the random number seed using
> srand(EXPR) in each child else all the children will attempt to walk
> through the same set of random file names and may well cause
> themselves to give up if they exceed the number of retry attempts.
>
>
> We are running under mod_perl. Could it be as simple as the procs all
> were in sync? I'm just surprised this has not happened before. Is there
> another explanation?
>
> Where would you suggest to call srand()?
>
>
> Another problem, and one I've
> commented<https://rt.cpan.org/Public/Bug/Display.html?id=84004>on before, is
> that HTTP::Body doesn't use File::Temp's unlink feature and
> depends on Catalyst cleaning up. This results in orphaned files left on
> temp disk.
>
>
>
>
>
> --
> Bill Moseley
> mose...@hank.org
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: Response traits.
On Thu, Oct 31, 2013 at 12:51 AM, Aristotle Pagaltzis wrote: > CatalystX::RoleApplicator Thanks. That was what I was looking for. Just missed it when looking. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Response traits.
What is the recommended way to apply a Response trait? Include it in a
response subclass similar to how Catalyst::Action::REST injects
Catalyst::Request::REST?
http://cpansearch.perl.org/src/JJNAPIORK/Catalyst-Action-REST-1.12/lib/Catalyst/Action/SerializeBase.pm
What I was interested in is wrapping redirect and filtering out any
white-space to prevent response splitting.
sub redirect {
my $self = shift;
if (@_) {
my $location = shift;
my $status = shift || 302;
$self->location($location);
$self->status($status);
}
return $self->location;
}
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Have exceeded the maximum number of attempts (1000) to open temp file/dir
I have an API where requests can include JSON. HTTP::Body saves those off to temp files. Yesterday got a very large number of errors: [ERROR] "Caught exception in engine "Error in tempfile() using /tmp/XX: Have exceeded the maximum number of attempts (1000) to open temp file/dir The File::Temp docs say: If you are forking many processes in parallel that are all creating > temporary files, you may need to reset the random number seed using > srand(EXPR) in each child else all the children will attempt to walk > through the same set of random file names and may well cause > themselves to give up if they exceed the number of retry attempts. We are running under mod_perl. Could it be as simple as the procs all were in sync? I'm just surprised this has not happened before. Is there another explanation? Where would you suggest to call srand()? Another problem, and one I've commented<https://rt.cpan.org/Public/Bug/Display.html?id=84004>on before, is that HTTP::Body doesn't use File::Temp's unlink feature and depends on Catalyst cleaning up. This results in orphaned files left on temp disk. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Optional path prefix
On Mon, Oct 21, 2013 at 9:08 AM, John Napiorkowski wrote:
> I'd probably myself want some plack middleware that would convert
>
> /api/v1/account//widget/
>
>
> to
>
>
> /account//widget/with accept type application/mycompany.v1+json
>
I guess that would separate that out of the app.
I'm currently using this approach in an app role:
my @path_seg = split '/', $c->req->path, -1;
my $base_uri = $c->req->base;
return unless @path_seg && $path_seg[0] =~ /$path_prefix_version_regex/;
my $match = $1;
die "path_prefix_version_regex ($path_prefix_version_regex) matched but
failed to capture any value"
unless defined $match;
$c->stash->{path_prefix_version} = $match;
$base_uri->path( $base_uri->path . shift( @path_seg ) . '/' );
# Force $req->path to reload _path next time $req->path is called.
$c->req->_clear_path;
>
> But you could probably support changing the URL path pretty easily with
> either setting the controller namespace to have v1 in it, or adding a root
> tot he change that specifies the new extra path part.
>
But would that support it being an optional prefix? Need both to work at
the same time.
> I understand the development word seems to prefer making version part of
> the path. depending on your logic and the type of changes introduced it
> may or may not be easier to take one approach or the other.
>
It does seem like that. Deciding to go with the flow vs. doing it the
"right" way is the decision to be made. I like your suggestion to map it
to an Accept header -- best of both worlds.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Catalyst::Request path method as a setter
On Mon, Oct 21, 2013 at 9:13 AM, John Napiorkowski wrote:
> My guess here is that path should be RO and that if you need to write code
> that messed with the path, that should happen in plack middleware.
>
Although I do want to let the app know what was removed from the path.
Essentially, (for better or worse) I have existing chained action
/account/123/foo/455 and want to add *optional* path prefixes. So, that
means accepting /api/v4/account/123/foo/455. So, the idea is simply pull
off the /api/v4/ from the path and stick it on $req->base base leaving the
original dispatching in place.
The version "v4" is to version specific end points. Something we don't
want to do very often. The version would be available in the stash so
that the endpoint can adjust its behavior.
I'm not thrilled by any of that, but that's the solution I'm looking at
right now.
>
> I'll ask around IRC to try and find out why this was allowed in the first
> place. My guess it that we wanted to allow people to change the path for
> doing complex path rewriting.
>
Thanks. Just seemed a bit odd that setting $req->path to its current
value would change $req->uri dramatically.
>On Monday, October 21, 2013 10:00 AM, Bill Moseley
> wrote:
>
> =head2 $req->path
>
>
>
> Returns the path, i.e. the part of the URI after $req->base, for the
> current request.
>
>
> Pasted below is Catalyst::Request's path method. Note from the final
> else block that $req->path returns the request uri's path ($req->uri->path)
> with the $req->base->path *removed* as the documentation says.
>
> So, if the request URI and base URI are these:
>
> http://localhost/myapp/path/to/action # $req->uri
>
> http://localhost/myapp/ # $req->base
>
>
> then $req->path is:
>
> path/to/action
>
>
> Using the example above, and looking at what $req->path( ) does as a
> setter:
>
> $req->path( $req->path );
>
>
> would result in a new request URI of:
>
> http://localhost/path/to/action.
>
>
> The path method doesn't document what it does as a setter, but this
> behavior looks broken because it alters the request URI's path.
>
> What do you think?
>
>
> sub path {
> my ( $self, @params ) = @_;
>
> if (@params) {
> $self->uri->path(@params);
> $self->_clear_path;
> }
> elsif ( $self->_has_path ) {
> return $self->_path;
> }
> else {
> my $path = $self->uri->path;
> my $location = $self->base->path;
> $path =~ s/^(\Q$location\E)?//;
> $path =~ s/^\///;
> $self->_path($path);
>
> return $path;
> }
> }
>
>
>
>
>
> --
> Bill Moseley
> mose...@hank.org
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
>
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Catalyst::Request path method as a setter
>
> =head2 $req->path
>
Returns the path, i.e. the part of the URI after $req->base, for the
> current request.
Pasted below is Catalyst::Request's path method. Note from the final else
block that $req->path returns the request uri's path ($req->uri->path) with
the $req->base->path *removed* as the documentation says.
So, if the request URI and base URI are these:
http://localhost/myapp/path/to/action # $req->uri
http://localhost/myapp/ # $req->base
then $req->path is:
path/to/action
Using the example above, and looking at what $req->path( ) does as a setter:
$req->path( $req->path );
would result in a new request URI of:
http://localhost/path/to/action.
The path method doesn't document what it does as a setter, but this
behavior looks broken because it alters the request URI's path.
What do you think?
sub path {
my ( $self, @params ) = @_;
if (@params) {
$self->uri->path(@params);
$self->_clear_path;
}
elsif ( $self->_has_path ) {
return $self->_path;
}
else {
my $path = $self->uri->path;
my $location = $self->base->path;
$path =~ s/^(\Q$location\E)?//;
$path =~ s/^\///;
$self->_path($path);
return $path;
}
}
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Optional path prefix
Over in this thread<http://www.mail-archive.com/catalyst@lists.scsys.co.uk/msg14226.html> was a discussion on API versioning and implementing via Accept: headers vs. adding a version in the URL. Looks like using a version in the URL is winning. We have existing chained actions that might look like this: /account//widget/ If want to migrate to a new version scheme in the URL like this: /api/v1/account//widget/ This would be the same action chain as the first path -- and both would work at the same time. Is there any way to support both actions via Chained dispatching? Or will I need a role that looks for that pattern and strips it of the request during prepare_action? I've done something similar in the past where I added a language tag at the start of every path: /en_us/some/path/1234 I strip that off and then update $c->req->path for dispatching. Again, I'm in the Accept: header camp for versioning, but I'm finding more and more discussion on using URLs.There's an e-book <http://pages.apigee.com/web-api-design-ebook.html>that seems to be cited often. I'd be interested in other's view on that book -- it seems written from a practical Rails programmer point of view instead of a REST purist view. There's a lot in that e-book I don't really agree with (plural nouns?), but the practical usage seems to be winning out. Hope it's not a mistake in the long run. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] REST and versioning
On Tue, Sep 17, 2013 at 10:12 AM, John Napiorkowski wrote:
>
> People seem to get religious over how to version their API. I doubt I'd
> want to take sides on this but here's how I think we could do both side
> (URL version and content type versioning
Ya, I'm swayed by the Accept: header approach because in my mind
/api/v1/account/123 and /api/v2/account/123 seems like different resources.
(Well, I guess they are.) So, the versions are lazy way to make a new
resource location.
And even more so, seems like a dark path to go down. Is just that
individual resource versioned or is the entire API versioned? And if it's
the entire API, and the app needs to support multiple versions at the same
time, then need a way for methods to "fall-back" to v1 when only a few
methods change.
Or maybe the client would have to know which methods are v2 vs. v1 and
pick-and-choose.
Realistically, the problem that would likely come up is more related to
client versioning where an old client cannot support some new feature of
the API.
For example, say a service has a method to fetch a widget and a method to
list them.
GET /widget/123 # get widget 124
GET /widget # list all widgets.
So, some client app is designed to list the widgets and then fetch them
(for display or whatever).
Later, the service is upgraded and adds a new *type* of widget -- and that
is a type that the existing old client app cannot support.
Does the service need to know what the client can support?
sub widget_GET : Args(0) ... {
...
push @include_types, $new_widget_type if $client_version >= 1.1;
That's going to lead to some nasty spaghetti code over time.
Maybe not such a great example as one could argue here that the client
could GET /widget?type=1&type=2&type=3, but other changes might make that
not so easy.
In your chained example below how does that work with longer paths?
I'm trying to come up with a good example.
/v1/document/1234 # fetch a document
/v1/document/1234/share # list who the document is shared with.
Then what happens if it's just the share method that has a new version?
> package Myapp::Web::Controller::API;
>
> use base 'Catalyst::Controller';
>
> sub start : ChainedParent
> PathPrefix CaptureArgs(0)
> {
> my ($self, $ctx) = @_;
> }
>
> sub version_one : Chained('start') PathPart('1') Args(0) { ... }
>
> sub version_two : Chained('start') PathPart('2') Args(0) { ... }
>
> 1;
>
> package Myapp::Web::Controller::API::1;
>
> use base 'Catalyst::Controller';
>
> sub start : ChainedParent
> PathPrefix CaptureArgs(0)
> {
> my ($self, $ctx) = @_;
> }
>
> 1;
>
> package Myapp::Web::Controller::API::2;
>
> use base 'Catalyst::Controller';
>
> sub start : ChainedParent
> PathPrefix CaptureArgs(0)
> {
> my ($self, $ctx) = @_;
> }
>
> 1;
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] REST and versioning
I've once again used up an hour or so reading Stack Overflow and blog posts
on REST API versioning. (And the arguments for and against.)
Perhaps extending the discussion on how Catalyst supports REST:
https://github.com/perl-catalyst/CatalystX-Proposals-REStandContentNegotiation
I'm wondering if Catalyst might help in supporting API versions. Somewhat
similar to how Catalyst::Action::REST will call methods based on the HTTP
method, perhaps call actions based on some version (provided by some means
-- like a version in the path or in an Accept header).
Catalyst::Action::REST helps keep the actions tidy by calling methods
specific to each method (foo_GET, foo_PUT). Obviously, we could simply
check if ( $req->method eq 'GET' ) but would end up with pretty ugly
actions and no automatic "Allow" header.
With versions I'm concerned about that my foo_GET method will end up with a
bunch of "if ( $version > 1.1 ) {} elsif ($version >1.0 ) ...
So, running with the C::Action::REST approach, something like:
sub foo_GET { ... } # Default
sub foo_GET : Version( 1.1 ) { ... } # Use if client requests version is
1.1
Frankly, seems like maintenance nightmare and Action explosion. Where
that version comes from (url, Accpet header) is often debated (see links).
Any better ideas how to support versioning in Catalyst actions?
The subject of versioning is a bit overwhelming. Here's some starting
points, if curious:
-
http://stackoverflow.com/questions/389169/best-practices-for-api-versioning
-
http://www.lexicalscope.com/blog/2012/03/12/how-are-rest-apis-versioned/
- http://www.subbu.org/blog/2008/05/avoid-versioning-please
-
http://stackoverflow.com/questions/2024600/rest-api-versioning-only-version-the-representation-not-the-resource-itself?lq=1
- http://www.informit.com/articles/article.aspx?p=1566460
- http://stackoverflow.com/questions/972226/how-to-version-rest-uris
- and plenty more...
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] More detailed proposal for changes related to content negotiation and REST
st->body_data is intended to be lazy, we won't run that parse > code until you ask for the data. We don't need to parse the data to do the > basic match here, this is just based on the HTTP meta data, no the actual > content. I think for common cases this is fine (I realize that yet again > this might not be the best approach for multipart uploads...) > Another tough one. Just seems like PUT /user should accept the same data regardless of how it is serialized. And GET /user would get the user data and then serialize that to JSON or whatever but it's the same data. But, maybe you have a point.I would worry that someone assumes JSON and adds that content type match and then wonder why later it's not working for other request serializations. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] More detailed proposal for changes related to content negotiation and REST
On Thu, Aug 8, 2013 at 9:27 PM, John Napiorkowski wrote:
>
>
> https://github.com/perl-catalyst/CatalystX-Proposals-REStandContentNegotiation
>
I currently extend HTTP::Body in a similar way that you describe. But I
have them a separate classes so I just do:
use My::Multipart;
then in that I hack in my type for HTTP::Body:
package My::MultiPart;
use HTTP::Body;
$HTTP::Body::TYPES->{'multipart/form-data'} = 'My::MultiPart';
As you propose, mapping a mime type to a sub seems pretty flexible. I
assume the sub could return a filehandle. File uploads still need to
stream the uploads to disk while making the parameters available as
HTTP::Body does now.
I like the regex mimetype keys, but should they be an array instead of a
hash so can set the order checked?
I think we must consider large inputs/streams.You say $_ is the request
content. Is that the full request with headers? Is the request already
de-chunked, for example? Or am I thinking too low level?
In some apps I'm using Catalyst::Action::REST and others I have some custom
code where I use HTTP::Body to parse JSON. I'm mixed about having the
request data end up in $c->req->parameters vs $c->req->data.I don't
really see why form-data/urlencoded should be treated differently than
other encodings like JSON.
I not quite sure about $c->res->body( \%data ); I think body should be
the raw body. What does $c->res->body return? The serialized json? The
original hashref?
If a parser dies what kind of exception is thrown? You say they would not
set any response status, but wouldn't we want to catch the error and then
set a 400? (I use exception objects that carry http status, a message to
return in the body and a message used for logging at a given level.)
I'm not sure what to think of the Provides and Consumes controller
attributes. It's one thing to realize early that the client cannot handle
any response I'm willing to provide (json, yaml, whatever), but I'm worried
this would blur separation of concerns. That is, by the time the
controller runs we would have already decoded the request and want to just
work with the data provided. And would we want different data returned if
the client is asking for json vs yaml?
Maybe I'm missing the point of that.
BTW - I practice I find it pretty handy to be able to specify/override
response encoding via a query param.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] CSV / UTF-8 / Unicode
On Tue, Jul 2, 2013 at 9:29 AM, Craig Chant wrote: > >> All the above seems overkill. I suspect what you want is closer to > this: (but see notes below). > > > > Tried that, didn’t work, ended up in a long Catalyst discussion where it > was worked out that I needed to wrap any XLS output to an IO:FILE handle > otherwise Catalyst dies with an “out of memory” error something to do with > streaming data support issues in Catalyst so the work round is to wrap the > output into an IO:File object. > $xls is a scalar you have in memory of the comma separated value data? > > > >>Second, be aware that $c->response->content_length(length($xls)); > > > > Yes, I was doing the encode then using Length (I did read on perldocs > about requesting the length against the octet) , either way, the length was > the least of my worries, keeping Catalyst from falling over with ‘Wide > Character’ errors, or not getting garbage was my main concern. > > > > And yes, the output is CSV not strictly XLS but I have been told and > looked it up on the net that 'application/vnd.ms-excel' Is the correct > MIME header to pass for CSV that you want MS Excel to open. > http://www.ietf.org/rfc/rfc4180.txt, but how Windows decides how to open files is something I'm not familiar with. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] CSV / UTF-8 / Unicode
Before we get a long utf8 discussion here: On Tue, Jul 2, 2013 at 8:58 AM, Bill Moseley wrote: > > Personally, I think the correct approach is to only encode *character* data > -- that is check to see if the utf8 flag is set before calling encode. > I say that with the caveat that ALL textual input data is correctly decoded -- as it should be. That is, everything that was decoded into characters has the utf8 flag set. $ perl -MEncode -le 'print "yes, this is characters" if Encode::is_utf8( Encode::decode( "ASCII", "This is ASCII"))' yes, this is characters -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] CSV / UTF-8 / Unicode
On Tue, Jul 2, 2013 at 2:59 AM, Craig Chant
wrote:
>
> # output header
> $c->response->content_type('application/vnd.ms-excel');
> $c->response->content_length(length($xls));
> $c->response->header(Content_Disposition =>
> 'attachment;filename=NBCS_Export.csv');
>
> # create an IO::File for Catalyst
> use IO::File;
> my $iof = IO::File->new;
>
> $iof->open(\$xls, "r");
> $iof->binmode(":encoding(UTF-8)");
>
> # output XLS data
> $c->response->body($iof);
>
All the above seems overkill. I suspect what you want is closer to this:
(but see notes below).
$c->response->content_type('text/csv');
$c->response->body($xls);
$c->response->header(Content_Disposition =>
'attachment;filename=NBCS_Export.csv');
Then with that content type the plugin would encode $xls as utf8 and add
;charset=utf8 (or whatever it is configured to encode as).
Notes:
First, you are not returning Excel, so the content type is not what you
first listed above, right?
Second, be aware that $c->response->content_length(length($xls)); could be
very wrong. If $xls is really CSV text AND it's decoded then length($xls)
is the length in characters, not octets. Don't set the content length.
Third, Catalyst::Plugin::Unicode::Encoding, IMO, has some issues.
The plugin limits to just these content types.
return $c->next::method(@_)
unless $c->response->content_type =~ /^text|xml$|javascript$/;
Then it does this:
$c->response->body( $c->encoding->encode( $body, $CHECK ) )
if ref(\$body) eq 'SCALAR';
Personally, I think the correct approach is to only encode *character* data
-- that is check to see if the utf8 flag is set before calling encode.
Maybe limit to the content types listed above, but throw an exception for
other content types where the body is a scalar AND has the utf8 flag on.
After all, we can only write out octets or else we get the Wide Character
error.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] unicode warnings (+ errors C::P::UploadProgress)
On Fri, Jun 14, 2013 at 11:29 AM, John Napiorkowski wrote:
> Yup this is a regression! We have a patch in HEAD and it should go out
> later today. Sorry for the trouble ( but at least I know who's upgrading)
>
I upgraded yesterday and noticed the same thing in out tests:
[error] Caught exception in engine "Can't use string ("Can't call method
"decode" on an"...) as a HASH ref while "strict refs" in use at
/Users/bill/perl5/perlbrew/perls/perl-5.16.1/lib/site_perl/5.16.1/Catalyst/Plugin/Unicode/Encoding.pm
line 101."
I'll wanted to investigate I have my own code that handles decode/encode
and need time to move to the Catalyst code. Until we can migrate what
would you recommend so that we don't have two plugins trying to handle the
decoding?
I also noticed that UploadProgress got pulled in -- but I don't think we
are using that so was a bit surprised by that requirement.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] /Depreci?ate/ message
Could this message be disabled by environment variable or config? Besides ending up in logs, we have tests that are failing because of it. Our tests trap all warnings and will fail on unaccounted for warnings. Thanks, On Thu, May 23, 2013 at 3:16 PM, John Napiorkowski wrote: > speeling error noted! thanks! -john > > - Original Message - > > From: Toby Corkindale > > To: The elegant MVC web framework > > Cc: > > Sent: Wednesday, May 22, 2013 1:52 AM > > Subject: [Catalyst] /Depreci?ate/ message > > > > I noticed that the recent Catalyst release has added a Depreciation > > warning to the Regex class. > > Are you sure you meant depreciate? I think old features are normally > deprecated. > > > > Cheers, > > Toby > > > > -- > > Turning and turning in the widening gyre > > The falcon cannot hear the falconer > > Things fall apart; the center cannot hold > > Mere anarchy is loosed upon the world > > > > ___ > > List: Catalyst@lists.scsys.co.uk > > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > > Dev site: http://dev.catalyst.perl.org/ > > > > ___ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > > -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Chained and exceptions
On Fri, May 10, 2013 at 1:29 AM, Tomas Doran wrote: > > > You're after this: > https://metacpan.org/module/Catalyst::ActionRole::DetachOnDie > > which gives you the alternate behaviour (i.e. detaching from the chain on > first exception). > We have a number of applications, a few quite large, where some controllers inherit from different base classes. We could try and retro fit all existing code, but it would be a good-sized project. So the monkey patch we did (as well as Dami Laurent had done in 2008<http://lists.scsys.co.uk/pipermail/catalyst/2008-March/017789.html>) is better for us. I'm pretty sure this issue is not well known amongst our current (and future) developers and thus it's quite likely someone would forget this in a new Controller. We all understand that an uncaught exception should not bring down the server and instead generate a 500, but I think few would assume that when using Chained an exception would not stop the request dead in its tracks and instead is implicitly trapped and allowed to continue. I think the more likely situation now is code running when it is not expected -- which could be a serious security issue if the earlier action in a chain is used for access control. What would the developers think of deprecating this behavior (for the few that might actually be relying on this) and issue a warning if a config option is not set that fixes the issue? -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Re: Chained and exceptions
On Thu, May 9, 2013 at 9:34 AM, Aristotle Pagaltzis wrote:
> * Bill Moseley [2013-05-09 15:30]:
> > What's the reasoning that chained actions continue to run after an
> > earlier exception?
>
> Seems like an accident of the design to me, borderline bug.
>
Agreed. Seems like something that could be easily overlooked.
>
> If like me you don’t like it, Catalyst::ActionRole::DetachOnDie
>
Oh, that's nice. Tricks for applying it globally?
I went the monkey patch route. Shield your eyes:
use Catalyst::ActionChain;
sub Catalyst::ActionChain::dispatch {
my ( $self, $c ) = @_;
my @captures = @{$c->req->captures||[]};
my @chain = @{ $self->chain };
my $last = pop(@chain);
foreach my $action ( @chain ) {
my @args;
if (my $cap = $action->number_of_captures) {
@args = splice(@captures, 0, $cap);
}
local $c->request->{arguments} = \@args;
$action->dispatch( $c );
* return if @{ $c->error }; # Patch*
}
$last->dispatch( $c );
}
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Chained and exceptions
On Thu, May 9, 2013 at 8:31 AM, Bill Moseley wrote: > Hi Bill, > >> >> This is because you don't want Catalyst to die. Imagine you are running >> a fastcgi server and you accidentally created an action which dies on >> certain user input. >> > > Hi Lukas, > > Sorry, you missed the point. Yes, Catalyst traps exceptions. That is > expected and is done in handle_request when calling $c->dispatch. > No, sorry, I responded too quickly. I'm not talking about the entire have exiting, of course. I'm just not sure it makes sense to not check $c->errors after calling each action in a chain. What's the use case for that? -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Chained and exceptions
On Thu, May 9, 2013 at 7:25 AM, Lukas Thiemeier
wrote:
> On 05/09/2013 03:25 PM, Bill Moseley wrote:
> > I have a feeling I asked this before, but cannot find the post.
> >
> > [info] Exception powered by Catalyst 5.90030
> >
> > What's the reasoning that chained actions continue to run after an
> > earlier exception?
> >
> >
> > sub start : Chained( '/' ) : CaptureArgs(0) {
> > warn "in start\n";
> > }
> >
> > sub middle : Chained( 'start' ) : CaptureArgs(0) {
> > warn "in middle\n";
> > die "died in middle\n"; # or e.g. throw access violation
> > }
> >
> > sub lastpart : Chained( 'middle' ) : Args(0) {
> > my ( $self, $c ) = @_;
> > $c->res->body( "finished\n" );
> > warn "in lastpart\n";
> > }
> >
> > $ script/exception_test.pl <http://exception_test.pl>
> /start/middle/lastpart
> > in start
> > in middle
> > *in lastpart*
> > [error] Caught exception in Exception::Controller::Root->middle "died in
> > middle"
> >
>
> Hi Bill,
>
> This is because you don't want Catalyst to die. Imagine you are running
> a fastcgi server and you accidentally created an action which dies on
> certain user input.
>
Hi Lukas,
Sorry, you missed the point. Yes, Catalyst traps exceptions. That is
expected and is done in handle_request when calling $c->dispatch.
I'm talking about breaking a chain of actions. Why would a later part
of the chain run if an earlier part threw an exception. For example, what
if an earlier part of a chain threw a access violation. In that case you
would not want the later chain to run.
> If Catalyst would die, the fastcgi server would die and your whole
> application would not be available anymore. Instead, you want to report
> the incident and keep the fastcgi server (Catalyst) running.
>
> Because of this, every action is wrapped in an "eval{...}". Potential
> errors are logged, but the server keeps running.
>
> See "execute" in Catalyst.pm for implementation details.
>
>
> To solve your problem, you can wrap your unsafe code in an eval or use
> Try::Tiny (or whatever you prefer) and detach if the unsafe code dies.
>
> Your Example would look like this:
>
> use Try::Tiny;
>
> sub start : Chained( '/' ) : CaptureArgs(0) {
> warn "in start\n";
> }
>
> sub middle : Chained( 'start' ) : CaptureArgs(0) {
> my ($self, $c) = @_;
> warn "in middle\n";
> try{
> die "died in middle\n"; # or e.g. throw access violation
> }
> catch{ $c->detach };
> }
>
Don't forget your semicolon. :)
>
> sub lastpart : Chained( 'middle' ) : Args(0) {
> my ( $self, $c ) = @_;
> $c->res->body( "finished\n" );
> warn "in lastpart\n";
> }
>
> If you do it like this, actions chained to the unsafe action will not
> get executed if the unsafe action dies. In your case, "lastpart" will
> never be executed, because "middle" always dies.
>
> Lukas
>
> ___
> List: Catalyst@lists.scsys.co.uk
> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
> Searchable archive:
> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
> Dev site: http://dev.catalyst.perl.org/
>
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Chained and exceptions
I have a feeling I asked this before, but cannot find the post.
[info] Exception powered by Catalyst 5.90030
What's the reasoning that chained actions continue to run after an earlier
exception?
sub start : Chained( '/' ) : CaptureArgs(0) {
warn "in start\n";
}
sub middle : Chained( 'start' ) : CaptureArgs(0) {
warn "in middle\n";
die "died in middle\n"; # or e.g. throw access violation
}
sub lastpart : Chained( 'middle' ) : Args(0) {
my ( $self, $c ) = @_;
$c->res->body( "finished\n" );
warn "in lastpart\n";
}
$ script/exception_test.pl /start/middle/lastpart
in start
in middle
*in lastpart*
[error] Caught exception in Exception::Controller::Root->middle "died in
middle"
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] How to get IP address of the interface which the request come through
On Sat, Mar 23, 2013 at 7:31 AM, N.A. wrote: > (2013年03月23日 23:03), Peter Flanigan wrote: > > On 23/03/13 12:12, N.A. wrote: > >> I wan to get the IP address of the interface(network device) which the > >> request come through. > > > > My bad. > > > > Use $c->req->uri->host to get the hostname of the server > > > > Sorry, $c->req->uri->host is 'hostname', not a IP. > I want to get IPv4 address like '127.0.0.1' even if I access the page > by 'http://localhost/XXX' > Interesting. What's your use case here? -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Catalyst, ExtJS and TT
On Mon, Mar 18, 2013 at 3:56 AM, Dave Howorth wrote:
> >
> > I have a javascript file named EntrepriseWindow.js in root dir. It
> > contains some TT code. I must add a extension (I renamed the file
> > EntrepriseWindow.js) for it to work. Otherwise, it does not work :
> >
> > sub EntrepriseWindow :Path('EntrepriseWindow.js') {
> >
> > my ($self, $c) = @_;
> >
> > $c->stash->{template} = 'EntrepriseWindow.js.tt';
> >
> > $c->response->content_type('application/javascript');
> >
> > }
> >
> > I cannot write this ?
> >
> > $c->stash->{template} = 'EntrepriseWindow.js';
>
> Most frameworks that use TT have a mechanism to find template source
> files based on their extension. In the case of Catalyst, see
>
>
> http://search.cpan.org/~jjnapiork/Catalyst-View-TT-0.41/lib/Catalyst/View/TT.pm#TEMPLATE_EXTENSION
You should be able to specify the full template name via the stash. Here's
the code:
sub process {
my ( $self, $c ) = @_;
my $template = $c->stash->{template}
|| $c->action . $self->config->{TEMPLATE_EXTENSION};
If the template is explicit in the stash then TEMPLATE_EXTENSION should not
come into use.
> If you think about it, the file you want TT to process is *not* a .js
> file, it's a file with some TT syntax in it that needs processing to
> *become* a .js file. So it would be unhelpful if Catalyst tried to
> TT-process real .js files and it's reasonable, IMHO, to have something
> like a .js.tt extension.
>
In this case Jean-Marc said that EntrepriseWindow.js has some TT code
added. I agree that it would be clearer if it had a different extension
so that later someone does not mistaken it for a js file.
But, a javascript file really should be served statically.
Jean-Marc, what per-request changes are you making to that file?
Something like setting strings based on the user's language preference for
that request?
I would think it would be better to serve it statically (with caching
headers) and then serve any pre-request changes in the actual html
response. Serving up different javascript files with the same name could
lead to problems.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Re: Not cleaning up temporary files / HTTP::Body
Just to be clear, this opens up all Catalyst apps to a pretty easy DoS --
or in our case denial of sleep as people were woken up in the middle of the
night as servers ran low on disk space. Good thing we have monitoring set
up.
The risk to any individual Catalyst user is obviously limited, but
something to be aware of.
The design flaw here means Catalyst apps, by default, give write acces to
/tmp (which often isn't that big) to any user of your app and doesn't clean
up after itself. Running out of /tmp space is never fun.
HTTP::Body should be fixed because of this issue, and I've created a ticket
against it. A fix there could be done w/o any changes needed for Catalyst.
But, there's other things to consider. For example, there's nothing to
limit the size of a single request. Some of those tests should be higher
up in the stack.
Apache has a setting "LimitRequestBody" but IIRC that only looks at the
Content-Length header. But, HTTP::Body does limit the body to what is
reported in the Content-Length header.
Starman buffers requests to /tmp (in an unlinked file) w/o any limit that I
could see in my limited tests.
On Thu, Mar 14, 2013 at 9:53 PM, Bill Moseley wrote:
>
>
> On Wed, Jan 13, 2010 at 6:53 AM, Bill Moseley wrote:
>
>> HTTP::Body::Multipart creates temporary files for uploads. The temp
>> files are created with File::Temp( UNLINK => 0 ).
>>
>
> Well, this is still broken.
>
> Yes, since 2010 HTTP::Body has been updated to have a DESTROY where it
> removes files, and Catalyst::Request calls the $body->cleanup(1) attribute
> to enable this feature.
>
> But, the problem is DESTROY (see below) looks for files to remove in
> $self->{upload} hash. But, that is an empty hash until the upload is
> completed. So, if the upload is aborted (which is when you need the clean
> up to work) there's nothing in the upload hash -- so nothing to unlink.
>
> Three years ago when I first posted about this I copied
> HTTP::Body::MultiPart and set UNLINK => 1 on File::Temp->new and created
> my own version that overrides.That still works.
>
> But, if I simply modify the current H::B::MultiPart and add UNLINK=>1 then
> the file gets deleted and I get an error. So, the fix isn't that simple any
> more. That needs some looking at.
>
>
> Clearly, what we are after is deleting the temp files. So, might as well
> let File::Temp do its job. Just have to make sure the handle doesn't go
> out of scope too early.
>
>
> Oh, BTW, you can't test this with the dev server. I had to use Apache.
> With Apache chunks are sent to Catalyst. But, when running on the dev
> server it seems like the entire upload was buffered before prepare_body
> gets called -- so even with a 16MB upload I could not hit "esc" to abort
> the upload while HTTP::Body was processing.
>
> sub DESTROY {
> my $self = shift;
>
> if ( $self->{cleanup} ) {
> my @temps = ();
> *for my $upload ( values %{ $self->{upload} } ) {*
> push @temps, map { $_->{tempname} || () }
> ( ref $upload eq 'ARRAY' ? @{$upload} : $upload );
> }
> unlink map { $_ } grep { -e $_ } @temps; ## this is empty when
> aborted
> }
>
> }
>
>
>
>
>
>>
>> Catalyst then deletes these temp files in $c->finalize. The problem is
>> that an exception can happen and then the temp files are not deleted.
>> Happens quite often, it turns out. I seem to always see this in the logs
>> at the time of the orphaned temp file:
>>
>> Caught exception in engine "Apache2::RequestIO::read: (104) Connection
>> reset by peer
>>
>> Aborting an upload isn't that unusual.
>>
>> I can set temp directory for these uploads and use cron to clean them
>> out, but I wonder if they could not be cleaned up better automatically.
>>
>> For example, unlink in DESTROY when the request object goes out of
>> scope. Or perhaps better, don't have HTTP::Body set UNLINK => 1 by default
>> and when the upload object finally goes out of scope File::Temp will remove
>> the temp file. HTTP::Body explicitly removes the File::Temp object from the
>> upload part, but I'm not sure why it needs to do that. Why not leave the
>> File::Temp object in the upload part object then let it go out of scope at
>> the end of the request.
>>
>> Where should this be addressed? In Catalyst or in HTTP::Body?
>>
>> --
>> Bill Moseley
>> mose...@hank.org
>>
>
>
>
> --
> Bill Moseley
> mose...@hank.org
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Re: Not cleaning up temporary files / HTTP::Body
On Wed, Jan 13, 2010 at 6:53 AM, Bill Moseley wrote:
> HTTP::Body::Multipart creates temporary files for uploads. The temp files
> are created with File::Temp( UNLINK => 0 ).
>
Well, this is still broken.
Yes, since 2010 HTTP::Body has been updated to have a DESTROY where it
removes files, and Catalyst::Request calls the $body->cleanup(1) attribute
to enable this feature.
But, the problem is DESTROY (see below) looks for files to remove in
$self->{upload} hash. But, that is an empty hash until the upload is
completed. So, if the upload is aborted (which is when you need the clean
up to work) there's nothing in the upload hash -- so nothing to unlink.
Three years ago when I first posted about this I copied
HTTP::Body::MultiPart and set UNLINK => 1 on File::Temp->new and created
my own version that overrides.That still works.
But, if I simply modify the current H::B::MultiPart and add UNLINK=>1 then
the file gets deleted and I get an error. So, the fix isn't that simple any
more. That needs some looking at.
Clearly, what we are after is deleting the temp files. So, might as well
let File::Temp do its job. Just have to make sure the handle doesn't go
out of scope too early.
Oh, BTW, you can't test this with the dev server. I had to use Apache.
With Apache chunks are sent to Catalyst. But, when running on the dev
server it seems like the entire upload was buffered before prepare_body
gets called -- so even with a 16MB upload I could not hit "esc" to abort
the upload while HTTP::Body was processing.
sub DESTROY {
my $self = shift;
if ( $self->{cleanup} ) {
my @temps = ();
*for my $upload ( values %{ $self->{upload} } ) {*
push @temps, map { $_->{tempname} || () }
( ref $upload eq 'ARRAY' ? @{$upload} : $upload );
}
unlink map { $_ } grep { -e $_ } @temps; ## this is empty when
aborted
}
}
>
> Catalyst then deletes these temp files in $c->finalize. The problem is
> that an exception can happen and then the temp files are not deleted.
> Happens quite often, it turns out. I seem to always see this in the logs
> at the time of the orphaned temp file:
>
> Caught exception in engine "Apache2::RequestIO::read: (104) Connection
> reset by peer
>
> Aborting an upload isn't that unusual.
>
> I can set temp directory for these uploads and use cron to clean them out,
> but I wonder if they could not be cleaned up better automatically.
>
> For example, unlink in DESTROY when the request object goes out of scope.
> Or perhaps better, don't have HTTP::Body set UNLINK => 1 by default and
> when the upload object finally goes out of scope File::Temp will remove the
> temp file. HTTP::Body explicitly removes the File::Temp object from the
> upload part, but I'm not sure why it needs to do that. Why not leave the
> File::Temp object in the upload part object then let it go out of scope at
> the end of the request.
>
> Where should this be addressed? In Catalyst or in HTTP::Body?
>
> --
> Bill Moseley
> mose...@hank.org
>
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Using the Catalyst Makefile to install
On Mon, Mar 11, 2013 at 1:29 PM, Alejandro Imass wrote: > > > This pre-dates Module::ShareDir. > > > > So the templates, images (i.e. the /root) are considered shared data? > When I was converting our Catalyst apps to use Dist::Zilla I looked at changing where Catalyst looks for /root -- just so I could use Module::ShareDir (supported by Dist::Zilla) and not have to rewrite Module::Install::Catalyst. I think I had some discussion of these back in July: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/msg13186.html In the end I did implement the code to copy in the /root (and other files and dirs) into blib just like Module::Install::Catalyst does. And the configuration file should go for example in /etc in Debian and > /usr/local/etc on FBSD?? > We deploy with RPM. FWIW, this is something we discussed at work quite a bit. Operations wanted config in standard location, but we wanted config with the application. In general, of our config is more constants that config and so there's rarely changes to our app config. Plus, having config with the app makes it a bit easier to have multiple copies installed. (In the past used symlinks to point to live version.) In other words, the bulk of our config is more like static data, so placing in share would be ok. What we did in the end to make the site operations team happy was update our config system to look in /etc/$app_name/config.yml for an extra config file that would override (merge with) the app's config. Our config loads config.yml filrst, then it loads and merges any mode-specific (i.e. qa, staging, production) config, then finally looks for /etc/$app_name/config.yml. Some day we will look more closely at centralized configuration tools -- Puppet and Chef, for example. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Mon, Mar 11, 2013 at 4:53 AM, Will Crawford wrote: > While it's not Catalyst's fault, I've found over the years that > interacting with underlying libraries, databases and legacy systems is > generally easier when I *don't* try to force anything. I have custom code > in place to deal with know sources of inconsistent encodings (check to see > if it's valid UTF8, look up the remainder in a table painstakingly > assembled over a period of time that catches a few odd MacRoman characters > that show up in some of our contributors' data, fall back to latin1 or > cp1252 for the remainder, leave anything else as \xNN). Everywhere else, > UTF8 can be passed through quite transparently, so I don't really see the > point of adding extra decoding and encoding all over the place to switch > from utf8, to some internal wide character encoding, then back to utf8 > again for output. One of the positive features of UTF8 has always been that > code that doesn't need to identify any of those fancy accented characters > can just treat it the same as ASCII, Latin-$WHATEVER or cp1252 without any > overhead. Overall I can't see the point of forcing everything to be > converted multiple times ... > > I think we can all agree that historically encoding has been confusing, misunderstood, and frequently ignored. And very often just done plain wrong. I suspect since this is currently a plugin that it's often ignored, especially by newer developers. That means "out of the box" Catalyst, as a web framework, for its typical use, is broken. One of the typical uses for a Catalyst application is building a web app that outputs character data. This character data must be encoded when sent over the wire. Likewise, request data that is character data must be decoded. Doing these as close to the "edge" of the application as possible is the best approach. That's what the plugin does. As t0m says, if you are ignoring encoding your app is broken. Sure, it may not seem so. Sure, you can ignore those "fancy accented characters" and if you app only works in an something like ASCII never notice -- it's just like before Unicode support was added to Perl. And you still should set a charset on the content-type when you send the response, so what are you going to set it to? Plus, once you do get some of those fancy characters (used by billions of people) into your app then all those length() and every other thing that works with characters (hey, this is Perl) will be broken. My wild guess from your description above is you are not handling encoding correctly. But, in the real world you get character data thrown at you that is broken in some way. Perhaps your input is so broken you have to do what you described. (Still, I think the correct approach is to decode() with a useful CHECK value.) If you are "passing through" UTF8 undecoded then unless you not touching that input (as character data) then that's broken. You say it's best not to force anything, which I assume you mean force as some encoding. If you have character input then by nature it's encoded. You have to know what the encoding is, and decode it as such and be prepared for bad data. You wouldn't ignore it if it was base64 or gzipped, right? Those are not character encodings, but it's essentially the same issue. I have never considered any performance aspect of this. It never shows up when we profile "slow" responses. Plus, it's never been an optional operation. We manipulate characters and we exchange data as bytes. You have to convert between those. The plugin should be core to Catalyst. It think it's pretty safe to add it if it only encodes if the utf8 flag is set on the body -- that should prevent double-encodings. And having a config option to disable is easy. And if the plugin is found on the app issue a warning. It's possible that someone has their own modified version of the plugin using the same name. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Sun, Mar 10, 2013 at 2:06 PM, Peter Flanigan wrote: ># Only touch 'text-like' contents > return $c->next::method(@_) > unless $c->response->content_type =~ /^text|xml$|javascript$/; > > smells bad, it will apply to text/x-json but not application/json > The code that I use only encodes if the utf8 flag is set. The JSON encoders (at least what I use) encode to utf8, so my code doesn't touch JSON as it's already been encoded. I decode all input -- database, templates, text files, web requests and API requests. All data that represents characters must be decoded on input. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Using the Catalyst Makefile to install
On Fri, Mar 8, 2013 at 1:43 PM, Alejandro Imass wrote: > Thanks to both of you for your valuable insight. > > We've also looked at the PAR route. Up until now we've always > installed manually and only used the Makefile to keep track of > dependencies and easier installs of them, but we've never used it to > install the app as such. We are currently deploying a beautiful code > and we wanted to do it end-to-end including the make install. We're > going to experiment with this first and then analyze all other > options. > Just to add a bit: As you know, Catalyst is like any other Perl module. So, we deploy our Catalyst apps to an internal "DarkPAN" along with all our other internal modules. What that means is on a new machine (with a few environment variables set) you can type: cpanm "Our::Catalyst::App" and it an all its dependencies are installed. So, that Makefile actually comes in pretty handy. (And since we use Dist::Zilla, releasing to our internal CPAN is just "dzil release". So, that's pretty slick.) We then build RPMs from these modules and that's what is used for deployment. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Using the Catalyst Makefile to install
On Thu, Mar 7, 2013 at 8:57 PM, Francisco Obispo wrote: > > Module::Install lets you do some limited configuration, but if you're > looking to customize it, you're better off with a custom distribution > package. > A bit off the topic, but I've been using Dist::Zilla to help package Catalyst apps. My favorite features are auto versioning, managing the Changes file, and automatic dependency creation. And, of course, having a custom PluginBundle that centralizes how we build distributions. Perhaps not for everyone, but worth a look. Check out the Dist::Zilla::Plugins on CPAN. But the end result is still just a normal Perl module. All our modules (Catalyst and others) use Dist::Zilla now. I have a Dist::Zilla::Plugin::CatalystFiles that helps build the dist and builds a custom Makefile.PL. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Tue, Mar 5, 2013 at 7:09 AM, John Napiorkowski wrote: > Bill, I think there's general agreement that the plugin is stable enough > and right enough that it should represent core behavior. Although one of > Catalyst's strengths has been to be un-opinionated and leave stuff to > external distributions when possible, I think this does fall under the 'all > good apps should work this way'. I've tentatively pegged it for the March > Catalyst release, since it seems pretty straightforward, at least for now. > If you think you can fit it in (or talk your boss/clients/whatever into > paying for your time) please give a shout out. It would be great to see > more people get to know Catalyst internals a bit, otherwise I don't really > see how we are going to be able to move the platform forward. > The month of May is more likely for me, unfortunately. I was thinking of turning the plugin into a Role and then consuming if the plugin was not already listed or if a config option to disable was not set. That is, on by default but can be disabled. Is that in line with what you were thinking? Ha! I was looking at the code just now and saw a comment that included this URL: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/msg02350.html I guess I've thought about this before > > > > I wonder what percent of Catalyst apps make use of that plugin. > > Not sure, but I think the play-perl ticket has a good way to not break > most people's code > > > -- > Bill Moseley > mose...@hank.org > > > -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Mon, Mar 4, 2013 at 9:24 AM, John Napiorkowski wrote: > http://play-perl.org/quest/5134cdc18e0a96450f38 > > There was a discussion about this on #catalyst-dev and in general people > seem to think its ok. > > Bill, are you up to giving the work a try? I'd be happy to mentor you. I > don't think its killer work, but would really help > I would like to give it a try, yes. I'm just not sure when. I guess there's no rush since there's a plugin. I was just thinking it would be wise to make it a standard part of the framework since it's something that every app probably should to, but easy to ignore or get wrong. I wonder what percent of Catalyst apps make use of that plugin. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Backlog for proposed changes in next Catalyst release
On Fri, Mar 1, 2013 at 9:38 AM, John Napiorkowski wrote: > Hey All, > > > http://jjnapiorkowski.typepad.com/modern-perl/2013/03/catalyst-backup-for-next-release-on-play-perl.html > > This is over on play Perl (http://play-perl.org/player/jnap) for your > comments and votes. Hope to see you there! > I think I've asked this before, but is there any talk of native encoding support -- meaning make Catalyst::Plugin::Unicode::Encoding part of the framework? Having it a plugin makes it appear as optional, but the correct approach is to decoded all text requests and encode all text responses. This is fresh in my mind because last week had problems with two separate encoding issues. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Catalyst::Controller::REST and Serializer/Deserializer
On Sun, Feb 24, 2013 at 11:13 PM, Jean-Marc Choulet wrote: > Hello, > > I'm new to Catalyst and I've certainly a stupid question... I use a > Catalyst::Controller::REST to implement a web service. My question is : How > can I force a application/json content-type response when I ask my ws with > FF or Chrome... ? > Do you mean adding "&content-type=application/json" to your GET request query parameters in the browser? -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] May be asynchronous communication between Catalyst applications
On Sun, Feb 24, 2013 at 8:16 AM, James R. Leu wrote: > I do not have any suggestions about the App1 to App2 hand off > but perhaps my technique for handling async/long running requests > will help. > > I solved my async requirements by using Catalyst::Plugin::Cache > and Catalyst::Plugin::RunAfterRequest. > Long running requests have to run somewhere, of course, and often running in the Catalyst app is the easy solution and may work fine if you are not running a busy site or expecting many concurrent users -- ever. But, in general designing your web app in a way that you expect processes to block for a long time is very risky. You don't want to DoS your own site. It's all about the specifics, but long running processes are often resource intensive, and when they run long users start reloading which users more resources and just compounds the problem. Soon you have an unresponsive site. Maybe you can manage or limit the number of long running web processes in some way, but this is a problem that's already been solved. Francisco gave very good advice. Use a queue to decouple the web app from the long-running jobs. You then have control over your resources -- workers pull in work as they are free instead of stuffing work into web processes. Scaling is then trivial as it's just more workers. It may seem like more work up front to set up but will making things like this much easer -- and safer. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Catalyst::Model::DBIC::Schema and caching row objects
On Tue, Feb 12, 2013 at 10:05 PM, Octavian Rasnita wrote: > ** > *From:* Bill Moseley > > If you want to use DBIC in more apps, don't use Catalyst::Model::... but > create a standalone module that can be used from more apps, even in CLI > scripts. > And access that standalone module using Catalyst::Model::Adaptor. > > To be clear, I do have a separate DBIC schema class that can be, and is, used by CLI scripts. I also think it's a good recommendation to use Catalyst::Model::Adaptor instead. Unfortunately, the two apps sharing the schema are years old and followed best practices at the time and used Catalyst::Model::DBIC::Schema. It would be a big project to change at this point. Can anyone explain the reasoning of this code in Catalyst::Model::DBIC::Schema? Why call compose_namespace($class) here? my $class = $self->_original_class_name; ... $self->composed_schema(*$schema_class->compose_namespace($class*)) unless $is_installed; My question is what would break if instead it was just this? $self->composed_schema( *$schema_class *) unless $is_installed; That would allow row object to use their default class. In case that's not clear, in the first code snippet $class is the name of the *Catalyst model* class. For example "MyApp::Model::MyDB". Using that class in compose_namespace means an "Account" row object ends up as "*MyApp::Model::MyDB::Account*" instead of "*Schema::Result::Account*". Again, that is the problem for me because that's the class name that ends up in Memcached when cached and if want to share those cached row object between Catalyst apps it breaks because "OtherApp" doesn't know how to load "MyApp::Model::MyDB::Account" because that class only exists in MyApp. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Catalyst::Model::DBIC::Schema and caching row objects
Not sure if the is a Catalyst or DBIC question. Catalyst::Model::DBIC::Schema returns objects blessed into classes related to the app. Then the problem is I cannot share (cached) objects across applications. I have a set of DBIC classes "MySchema". I have an often used and expensive query where I cache the returned rows. So, a row object like this: $schema->resultset( 'Account' )->find( 123 ); ends up blessed into "MySchema::Result::Account". When row objects are cached the class name is stored so it can be thawed back into that class, of course. Now, I have an app "MyApp" that uses Catalyst::Model::DBIC::Schema to access "MySchema". In this case row objects are blessed into: MyApp::Model::MyDB::Account. And that's the name that is serialized when freezing. The problem is another app "OtherApp" also uses the DBIC MySchema class and when it fetches a cached entry it tries to bless them into MyApp::Model::MyDB::Account which it cannot do. (Needs to thaw into "OtherApp::Model::MyDB::Account".) Error: Can't locate MyApp/Model/MyDB/Account.pm in @INC Any ideas what might be done here, short of not storing the DBIC row objects? I do want to end up with row objects after thawing, so they can be used just as if I had done the query directly. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Detecting when running under mod_perl
Pre-Plack I had used this to determine if running under Apache/mod_perl.
if ( $c->engine->can( 'apache' ) ) { ... }
Is this the best, future-proof approach now?
During setup check $ENV{MOD_PERL}
and per request check if $c->req->env->{'psgi.input'}
isa 'Apache2::RequestRec'?
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Use of uninitialized value in delete
On Mon, Oct 29, 2012 at 3:45 AM, Will Crawford
wrote:
> On 19 October 2012 18:37, Bill Moseley wrote:
> > delete $c->stash->{foo};
>
> Is there a function / method called "foo" anywhere in scope?
>
I've ignored this for a while, but still seeing in the app's logs.
No, there's no function foo defined.
Another thing that is odd is I have a $SIG{__WARN__} handler and all other
warning are caught by that but not the "Use of uninitialized value in
delete".
This machine is running 5.10.1 -- maybe it's related to that version of
Perl.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] IPv6 client addresses
I'm just starting to look into IPv6 and how it might effect our applications. Perhaps this is a Plack question, but I'm currently using $c->req->address to get at the client address. Our plan is to run dual-stack v4 and v6. Will there be a separate method for the IPv6 address, or will $c->req->address return either v4 or a v6 address? Anyone already in a dual-stack environment? Any other gotchas to consider? I use $c->req->address to limit access -- for example to limit some actions to our local LAN or for customers to limit access to our API via a customer-supplied CIDR. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Does uri_for() URL-escape arguments correctly ?
On Tue, Dec 4, 2012 at 5:22 AM, Marc SCHAEFER wrote: > Hi, > > for some time I write things like this in my templates: > > > I've always used href="[% c.uri_for( ... ) | html %]" -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Catalyst with Twiggy with Pocket.IO (Comet)
On Wed, Nov 28, 2012 at 4:21 AM, Jaro Zajonc wrote: > But if I direct traffic from Apache directly to Twiggy server > I'd bypass Catalyst Authentication/Authorization part for Comet session, > right? > I'd like to allow only authenticated users to subscribe to comet channel. > I am sure I am missing some really simple piece of the puzzle :-\ > Are you over SSL by chance? I've done this by constructing a token on the authenticated server and then have the secondary server that can't fully authenticate validate the token which might be a simple digets of secret + timestamp. That is, the server w/o the auth validates that the token is legitimate and the SSL tells me it came from the client I gave it to. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] log4perl and logging request parameters
I use a log4perl config file, and I have two appenders -- one is for INFO
and above and is a screen appender (web server logs), and the other is for
ERROR and is directed to a separate file.
For *only *WARN and above messages I want to include a dump of request
parameters in the log message. That is, for INFO messages I don't want to
include the parameters.
That's pretty easy for normal exceptions. At the end of a request I look
for $c->error and when found I use Log::Log4perl::MDC->put( params =>
Dumper $c->req->parameters ) to include the params in the log using
%X{params}. (Except, I sanitize the params first -- e.g. replace values
that match /password/).
I'd like to find a more generic way, and something that will work for
calling $c->log->warn and $c->log->error directly, too. (I also trap
__WARN__ and __DIE__ to use log4perl and dump a stack trace).
Again, I don't want the params in all messages, so I don't want to
set Log4perl::MDC->put( params => $dump ) at the start of each request. I
only want to do that for WARN and ERROR messages.
Should I put $c->req->parameters in a global at the start of each request
and then wrap the warn() and error() methods and set the MDC params in
those methods?
Monkey-patch (redefined) the warn() and error() log methods each request
with the current params?
Neither of those sound that great. Is there an approach I'm missing?
Thanks,
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Access Catalyst context object from script
On Wed, Oct 31, 2012 at 6:10 AM, Anthony Gladdish <
anthony.gladd...@newcastle.ac.uk> wrote:
>
> Great - this worked, thanks. I used:
>
> use Catalyst::Test 'MyApp';
> my($res, $c) = ctx_request('/');
>
How does uri_for() know what host:port and script path to use to build
correct URLs to your site?
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Global 'helper' methods
On Tue, Oct 30, 2012 at 7:03 AM, Craig Chant
wrote:
> So it seems it’s OK to whack them in the main MyApp.pm , here is an
> example of what is currently in a ‘MemberGlobs.pm’, which is ‘required’ in
> 90% of all the perl scripts.
>
Be careful with that approach. I'd stick with method that need to get
inserted into the request cycle -- not just a catch-all for something you
don't know where it really belongs.
>
>
> ##
>
> # Yank Date #
>
> ##
>
> sub YankDate {
>
>
>
> #_[0] = UK Date
>
>
>
> # split to parts
>
> my @dte = split(/\//, $_[0]);
>
Where does $_ come from? Now might be a good time to start using
PerlCritic because it's a pain later to try and clean up code.
>
>
> # rebuild to yank date
>
> my $yank = "$dte[2]-$dte[1]-$dte[0]";
>
Where's this date coming from? The database? Look at inflating all dates
and times to DateTime objects.
Then the rendering of that object becomes a View issue.
How do you see the timezone? That's also a rendering/View issue.
It's not pretty, but what I tend to do in the View (say, in a Template) is:
Event Time: [% set_user_time_zone( event.start_time ).strftime( loc(
'_time_with_zone' ) ) %]
set_user_time_zone clones the DateTime object and sets the timezone and
locale on the DateTime objet. The _time_with_zone is a string that gets
"translated" -- although mostly to %c, %x, and %X. But, by using
localization it can be anything.
>
> # return result
>
> $yank;
>
Yes, check out PerlCritic.
> }
>
>
>
> I’m forever having to switch between UK dates and USA dates so have helper
> methods.
>
See above.
>
>
> Where do these go?
>
>
>
> Also where do you put application global constants?
>
Not a clear line, but you have an app config for app-specific config. But,
I also have App::Constants that export constats which is useful for when
you need the constants outside of Catalyst.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Use of uninitialized value in delete
On Sat, Oct 27, 2012 at 9:34 PM, Larry Leszczynski wrote:
> Hi Bill -
>
> On Fri, Oct 19, 2012, at 11:37 AM, Bill Moseley wrote:
>
> > Use of uninitialized value in delete
> >
> > with a line number pointing to this line:
> >
> > delete $c->stash->{foo};
> >
> > I didn't think that delete() issued a warning,
>
> Are both $c and $c->stash defined at that point?
>
Those would be different errors in Perl:
$ perl -wle 'my $c; $c->foo'
Can't call method "foo" on an undefined value at -e line 1.
$ perl -wle 'sub Foo::stash { return {} }; $s = bless {}, "Foo"; delete
$s->stash->{bar}'
$ perl -wle 'sub Foo::stash { undef }; $s = bless {}, "Foo"; delete
$s->stash->{bar}'
Can't use an undefined value as a HASH reference at -e line 1.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Progress bar
On Thu, Oct 25, 2012 at 5:29 AM, Aaron Trevena wrote: > On 24 October 2012 17:59, Bill Moseley wrote: > > PerlBal (as in this old post: > > http://lists.danga.com/pipermail/perlbal/2005-November/000138.html ) > can do > > this as well. > > > > I wonder about the topology. We used to run with Perlbal (and heartbeat > > and IP failover) in front of a pool of web servers. We now run with > > hardware load balancers in front of a pool of web servers. > > > > The load balancer does make it easy to adjust the pool -- as well as > > gracefully handle a web server dropping out of the pool. I don't want > to > > add yet another set of servers for an extra proxy layer. > > > > So, I'm currently thinking of running Nginx on each web server. > (Keep-alive > > between the load balancer and Nginx, and no keep-alive between Nginx and > > Catalyst with maybe Starman.) > > Um... how is adding nginx instead of perlbal not "adding yet another > set of servers"? > I'm talking hardware. I'd rather not add another set of servers between the load balancer and the web pool. It's just more to manage and monitor and one more place to fail. We used to run Perlbal like that when we had less capable load balancers. If I put Nginx/Perlbal on each web server it simplifies the architecture. No need to run redundant Perlbal/Nginx with heartbeat and failover on a separate set of servers in each data center. > What features of nginx are you looking to use vs say perlbal - depends > on how you'd use it and what for, and how easily either would acheive > your goals easily - perlbal *could* have a short/shallower learning > curve, or nginix may be drop-in job that just works without any > customisation or special extensions > Obviously, I'm looking for working progress bar, but also upload buffering so the web app processes are not tied up waiting on client uploads. Of course, scaling extra web processes is pretty easy which is why we haven't worried too much about buffering uploads. Haven't had the need to reproxy. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Progress bar
On Wed, Oct 24, 2012 at 5:44 AM, Lorn wrote: > I'm not following the thread but, did you guys know about > http://wiki.nginx.org/HttpUploadProgressModule. ? > PerlBal (as in this old post: http://lists.danga.com/pipermail/perlbal/2005-November/000138.html ) can do this as well. I wonder about the topology. We used to run with Perlbal (and heartbeat and IP failover) in front of a pool of web servers. We now run with hardware load balancers in front of a pool of web servers. The load balancer does make it easy to adjust the pool -- as well as gracefully handle a web server dropping out of the pool. I don't want to add yet another set of servers for an extra proxy layer. So, I'm currently thinking of running Nginx on each web server. (Keep-alive between the load balancer and Nginx, and no keep-alive between Nginx and Catalyst with maybe Starman.) Anyone see why this might be a bad (or good) approach? -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Progress bar
On Sat, Oct 20, 2012 at 1:51 PM, Tomas Doran wrote: > And UploadProgress is shipped, should be available once it's reindexed > (permissions cock up), which should be shortly :) > So, when running under Starman the uploads are buffered before chunked to Catalyst, which means the progress bar data isn't updated until the upload has completed. This renders the upload progress bar pretty useless with Starman. The progress bar works fine running the app under mod_perl. I suppose using something like Nginx or Perlbal in front of the app would work (because those do cache uploads but also provide a hook for reading upload progress). But, we already have hardware load balancers in front of the app, so don't really need an extra proxy in front of every web server. Any other options? Using a upload/request caching proxy is probably THE correct answer since don't really want to tie up the app with slow uploads. I guess I should test, but I wonder if there's a limit on what Starman will buffer -- I assume it's buffering in memory. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Use of uninitialized value in delete
Sorry for the duplicate if you are on the SF Perl list.
In server logs I'm seeing this warnings:
Use of uninitialized value in delete
with a line number pointing to this line:
delete $c->stash->{foo};
I didn't think that delete() issued a warning, and I can't seem to make it
happen:
$ perl -Mwarnings -Mstrict -Wle 'use warnings; use strict; my $x = {};
delete $x->{a}'
Is there something special about $c->stash that might trigger this in Perl?
Anyone else spot these in the logs.
This is on Perl 5.14.2 / Catalyst 5.90016 / mod_perl 2.0.7
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Progress bar
On Wed, Oct 17, 2012 at 12:34 AM, Tomas Doran wrote: > > Can I get a 'yes this new code works' from someone trying it before we > ship new versions? :) > Yes it works like it used to, now. Thanks. >From what I remember this can't work on Safari and Chrome (Webkit). Need to use the iframe trick on those browsers. https://bugs.webkit.org/show_bug.cgi?id=23933 How close is this version of Catalyst for a release? Unfortunately, I noticed this while preparing for an app release next week. Thanks, -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Progress bar
On Wed, Oct 17, 2012 at 12:34 AM, Tomas Doran wrote: > > > This needs a Catalyst release first - I was waiting till there were a few > more changes for that, but if it's hurting people now, we could do a new > version with just the tiny changes I made to support this. > By the way, I wonder if it wouldn't be cleaner to use _build_request instead of a builder for the args. In this case might have to completely override _build_request to inject in the cache -- or make the cache "rw" so that it can be set after _build_request. Seems like giving the request the app as an attribute would be a bad idea, but not clear what that is. :) I've been struggling a bit with hierarchy of classes where I need to pass child construction params down to child constructors. I've added "parent" attributes to the child classes, manually passed args (as you are doing here), and even had separate config objects that I pass around. Haven't found anything that I'm that fond of. > > Can I get a 'yes this new code works' from someone trying it before we > ship new versions? :) > > Cheers > t0m > > > ___ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Progress bar
On Wed, Oct 17, 2012 at 12:34 AM, Tomas Doran wrote: > > On 17 Oct 2012, at 07:13, Dimitar Petrov wrote: > > > Hey Bill, > > > > I think t0m fixed that a week ago in trunk and it should be working. You > might want to check the trunk of the Catalyst and the Plugin and to see if > it works for you and probably ask t0m to upload new version on CPAN. > > This needs a Catalyst release first - I was waiting till there were a few > more changes for that, but if it's hurting people now, we could do a new > version with just the tiny changes I made to support this. > > Can I get a 'yes this new code works' from someone trying it before we > ship new versions? :) > I'll try and test today. Thanks for addressing this before I even was aware it was a problem. ;) Is $c->prepare_body_chunk every called anymore? I'm wondering if that should be removed so that if someone's app is wrapping that they would see an error. That would identified this issue sooner, perhaps. I was looking at a separate issue where large uploads were stalling. That's when I noticed the progress bar not working. I saw in the logs was this: $ fgrep Read error_log Read error: Read error: Read error: Read error: followed immediately by a restart of the child process (testing with Starman). So, that's something else I need to try and track down. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Progress bar
At one point Catalyst::Engine did this:
# Check for definedness as you could read '0'
while ( defined ( my $buffer = $self->read($c) ) ) {
$c->prepare_body_chunk($buffer);
}
Which meant that Catalyst::Plugin::UploadProgress could wrap/override
$c->prepare_body_chunk.
The code now does this ( in Catalyst::Request ):
while ( defined ( my $buffer = $self->read() ) ) {
$self->prepare_body_chunk($buffer);
}
Which breaks that functionality.
How can I get the upload byte count into $c->cache so that my AJAX request
can get updated upload stats?
Thanks,
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] thoughts on a catalyst roadmap
On Thu, Oct 11, 2012 at 8:04 AM, John Napiorkowski wrote: > > > http://jjnapiorkowski.typepad.com/modern-perl/2012/10/thoughts-on-a-catalyst-roadmap-1.html > I tend to describe Catalyst as glorified dispatcher and object instance manager. The goal there is to make people think about thin controllers and lots of models. Controller bloat tends to happen if not careful. Plugins are big, but looking at our apps we probably use more custom plugins than CPAN ones. Granted, mostly CPAN inspired but end up customizing for our own needs. Obviously, being able to easily hook into the request process is huge. I agree with your thoughts about REST and web sockets. Much (most?) of our development is REST based. I've written two REST replacement systems now (Controller base class and Action class). One is very similar to Catalyst::Action::Rest, but much more heavy weight with CRUD and a bit of magic built in, and another rather light weight based on chaining. Views are becoming less important/interesting since more of that is client-side and much is just returned as JSON. Pain points I have are around scalability. I'd like to have our "app" be made up of many small apps because dev on a very large app can get slow. Having separate apps isn't too hard, although it's a bit more of a pain for rendering templates and sharing a standard layout across separate apps. I've also been looking at ways to manage features -- or really a way to control features dynamically. In other words, all developers work adding features in trunk and always release from trunk. And have a way to dynamically turn features on or off -- or on to just a sub-set of accounts/IP addresses/etc. That's mostly looking for ways to make the build, test, release process less painful -- and less expensive. And releases more frequent and less anxious. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Trapping added errors with the correct caller
On Thu, Oct 11, 2012 at 5:38 AM, Robert Rothenberg wrote:
>
> I would like to trap every error added to $c->error() and log it, noting
> the
> caller (filename, line number) in the logs.
>
> I've not gotten Catalyst::Plugin::ErrorCatcher to work, so I wrote my own
> plugin that overrides $c->error with the following method:
>
> use MRO::Compat;
> use namespace::autoclean;
>
> sub error {
> my ($c, @args) = @_;
>
> foreach my $arg (@args) {
> if ($arg) {
>
> $c->log->error($arg);
> }
> }
>
Isn't that what finalize already does?
I have a very old and out-dated plugin I use that wraps execute() and sets
__DIE__ and __WARN__ to catch those and uses Devel::StackTrace to add in a
stack trace. Then I use log4perl to format and direct the messages.
I also use Moose's Throwable which includes a stack trace.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Better Testing
Anyone using Test::Sweet or Test::Class(::Most) with your large Catalyst apps? Opinions about either -- or lessons learned? Anything else to look at? We have tended to write pretty standard procedural-style tests which are easy to understand across our team (for a few dozen developers) and often end up factoring out common code into utility classes. But, those utility classes are often one-offs and at the whim of the developer how to structure it. I do want something more standardized but easily extensible. I'm finding Test::Class works great for unit tests -- especially when testing related sub-classes -- but we tend to write more feature/integration tests. In other words, tests are longer and depend on previous tests. I guess another term might be workflow tests. I want a framework that enforces good structure, and also that makes it easy to run single tests (that is, run a single tests within a single .t file) to make debugging faster. (e.g. Run the first five tests of 20.) On the other side, and perhaps unrelated to above, but I do want to make sure our tests can be run in parallel so we can run full-test more often and take advantage of our larger multi-core machines. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Modify config in a test
On Wed, Sep 26, 2012 at 1:15 AM, Manni Heumann wrote: > Bill Moseley schrieb am 25.09.2012: > > > The app has a myapp.yml config which includes configuration for > > creating an instance of a Model component -- and part of that Model's > > config is a database dsn attribute. When running my test I do want > > to have the app load this config file -- but I want to modify it > > on-the-fly. > > Why? Why not simply create a myapp_test.yml file and set > CATALYST_CONFIG_LOCAL_SUFFIX to "test" when you run your test-suite? > I guess this is probably the cleanest/easiest. Create my database and write out the config in a BEGIN block -- or somehow otherwise delay loading Catalyst::Test. My other option could be to create my own instance of the Model object and then swap it out in the MyApp->components hash. The fact that my model is dynamically generated at startup based on config adds a bit of complexity to it -- but that can be handled. BTW -- for tests where I need to build a database on-the-fly what I currently do is set my "make test" config use dsn => 'dbi:Pg' and then set the PG* environment variables. But, in this case what I'm doing is swapping my Postgresql config in my configuration with one to use a SQLite database in a temporary file just for a few tests. Thanks for the ideas. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Modify config in a test
I'm using Catalyst::Test on an app. The app has a myapp.yml config which includes configuration for creating an instance of a Model component -- and part of that Model's config is a database dsn attribute. When running my test I do want to have the app load this config file -- but I want to modify it on-the-fly. My test builds a SQLite database when it runs, and I'd like to be able to modify the config as (before) Catalyst creates the instance of the model component -- that is just filter the config to inject my temporary database. The model "dsn" attribute is readonly -- otherwise I could just do MyApp->model( 'MyModel' )->dns( $new_dsn ); What's a good, clean way of doing this? Or perhaps a better approach all together? -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Best-practices question: caching a search
On Sat, Sep 15, 2012 at 7:06 PM, will trillich wrote: > Hi Francisco -- > > I'm not talking about paginating a resultset, I'm talking about returning > to a previously-established resultset on some future HTTP request. Here's > the scenario: > You are asking just how to display multiple pages of search results? > > 12:01 Bob submits search form for "Chicago between 1-Apr-2012 and > 30-Apr-2012" > Use a GET, not a POST. I'm Ignoring timezones, but you should not. http://example.com/event_list?city=Chicago&after=2012-04-01&before=2012-04-30&rows=100&page=1<http://example.com/event_list?city=Chicago&between=2012-04-01:2012-04-30&rows=100&page=1> > > 12:02 Bob sees first page of results, records 1-100 > > 12:03 Bob clicks to see second page of results, records 101-200 > <= how do we re-establish the recordset here, from the original search > form at 12:01? > http://example.com/event_list?city=Chicago&after=2012-04-01&before=2012-04-30&rows=100&page=<http://example.com/event_list?city=Chicago&between=2012-04-01:2012-04-30&rows=100&page=1> 2 > > 12:04 Bob clicks thru to see detail on record 135 > > 12:05 Bob clicks breadcrumbs to "return to search" > <= how do we re-establish the recordset here, from the original search > form at 12:01? > http://example.com/event_list?city=Chicago&after=2012-04-01&before=2012-04-30&rows=100&page=1<http://example.com/event_list?city=Chicago&between=2012-04-01:2012-04-30&rows=100&page=1> All independent requests -- someone may bookmark page 2 and go back there directly. Then (later) think about caching. That depends on usage, your database load, your SLA, your traffic, load, etc. You could fetch the entire result list on the first request and cache, for example, or you could instead cache the entire page (with a separate page cache layer). Use a cache, not the session, for caching. There's nothing here related to the session unless you wan to display something like "recent searches" to show on all pages -- and even that might be better in the cache keyed by user's ID if users are required to log in. > > > > On Sat, Sep 15, 2012 at 2:55 PM, Francisco Obispo wrote: > >> Some databases provide means to return a specific set of records, and >> even an offset, >> >> In DBIx::Class, when you search, you can actually specify the "page" as >> an option [1], >> >> if you're not querying against a database, you might want to use >> something like Memcached or the like to store your resultset and paginate >> accordingly. >> >> >> >> >> [1] >> http://search.cpan.org/dist/DBIx-Class/lib/DBIx/Class/ResultSet.pm#ATTRIBUTES >> >> >> On Sep 15, 2012, at 11:41 AM, will trillich >> wrote: >> >> > User enters some search parameters (location, date-range, etc). Gets >> 500 results which we paginate. Once the user pages to the item of interest, >> he/she can then click thru to edit or see more detail. >> > >> > It'd be nice to have 'breadcrumbs' that take the user back to that page >> of that search. >> > >> > What's the recommended way of doing that? >> > >> > A) stash the whole recordset into the session (can you even >> serialize/deserialize a recordset object?) >> > >> > B) stash the search params and page-no and page-size and recreate the >> recordset each time >> > >> > C) ...something else? >> > >> > >> > >> > -- >> > Will Trillich :: 812.454.6431 >> > >> > “Waiting for perfect is never as smart as making progress.” -- Seth >> Godin >> > ___ >> > List: Catalyst@lists.scsys.co.uk >> > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst >> > Searchable archive: >> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ >> > Dev site: http://dev.catalyst.perl.org/ >> >> Francisco Obispo >> email: fobi...@isc.org >> Phone: +1 650 423 1374 || INOC-DBA *3557* NOC >> PGP KeyID = B38DB1BE >> >> >> ___ >> List: Catalyst@lists.scsys.co.uk >> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst >> Searchable archive: >> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ >> Dev site: http://dev.catalyst.perl.org/ >> > > > > -- > Will Trillich :: 812.454.6431 > > “Waiting for perfect is never as smart as making progress.” -- Seth Godin > > ___ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > > -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Module::Install::Catalyst replacement
I'm using Dist::Zilla and creating a Plugin to replace what
Module::Install::Catalyst does. By default all files in home are copied
to blib (to allow installing) except the following patterns:
our @IGNORE =
qw/Build Build.PL Changes MANIFEST META.yml Makefile.PL Makefile README
_build blib lib script t inc .*\.svn \.git _darcs \.bzr \.hg
debian build-stamp install-stamp configure-stamp/;
I'm thinking of reversing that logic and having a Dist::Zilla plugin
that *explicitly
defines* what is copied. So, any file in home that match these patterns
would be included:
^root$
^Changes$
\.yml$
\.conf$
\.inc$
\.psgi$
Is there anything else that should be in that default list of patterns to
include?
And, of course, allow to add files/dirs to that list via dist.ini
configuration:
; replaces [MakeMaker]
[CatalystFiles]
include = conf
include = config.yml
I'm still not convinced this is the best approach, though. Copying the
files when Makefile.PL runs always seems a bit of a kludge -- compared to,
say, having "make" copy the files like a normal Makefile.
And one could argue that these files and directories should go into "share"
in the distribution and then get installed as File::ShareDir expects (i.e.
have Catalyst::Utils::home() use dist_dir( $c->config->{name} ) ). But,
that's kind of a big change.
Anyone have thoughts on the that?
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Applying roles that contain actions.
On Fri, Sep 7, 2012 at 11:50 AM, Tomas Doran wrote:
>
> On 5 Sep 2012, at 21:48, Bill Moseley wrote:
> >
> > Currently I just manually apply the role directly in the existing
> controller, so not a huge issue, but would be handy if I could just add a
> role to the base class and pull in all the components
>
> What's stopping you doing this?
>
Sorry, I wasn't very clear in that sentence. It's a bit more involved
than this, but I'm looking at applying a controller role (one with actions)
dynamically. So, the code ends up something like:
package MyApp;
after setup_components => sub {
my $self = shift;
my $controller $self->controller( 'Foo' );
use My::Role::With::Actions;
My::Role::With::Actions->meta->apply( $controller );
Results in:
Composing MooseX::MethodAttributes::Role::Meta::Role::Application onto
instances is unsupported
That makes sense to me -- for one thing Catalyst would not know about the
actions in the role.
--
Bill Moseley
mose...@hank.org
___
List: Catalyst@lists.scsys.co.uk
Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst
Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/
Dev site: http://dev.catalyst.perl.org/
[Catalyst] Applying roles that contain actions.
I've been injecting models into Catalyst apps (either using CatalystX::InjectComponent or just stuffing an instance into $app->components hash). I'd also like to be able to apply roles to existing controllers. If I try to ->apply the role it fails if using MooseX::MethodAttributes::Role. Which I assume is because the actions would not be registered with Catalyst. Is there a way to inject actions in this way? Currently I just manually apply the role directly in the existing controller, so not a huge issue, but would be handy if I could just add a role to the base class and pull in all the components -- or dynamically add the actions. In general, what I'm after is a way to specify "with App::Feature::Widget;" in my app base class and bring in an entire feature. I'm also looking at a "gatekeeper" type of feature control that Facebook is known for: https://www.facebook.com/video/video.php?v=10100259101684977&oid=9445547199&comments -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] [JOB] Oakland, CA, USA
Catalyst web app developer job posting: http://jobs.perl.org/job/16350 -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] working around request timeouts
On Mon, Aug 20, 2012 at 12:10 PM, Sergey Dmitriev < sergey.program...@gmail.com> wrote: > Looks like what you need is the job queue. Take a look at > > Queue::DBI (https://metacpan.org/module/Queue::DBI) - simpler > TheSchwartz (https://metacpan.org/module/TheSchwartz) > Yes, and what about long polling? Would that solve the ajax request timing out? That is, have your client code get sent a message when the backend is done instead of waiting. > > Sergey > > 2012/8/20 James R. Leu > >> Hello all, >> >> Problem description >> === >> I have a catalyst application server that responds >> to 'API' requests for web applications via XHTMLRequests. >> Sometimes the requests are timing out due to the backend >> database queries taking too long. I'm looking for ways to >> work around this and prevent the 'API' requests from >> timing out. >> >> I know some of the possible resolutions to this are >> - fix the queries >> - fix the database >> - frontend the RDBMS with NoSQL >> >> I'm working towards those fixes, but they are long >> term projects, I'm looking for an interim solution. >> That would notify the web application that it will >> need to come back later for the response (ie decouple >> request handling from the actual request/response). >> >> My attempt >> == >> In my handler I fork a child process. >> >> In the parent I send a response with a >> 'job id' so the web application knows >> to poll the 'API' for completion. >> >> In the child I close the IO socket so it cannot send >> a response and then let it finish processing the >> request, but it looks like I've lost my database >> connections so the request fails. >> >> My wish >> === >> What I would like to do is avoid the fork and instead >> have the handler send an early response to the >> web application and then finish processing the request >> and not try to send a response when done. >> >> Is there a common term for what I'm trying to do >> like continuation or something like that? >> Has anyone already done this? >> >> Thank you for your time. >> -- >> James R. Leu >> j...@mindspring.com >> >> ___ >> List: Catalyst@lists.scsys.co.uk >> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst >> Searchable archive: >> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ >> Dev site: http://dev.catalyst.perl.org/ >> >> > > ___ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > > -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] [ANNOUNCE] Catalyst-Runtime 5.90016
Thanks for the new release!I'll start testing with it today. I created https://rt.cpan.org/Ticket/Display.html?id=79043 to address the Catalyst::Test issue I posted about yesterday, although that might be a duplicate of https://rt.cpan.org/Ticket/Display.html?id=62015 Does the latest release fix this: https://rt.cpan.org/Ticket/Display.html?id=78090 And are those 2, 3, and even 4 year old tickets relevant any more? On Thu, Aug 16, 2012 at 6:46 PM, John Napiorkowski wrote: > > > The latest point release of your favorite web development platform is out > on CPAN! > > https://metacpan.org/release/JJNAPIORK/Catalyst-Runtime-5.90016 > > This is a bugfix release that makes a few tweaks to the way parameters are > prepared and makes some Catalyst methods a bit smarter when stringifying > objects. > > Here's the full changes: > > 5.90016 - 2012-08-16 15:35:00 > - prepare_parameters is no longer an attribute builder. It is now a > method > that calls the correct underlying functionality (Bill Moseley++) > - Updated Makefile.PL to handle MacOXS tar > - Fix uri_for to handle a stringifiable object > - Fix model/view/controller methods to handle stringifiable objects > - Fix RT#78377 - IIS7 ignores response body for 3xx requests, which > causes (a different) response to be broken when using keepalive. > Fixed by applying Middleware which removes the response body and > content length that Catalyst supplies with redirects. > > We anticipate no issues with this minor release, but if you are doing > weird stuff with objects and stringification, or liked to mess around with > Catalyst internal functions related to parameter, its not impossible you'd > have trouble. Please shout out if so. > > Thanks! > > John > > > ___ > List: Catalyst@lists.scsys.co.uk > Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst > Searchable archive: > http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ > Dev site: http://dev.catalyst.perl.org/ > -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
[Catalyst] Parsing of undecoded UTF-32 at /home/davidw/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Catalyst/Test.pm
http://cpansearch.perl.org/src/BOBTFISH/Catalyst-Runtime-5.90015/lib/Catalyst/Test.pm In the "mangle_response" sub the code does this: require HTML::HeadParser; my $parser = HTML::HeadParser->new(); $parser->xml_mode(1) if $resp->content_is_xhtml; $parser->utf8_mode(1) if $] >= 5.008 && $HTML::Parser::VERSION >= 3.40; $parser->parse( $resp->content ); Shouldn't that code check that the content-type is HTML before attempting to parse? For example, this response was causing problems: Cache-Control: no-store, no-cache, must-revalidate Content-Length: 24043 Content-Type: audio/mp4a-latm Expires: Wed, 31 Dec 1969 23:59:59 GMT Content-Disposition: attachment; filename*=UTF-8''trumpet.m4a (Warnings are fatal): Parsing of undecoded UTF-32 at /home/moseley/perl5/perlbrew/perls/perl-5.14.2/lib/site_perl/5.14.2/Catalyst/Test.pm line 314. -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
Re: [Catalyst] Change in $c->engine->prepare_parameters
Did this help? On Sat, Aug 4, 2012 at 6:19 PM, Bill Moseley wrote: > See attached. > > Let me know if you want something else. > > I'm mixed on of the clear should be private. I made it that way, but I > wonder if there could be a use case to reset the parameters so next time > $c->req->parameters is called they get built again. I guess in that case > why wouldn't they just call $c->prepare_parameters again? > > On Sat, Aug 4, 2012 at 6:53 AM, Tomas Doran wrote: > >> >> On 3 Aug 2012, at 14:39, Bill Moseley wrote: >> >> > What it doesn't do any more is set $c->req->parameters when called >> directly, only when called as a builder. >> > >> > >> > I don't think prepare_parameters should be both a builder AND a method. >> What about this approach: >> >> +1, except I think the clearer should be _clear_parameters - we don't >> want to add a new public method for this if it can be avoided. >> >> Fancy making a 'proper' patch someone can apply which does these changes >> and updates the relevant doc? >> >> Cheers >> t0m >> >> >> ___ >> List: Catalyst@lists.scsys.co.uk >> Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst >> Searchable archive: >> http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ >> Dev site: http://dev.catalyst.perl.org/ >> > > > > -- > Bill Moseley > mose...@hank.org > -- Bill Moseley mose...@hank.org ___ List: Catalyst@lists.scsys.co.uk Listinfo: http://lists.scsys.co.uk/cgi-bin/mailman/listinfo/catalyst Searchable archive: http://www.mail-archive.com/catalyst@lists.scsys.co.uk/ Dev site: http://dev.catalyst.perl.org/
