Re: Perl and bad characters - Solved!

[Bill Stephenson]

I found the simple “append” solution I was looking for on stackoverflow.com:

https://stackoverflow.com/questions/21718486/need-to-add-new-data-to-json-array-in-perl
 


Thank you all again for the help!

I’ve included the working script below. If anyone has suggestions for 
improvements please let me know.

Kindest Regards,

Bill Stephenson


—
#!/usr/bin/perl

use strict;
use warnings;
use JSON::XS;
use Data::Dumper;
use CGI;
use LWP::Simple;

my $cgi = CGI->new;
my $msg;

# Get and print the post doc ID:
my $id = $cgi->param('postID');
# $msg = "id:  $id";
# _message($msg);  

# Get the blog post from CouchDB
my $url = "https://user:pass\@cherrypc.com:6984/cherrypc/$id;;

my $blogdoc = get $url;
die "Couldn't get $url" unless defined $blogdoc;

# convert json to perl object
my $data_structure = decode_json($blogdoc);
 
# $msg = "Doc Title: ". $data_structure->{'title'};
# _message;

# $msg = "data_structure: \n" . Dumper($data_structure);
# _message($msg);

# -
#Append the list of comments:

my $newCommentDate = $cgi->param('commentDate'); 
my $newComment = $cgi->param('comment');  

my $newdata = {commentDate=>"$newCommentDate",comment=>"$newComment"};
push @{ $data_structure->{'comments'}  }, $newdata;
# -

# convert perl object back to json 
my $updatedDoc = encode_json $data_structure;

# Update the document
my $req = HTTP::Request->new(PUT => $url);
$req->content_type('application/json');
$req->content($updatedDoc);

my $ua = LWP::UserAgent->new; 
my $res = $ua->request($req);
# $res is an HTTP::Response.

if ($res->is_success) {
$msg = "Success: ". $res->as_string;
_message($msg);
  }
  else {
$msg =  "Failed: ". $res->status_line;
_message($msg);
  }

# $msg = Dumper($res);
# _message($msg);

print $cgi->header('text/plain;charset=UTF-8');  
  
exit;

#
sub print_message {
#
open (DUMPFILE, ">>/usr/lib/cgi-bin/debug.txt") or die "Unable to open 
/usr/lib/cgi-bin/debug.txt \n";
print DUMPFILE "\n--- message --- \n";
print DUMPFILE "$msg";
print DUMPFILE "\n- end - \n";
close(DUMPFILE);
$msg = "";
return;
}

Re: Perl and bad characters - Progress!

[Bill Stephenson]

As per the advice given I took a different route and used LWP to  get and put 
the document. 

And I’m going the ditch the building of the JSON like so:

my $returnJSON = qq`{"_id": "$_id", "_rev": "$_rev", "title": "$title", 
"subtitle": "$subtitle", "content": "$content", "docType": "text", 
"Text_publish": "yes", "publishDate": "$publishDate",$newCommentsList}`;

That just doesn’t work.  I don’t know why, but all I really need to do is add 
the comment to the Perl object so I’m going to work on that. Perl Objects do 
baffle me a bit but I’m going to try and trudge my way through figuring it out. 

Thank you all for the advice and pointers.

Below is what I’ve got so far and this does update the doc successfully so I 
have made progress with your help:
--

#!/usr/bin/perl

use strict;
use warnings;
use JSON::XS;
use Data::Dumper;
use CGI;
use LWP::Simple;

my $cgi = CGI->new;
my $msg;

# Get and print the post doc ID:
my $id = $cgi->param('postID');
# $msg = "id:  $id";
# _message($msg);  

# Get the blog post from CouchDB
my $url = "https://user:pass\@cherrypc.com:6984/cherrypc/$id;;

my $blogdoc = get $url;
die "Couldn't get $url" unless defined $blogdoc;

# convert json to perl object
my $data_structure = decode_json($blogdoc);

$msg = "Doc Title: ". $data_structure->{'title'};
_message;

# -
#Get the list of comments:

my %comments;
my $comments;
my @comments;
my $comment;
my $commentDate;
my $msgCounter = 1;
my $commentList;

$comments{ $_->{'comment'} } = $_->{'commentDate'} for @{ 
$data_structure->{'comments'} };

$msg = "Number of Comments: " . scalar keys %comments;
_message;

# Get and print old comments:
while ( ( $comment, $commentDate ) = each %comments ) { 
$commentList .= "Comment: $comment :: Comment Date : $commentDate \n";
}

$msg = "Old Comments: \n" .$commentList; 
_message;

# Need to add the new comment to the $data_structure perl object here
my $newCommentDate = $cgi->param('commentDate'); 
my $newComment = $cgi->param('comment');  

# Print new comment:
$msg = "New Comment: \nComment $newComment :: Comment Date : $newCommentDate";
_message($msg);

# I have no idea yet how to append the new comment to the Perl Object.
# Seems like this should be easy but I've yet to figure that out.

# -
# convert perl object back to json 
my $updatedDoc = encode_json $data_structure;
# -

# Update the document
my $req = HTTP::Request->new(PUT => $url);
$req->content_type('application/json');
$req->content($updatedDoc);

my $ua = LWP::UserAgent->new; 
my $res = $ua->request($req);
# $res is an HTTP::Response.

if ($res->is_success) {
$msg = "Success: ". $res->as_string;
_message($msg);
  }
  else {
$msg =  "Failed: ". $res->status_line;
_message($msg);
  }

# $msg = Dumper($res);
# _message($msg);

print $cgi->header('text/plain;charset=UTF-8');  
  
exit;

#
sub print_message {
#
open (DUMPFILE, ">>/usr/lib/cgi-bin/debug.txt") or die "Unable to open 
/usr/lib/cgi-bin/debug.txt \n";
print DUMPFILE "\n--- message --- \n";
print DUMPFILE "$msg";
print DUMPFILE "\n- end - \n";
close(DUMPFILE);
$msg = "";
return;
}

Re: Perl and bad characters

[Dave Cottlehuber]

On Sat, 7 Apr 2018, at 07:24, Bill Stephenson wrote:
> I’ve been working on a “comments” feature for my “CherryPC blog”.
>
> I don’t want readers to have to make a user account to comment so I’m
> wanting to use a perl script on the server side that has the user
> credentials in the $url variable below.
>
> This is the code I’m using to update the document with the comment.
>
> # Convert the JSON to a perl object
>
> my $data_structure = decode_json(`curl -X GET $url`);

Hi Bill,

I think all the responses so far have given you a piece of the puzzle, but not 
the whole. Hopefully this completes the picture.

TLDR just use Store::CouchDB,  and all of the conversion will "just happen" 
directly

https://metacpan.org/release/Store-CouchDB

[snip]

given this JSON object as an example of what gets converted to/from perl 
objects:

GET /testy/data HTTP/1.1
Host: localhost:5984
Accept: application/json
Accept-Encoding: gzip, deflate
User-Agent: bat/0.1.0

HTTP/1.1 200 OK
Content-Length : 111
Cache-Control : must-revalidate
Server : CouchDB/1.7.1 (Erlang OTP/19)
Etag : "1-75710f70ac64e43a12fbc76ca3305852"
Date : Sat, 07 Apr 2018 21:58:56 GMT
Content-Type : application/json

{
  "_id": "data",
  "_rev": "1-75710f70ac64e43a12fbc76ca3305852",
  "object": {
"nested": true
  },
  "array": [
1,
"2",
false,
null
  ]
}

The JSON should be self-explanatory but its useful to see how the same data is 
represented in perl.

So here's a very short perl program to show you how much easier this is using 
Store::CouchDB and the automatic perl<->JSON conversions.

#!/usr/bin/env perl

use Modern::Perl;
use Store::CouchDB;
use Data::Printer caller_info => 1;

# connect to our couch and existing db
my $c = Store::CouchDB->new({host => 'localhost', db => 'testy'});

# fetch a JSON doc and auto-convert it to a perl object
my $doc = $c->get_doc('data');

# dump out the perl object we got back
# it's not so scary
p $doc;

# look up a simple plain text value
say $doc->{'_id'};

# nested objects can be traversed by name
# true in JSON is mapped to 1 in perl (truthy)
# and will round-trip correctly
say $doc->{'object'}->{'nested'};
exit 0;


The output follows; in your terminal it will be in glorious colour:

 ./relax.pl 
Printing in line 15 of ./relax.pl:
\ {
_id  "data",
_rev "1-75710f70ac64e43a12fbc76ca3305852",
array[
[0] 1,
[1] 2,
[2] JSON::PP::Boolean  {
Parents   Types::Serialiser::BooleanBase
public methods (0)
private methods (0)
internals: 0
},
[3] undef
],
object   {
nested   JSON::PP::Boolean  {
Parents   Types::Serialiser::BooleanBase
public methods (0)
private methods (0)
internals: 1
}
}
}
data
1

note how the doc id "data" is a simple string, and the JSON::PP::Boolean 
object, and undef, is used to ensure that the JSON true, false, and null values 
respectively have an equivalent definition in perl that can be safely 
round-tripped back to JSON.

> From what I understand "use utf8” forces the all data to be utf-8...

I highly recommend getting chromatic's book 
https://pragprog.com/book/swperl/modern-perl-fourth-edition and reading it 
immediately. There's a free online version 
http://www.onyxneon.com/books/modern_perl/index.html but it's well worth the 
pennies. A few hours reading this will power up your perl skills enormously I 
think.

> If anyone has any ideas and/or advice on how to deal with this I’d sure
> appreciate them. I’ve pretty much ran out of them at this point.

TLDR use a library and then you can skip all the pain of shelling out to 
external programs. Also this is a very risky strategy - sanitising user input 
that is passed to/from curl and so forth is dangerous, especially on the 
internet. The same goes for date/time handling, look on metacpan.org and find a 
library to do the heavy lifting for you.

Finally, http://mojolicious.org/ is a very good modern perl web framework. In a 
future incarnation of your site/app it may be very helpful, but I suspect 
you've already done enough not to want to start from scratch.

Good luck!

A+
Dave

Re: Perl and bad characters

[Raimund Riedel]

The internal representation of character strings in Perl-5 is not 
identical to UTF-8 or UTF-X, although they both may occur in the same 
string variable. There is no automatic conversion; the "use utf8;" 
pragma is only to enable Perl-5 source code written in UTF-8 (see 
"perldoc utf8"). Therefore each UTF-8 text coming from outside the 
program must be decoded, as well as all data to leave the program as 
UTF-8 text must be encoded.


So please after including the "use Encode;" pragma replace your line
    my $data_structure = decode_json(`curl -X GET $url`);
by something like
    my $data_structure = decode_json(decode('utf8', `curl -X GET $url`));
and replace analogously
    my $Post = `curl -X PUT $url -d '$returnJSON'`;
by
    my $JSONutf8 = encode('utf8', $returnJSON);
    my $Post = `curl -X PUT $url -d '$JSONutf8'`;

This method helped my a lot to build and use a couch database with many 
international names in its texts. Since the error message you included 
is related to UTF-8, it should be worth while to try in your case.


Kind regards,
Raimund Riedel


Am 07.04.2018 um 07:24 schrieb Bill Stephenson:

I’ve been working on a “comments” feature for my “CherryPC blog”.

I don’t want readers to have to make a user account to comment so I’m wanting 
to use a perl script on the server side that has the user credentials in the 
$url variable below.

This is the code I’m using to update the document with the comment.

# Convert the JSON to a perl object

my $data_structure = decode_json(`curl -X GET $url`);

my $_id = $data_structure->{'_id'};
my $_rev = $data_structure->{'_rev'};
my $title = $data_structure->{'title'};
my $subtitle = $data_structure->{'subtitle'};
my $content = $data_structure->{'content'};
my $Text_publish = $data_structure->{'Text_publish'};
my $publishDate = $data_structure->{'publishDate'};


my $returnJSON = qq`{"$_id": "_id", "_rev": "$_rev", "title": "$title", "subtitle": "$subtitle", "content": "$content", 
"docType": "text", "Text_publish": "yes", "publishDate": "$publishDate",$newCommentsList}`;

my $Post = `curl -X PUT $url -d '$returnJSON'`;

This works fine with plain text, but the blog posts are made with TinyMCE and 
use HTML.  I can update them fine with Javascript and PouchDB, but Perl is 
dying on double quotes, single quotes, and backslashes:

‘ “ \

I’ve narrowed it down to just those 3 characters. If I strip those from the 
html and comments it will all post fine, but html doesn’t work without those so 
that’s not an option.

I’m using these modules:

use strict;
use warnings;
use utf8;
use JSON::XS;
use Data::Dumper;
use CGI;

 From what I understand "use utf8” forces the all data to be utf-8 encoded and 
I’ve used several different modules to encode the data and built the entire document 
in a perl object and converted that to JSON as opposed to a simple string like 
above, but it still dies on those three characters.

This is what the curl error tells me:

PUT Error: bad_request
reason: invalid UTF-8 JSON

So, it’s those 3 characters that are not being encoded correctly.

If anyone has any ideas and/or advice on how to deal with this I’d sure 
appreciate them. I’ve pretty much ran out of them at this point.

Kindest Regards,

Bill Stephenson






--
Raimund Riedel
__ rajmun...@gmail.com
__ Mi parolas Esperanton

Re: Perl and bad characters

[Michael Zedeler]

Hi Bill.

You need to escape the characters and unescape them again when you 
retrieve them from the server. What you should do is something like this:


use JSON;
use HTTP::Request::Common;

my $result = PUT $url, Content_Tipe => 'application/json', Content => 
encode_json($data_structure);


My Perl knowledge is a little rusty, but something along the lines above 
should work. If you use curl externally (with backticks as in your 
example), you're likely to run into a lot of escaping problems that 
really aren't worth working around.


Regards,

Michael.

On 07-04-2018 07:24, Bill Stephenson wrote:

I’ve been working on a “comments” feature for my “CherryPC blog”.

I don’t want readers to have to make a user account to comment so I’m wanting 
to use a perl script on the server side that has the user credentials in the 
$url variable below.

This is the code I’m using to update the document with the comment.

# Convert the JSON to a perl object

my $data_structure = decode_json(`curl -X GET $url`);

my $_id = $data_structure->{'_id'};
my $_rev = $data_structure->{'_rev'};
my $title = $data_structure->{'title'};
my $subtitle = $data_structure->{'subtitle'};
my $content = $data_structure->{'content'};
my $Text_publish = $data_structure->{'Text_publish'};
my $publishDate = $data_structure->{'publishDate'};


my $returnJSON = qq`{"$_id": "_id", "_rev": "$_rev", "title": "$title", "subtitle": "$subtitle", "content": "$content", 
"docType": "text", "Text_publish": "yes", "publishDate": "$publishDate",$newCommentsList}`;

my $Post = `curl -X PUT $url -d '$returnJSON'`;

This works fine with plain text, but the blog posts are made with TinyMCE and 
use HTML.  I can update them fine with Javascript and PouchDB, but Perl is 
dying on double quotes, single quotes, and backslashes:

‘ “ \

I’ve narrowed it down to just those 3 characters. If I strip those from the 
html and comments it will all post fine, but html doesn’t work without those so 
that’s not an option.

I’m using these modules:

use strict;
use warnings;
use utf8;
use JSON::XS;
use Data::Dumper;
use CGI;

 From what I understand "use utf8” forces the all data to be utf-8 encoded and 
I’ve used several different modules to encode the data and built the entire document 
in a perl object and converted that to JSON as opposed to a simple string like 
above, but it still dies on those three characters.

This is what the curl error tells me:

PUT Error: bad_request
reason: invalid UTF-8 JSON

So, it’s those 3 characters that are not being encoded correctly.

If anyone has any ideas and/or advice on how to deal with this I’d sure 
appreciate them. I’ve pretty much ran out of them at this point.

Kindest Regards,

Bill Stephenson





--
Michael Zedeler
70 25 19 99
mich...@zedeler.dk

dk.linkedin.com/in/mzedeler | twitter.com/mzedeler | github.com/mzedeler

RE: Perl and bad characters

[Keith Gable]

The last line is you taking user input and putting it into a shell argument 
without escaping. Use a real HTTP library so that you don’t get exploited by a 
robot. Does this still persist if you use a real HTTP library?




From: Bill Stephenson <b...@cherrypc.com>
Sent: Friday, April 6, 2018 10:24:50 PM
To: user@couchdb.apache.org
Subject: Perl and bad characters

I’ve been working on a “comments” feature for my “CherryPC blog”.

I don’t want readers to have to make a user account to comment so I’m wanting 
to use a perl script on the server side that has the user credentials in the 
$url variable below.

This is the code I’m using to update the document with the comment.

# Convert the JSON to a perl object

my $data_structure = decode_json(`curl -X GET $url`);

my $_id = $data_structure->{'_id'};
my $_rev = $data_structure->{'_rev'};
my $title = $data_structure->{'title'};
my $subtitle = $data_structure->{'subtitle'};
my $content = $data_structure->{'content'};
my $Text_publish = $data_structure->{'Text_publish'};
my $publishDate = $data_structure->{'publishDate'};


my $returnJSON = qq`{"$_id": "_id", "_rev": "$_rev", "title": "$title", 
"subtitle": "$subtitle", "content": "$content", "docType": "text", 
"Text_publish": "yes", "publishDate": "$publishDate",$newCommentsList}`;

my $Post = `curl -X PUT $url -d '$returnJSON'`;

This works fine with plain text, but the blog posts are made with TinyMCE and 
use HTML.  I can update them fine with Javascript and PouchDB, but Perl is 
dying on double quotes, single quotes, and backslashes:

‘ “ \

I’ve narrowed it down to just those 3 characters. If I strip those from the 
html and comments it will all post fine, but html doesn’t work without those so 
that’s not an option.

I’m using these modules:

use strict;
use warnings;
use utf8;
use JSON::XS;
use Data::Dumper;
use CGI;

>From what I understand "use utf8” forces the all data to be utf-8 encoded and 
>I’ve used several different modules to encode the data and built the entire 
>document in a perl object and converted that to JSON as opposed to a simple 
>string like above, but it still dies on those three characters.

This is what the curl error tells me:

PUT Error: bad_request
reason: invalid UTF-8 JSON

So, it’s those 3 characters that are not being encoded correctly.

If anyone has any ideas and/or advice on how to deal with this I’d sure 
appreciate them. I’ve pretty much ran out of them at this point.

Kindest Regards,

Bill Stephenson

Perl and bad characters

[Bill Stephenson]

I’ve been working on a “comments” feature for my “CherryPC blog”. 

I don’t want readers to have to make a user account to comment so I’m wanting 
to use a perl script on the server side that has the user credentials in the 
$url variable below.

This is the code I’m using to update the document with the comment.

# Convert the JSON to a perl object

my $data_structure = decode_json(`curl -X GET $url`);

my $_id = $data_structure->{'_id'};
my $_rev = $data_structure->{'_rev'};
my $title = $data_structure->{'title'};
my $subtitle = $data_structure->{'subtitle'};
my $content = $data_structure->{'content'};
my $Text_publish = $data_structure->{'Text_publish'};
my $publishDate = $data_structure->{'publishDate'};


my $returnJSON = qq`{"$_id": "_id", "_rev": "$_rev", "title": "$title", 
"subtitle": "$subtitle", "content": "$content", "docType": "text", 
"Text_publish": "yes", "publishDate": "$publishDate",$newCommentsList}`;

my $Post = `curl -X PUT $url -d '$returnJSON'`;

This works fine with plain text, but the blog posts are made with TinyMCE and 
use HTML.  I can update them fine with Javascript and PouchDB, but Perl is 
dying on double quotes, single quotes, and backslashes:

‘ “ \

I’ve narrowed it down to just those 3 characters. If I strip those from the 
html and comments it will all post fine, but html doesn’t work without those so 
that’s not an option.

I’m using these modules:

use strict;
use warnings;
use utf8;
use JSON::XS;
use Data::Dumper;
use CGI;

From what I understand "use utf8” forces the all data to be utf-8 encoded and 
I’ve used several different modules to encode the data and built the entire 
document in a perl object and converted that to JSON as opposed to a simple 
string like above, but it still dies on those three characters. 

This is what the curl error tells me:

PUT Error: bad_request
reason: invalid UTF-8 JSON

So, it’s those 3 characters that are not being encoded correctly.

If anyone has any ideas and/or advice on how to deal with this I’d sure 
appreciate them. I’ve pretty much ran out of them at this point.

Kindest Regards,

Bill Stephenson