Re: AWS API Dlang, hmac sha256 function.

[Marc Schütz via Digitalmars-d-learn]

On Sunday, 11 October 2015 at 23:16:51 UTC, holo wrote:

auto hmac_sha256(ubyte[] key, ubyte[] msg)
{
auto hmac = hmac!SHA256(key);
hmac.put(msg);
auto digest = hmac.finish;

return digest;
}

alias sign = hmac_sha256;


std.digest.hmac already contains a helper function for this:

hmac!SHA256(data, secret);

The order of arguments is this way to allow chaining with UFCS:

data.hmac!SHA256(secret);

But it seems the documentation generator can't handle ddoc 
comments in templates, so unfortunately it doesn't appear in the 
documentation.

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

Thank you for info, i changed my code to use that build in 
template and changed "cast(ubyte[]"  to "xxx.representation" and 
it is still working:


#!/usr/bin/rdmd -L-lcurl

module sigawsv4;

import std.stdio, std.process;
import std.digest.sha, std.digest.hmac;
import std.string;
import std.conv;
import std.datetime;
import std.net.curl;

void main()
{
auto accessKey = environment["AWS_ACCESS_KEY"];
auto secretKey = environment["AWS_SECRET_KEY"];

auto currentClock = Clock.currTime(UTC());

auto currentDate = cast(Date)currentClock;
auto curDateStr = currentDate.toISOString;

auto currentTime = cast(TimeOfDay)currentClock;
auto curTimeStr = currentTime.toISOString;

auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";


string method = "GET";
string service = "ec2";
string host = "ec2.amazonaws.com";
string region = "us-east-1";
string endpoint = "https://ec2.amazonaws.com;;
	string request_parameters = 
"Action=DescribeInstances=2013-10-15";



alias sign = hmac!SHA256;

	auto getSignatureKey(string key, string dateStamp, string 
regionName, string serviceName)

{
auto kString = ("AWS4" ~ key).representation;
auto kDate = sign(dateStamp.representation, kString);
auto kRegion = sign(regionName.representation, kDate);
auto kService = sign(serviceName.representation, kRegion);
auto kSigning = sign("aws4_request".representation, kService);

return kSigning;
}


string canonicalURI = "/";
string canonicalQueryString = request_parameters;
	string canonicalHeadersString =  "host:" ~ host ~ "\n" ~ 
"x-amz-date:" ~ xamztime ~ "\n";

string signedHeaders = "host;x-amz-date";
string payloadHash = sha256Of("").toHexString.toLower;
	string canonicalRequest = method ~ "\n" ~ canonicalURI ~ "\n" ~ 
canonicalQueryString ~ "\n" ~ canonicalHeadersString ~ "\n" ~ 
signedHeaders ~ "\n" ~ payloadHash;



string algorithm = "AWS4-HMAC-SHA256";
	string credentialScope = curDateStr ~ "/" ~ region ~ "/" ~ 
service ~ "/" ~ "aws4_request";
	string stringToSign = algorithm ~ "\n" ~ xamztime ~ "\n" ~ 
credentialScope ~ "\n" ~ 
sha256Of(canonicalRequest).toHexString.toLower;


	auto signingKey = getSignatureKey(secretKey, curDateStr, region, 
service);


	string signature = hmac!SHA256(stringToSign.representation, 
signingKey).toHexString.toLower;


	string authorizationHeader = algorithm ~ " " ~ "Credential=" ~ 
accessKey ~ "/" ~ credentialScope ~ ", " ~ "SignedHeaders=" ~ 
signedHeaders ~ ", " ~ "Signature=" ~ signature;


auto client = HTTP(endpoint ~ "?" ~ canonicalQueryString);
client.method = HTTP.Method.get;
client.addRequestHeader("x-amz-date", xamztime);
client.addRequestHeader("Authorization", authorizationHeader);
auto content = client.perform();

writeln(content);
}

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

After long fight with previous code i try to rewrite "one-to-one" 
python example from 
http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html (GET part) from begging to D with full success. Here is working code in clear D. Im using hmac function which is available from 2.069 of phobos). It is not too beautiful but like i said im beginner, here it is:



#!/usr/bin/rdmd -L-lcurl

module sigawsv4;

import std.stdio, std.process;
import std.digest.sha, std.digest.hmac;
import std.string;
import std.conv;
import std.datetime;
import std.net.curl;

void main()
{
auto accessKey = environment["AWS_ACCESS_KEY"];
auto secretKey = environment["AWS_SECRET_KEY"];

auto currentClock = Clock.currTime(UTC());

auto currentDate = cast(Date)currentClock;
auto curDateStr = currentDate.toISOString;

auto currentTime = cast(TimeOfDay)currentClock;
auto curTimeStr = currentTime.toISOString;

auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";


string method = "GET";
string service = "ec2";
string host = "ec2.amazonaws.com";
string region = "us-east-1";
string endpoint = "https://ec2.amazonaws.com;;
	string request_parameters = 
"Action=DescribeInstances=2013-10-15";



auto hmac_sha256(ubyte[] key, ubyte[] msg)
{
auto hmac = hmac!SHA256(key);
hmac.put(msg);
auto digest = hmac.finish;

return digest;
}

alias sign = hmac_sha256;

	auto getSignatureKey(string key, string dateStamp, string 
regionName, string serviceName)

{
ubyte[] kString = cast(ubyte[])("AWS4" ~ key);
auto kDate = sign(kString, cast(ubyte[])dateStamp);
auto kRegion = sign(kDate, cast(ubyte[])regionName);
auto kService = sign(kRegion,  cast(ubyte[])serviceName);
auto kSigning = sign(kService, cast(ubyte[])"aws4_request");

return kSigning;
}


string canonicalURI = "/";
string canonicalQueryString = request_parameters;
	string canonicalHeadersString =  "host:" ~ host ~ "\n" ~ 
"x-amz-date:" ~ xamztime ~ "\n";

string signedHeaders = "host;x-amz-date";
string payloadHash = sha256Of("").toHexString.toLower;
	string canonicalRequest = method ~ "\n" ~ canonicalURI ~ "\n" ~ 
canonicalQueryString ~ "\n" ~ canonicalHeadersString ~ "\n" ~ 
signedHeaders ~ "\n" ~ payloadHash;



string algorithm = "AWS4-HMAC-SHA256";
	string credentialScope = curDateStr ~ "/" ~ region ~ "/" ~ 
service ~ "/" ~ "aws4_request";
	string stringToSign = algorithm ~ "\n" ~ xamztime ~ "\n" ~ 
credentialScope ~ "\n" ~ 
sha256Of(canonicalRequest).toHexString.toLower;


	auto signingKey = getSignatureKey(secretKey, curDateStr, region, 
service);


	string signature = hmac_sha256(signingKey, 
cast(ubyte[])stringToSign).toHexString.toLower;


	string authorizationHeader = algorithm ~ " " ~ "Credential=" ~ 
accessKey ~ "/" ~ credentialScope ~ ", " ~ "SignedHeaders=" ~ 
signedHeaders ~ ", " ~ "Signature=" ~ signature;


auto client = HTTP(endpoint ~ "?" ~ canonicalQueryString);
client.method = HTTP.Method.get;
client.addRequestHeader("x-amz-date", xamztime);
client.addRequestHeader("Authorization", authorizationHeader);
auto content = client.perform();

writeln(content);
}

Now will try to "pack" it to some class and make it more usable 
and universal. I think that thread can be closed/solved (if 
something like this exist here)


Thank you all for help.

//holo

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

On Friday, 9 October 2015 at 16:30:26 UTC, holo wrote:
OK i find out error, in addRequestHeader i was using ":" after 
header name what casing problem. I removed it and now im 
getting "unauthorized". Here is how it looks right now:


 HTTP/1.1 401 Unauthorized\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 401 
Unauthorized\r\n]

Request Version: HTTP/1.1
Status Code: 401
Response Phrase: Unauthorized
Transfer-Encoding: chunked\r\n
Date: Fri, 09 Oct 2015 16:22:47 GMT\r\n
Server: AmazonEC2\r\n
\r\n
[HTTP response 1/2]
[Next request in frame: 8371]
HTTP chunked response
Data chunk (254 octets)
Chunk size: 254 octets
Data (254 bytes)

In data field i can read:

"AWS was not able to validate provided access credentials" Is 
my signing process incorrect?


Maybe i will put my present code again:

#!/usr/bin/rdmd -L-lcurl

import std.stdio;
import std.string;
import std.file;
import std.datetime;
import std.process;
import std.digest.sha;
import std.net.curl;
import std.uri;
import sigv4;


auto zone = "us-east-1";
auto service = "ec2";


void main()
{
auto accKey = environment["AWS_ACCESS_KEY"];
auto secKey = environment["AWS_SECRET_KEY"];

auto currentClock = Clock.currTime;

auto currentDate = cast(Date)currentClock;
auto curDateStr = currentDate.toISOString;

auto currentTime = cast(TimeOfDay)currentClock;
auto curTimeStr = currentTime.toISOString;

auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

string[string] empty;

SignableRequest r;
r.dateString = curDateStr;
r.timeStringUTC = curTimeStr;
r.region = zone;
r.service = service;
r.canonicalRequest = CanonicalRequest(
"POST",
"/",
["action" : "DescribeInstances", "version" : 
"2013-10-15"],
//  ["accept" : "*/*",
			["content-type" : "application/x-www-form-urlencoded; 
charset=utf-8",

 "host" : service ~ ".amazonaws.com",
 "x-amz-date" : xamztime],
			 cast(ubyte[])"");  
//cast(ubyte[])"Action=DescribeInstances=2013-10-15");


	auto qParm = 
canonicalQueryString(r.canonicalRequest.queryParameters);


auto sigHead = canonicalHeaders(r.canonicalRequest.headers);

auto sigStr = signableString(r);

auto sigKey = signingKey(secKey, curDateStr, zone, service);

	auto signature = sign(sigKey, 
cast(ubyte[])sigStr).toHexString().toLower();


writeln();  
writeln(qParm);
writeln();
writeln(sigHead);
writeln();
writeln(sigStr);
writeln();
writeln(signature);
writeln();
auto client = HTTP();
//  client.clearRequestHeaders;
	client.addRequestHeader("content-type", 
"application/x-www-form-urlencoded; charset=utf-8");

client.addRequestHeader("host", service ~ ".amazonaws.com");
client.addRequestHeader("x-amz-date", xamztime);
	client.addRequestHeader("authorization", "AWS4-HMAC-SHA256" ~ " 
" ~ "Credential=" ~ accKey ~ "/" ~ xamztime ~ "/" ~ zone ~ "/" ~ 
service ~ "/" ~ "aws4_request" ~ ", " ~ "SignedHeaders=" ~ 
"content-type;host;x-amz-date" ~ ", " ~ "Signature=" ~ signature);


	auto url = "ec2.amazonaws.com/?" ~ 
"Action=DescribeInstances=2013-10-15";

auto urlenc = encode(url);
writeln(url);
auto content = get(urlenc, client);
writeln(content);
}

Is my signing process correct?

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

OK i find out error, in addRequestHeader i was using ":" after 
header name what casing problem. I removed it and now im getting 
"unauthorized". Here is how it looks right now:


 HTTP/1.1 401 Unauthorized\r\n
[Expert Info (Chat/Sequence): HTTP/1.1 401 
Unauthorized\r\n]

Request Version: HTTP/1.1
Status Code: 401
Response Phrase: Unauthorized
Transfer-Encoding: chunked\r\n
Date: Fri, 09 Oct 2015 16:22:47 GMT\r\n
Server: AmazonEC2\r\n
\r\n
[HTTP response 1/2]
[Next request in frame: 8371]
HTTP chunked response
Data chunk (254 octets)
Chunk size: 254 octets
Data (254 bytes)

In data field i can read:

"AWS was not able to validate provided access credentials" Is my 
signing process incorrect?

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

I correct typo (thats whats happening when you trying to solve 
problem after night at 6:00 AM ).But still  have same error. When 
use curl with same generated creditentials im getting "400". What 
is strange when im using same generated headers and signature 
with command line curl im getting "500". My headers right now:


auto client = HTTP();
client.clearRequestHeaders;
client.addRequestHeader("Content-Type:", 
"application/x-www-form-urlencoded; charset=utf-8");
client.addRequestHeader("Host:", service ~ 
".amazonaws.com");

client.addRequestHeader("X-Amz-Date:", xamztime);
client.addRequestHeader("Authorization:", 
"AWS4-HMAC-SHA256" ~ " " ~ "Credential=" ~ accKey ~ "/" ~ 
xamztime ~ "/" ~ zone ~ "/" ~ service ~ "/" ~ "aws4_request" ~ ", 
" ~ "SignedHeaders=" ~ "content-type;host;x-amz-date" ~ ", " ~ 
"Signature=" ~ signature);


auto url = service ~ ".amazonaws.com?" ~ 
"Action=DescribeInstances=2013-10-15";

writeln(url);
auto content = get(url, client);
writeln(content);


Did i set up it correctly?

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

I was fighting with it a little bit and after all i just leave 
original function which was in code, removed dependencies from 
vibe.d and finally tried to contact api. Here is my present code:


#!/usr/bin/rdmd -L-lcurl

import std.stdio;
import std.string;
import std.file;
import std.datetime;
import std.process;
import std.digest.sha;
import std.net.curl;
import sigv4;


auto zone = "us-east-1";
auto service = "ec2";


void main()
{
auto accKey = environment["AWS_ACCESS_KEY"];
auto secKey = environment["AWS_SECRET_KEY"];

auto currentClock = Clock.currTime;

auto currentDate = cast(Date)currentClock;
auto curDateStr = currentDate.toISOString;

auto currentTime = cast(TimeOfDay)currentClock;
auto curTimeStr = currentTime.toISOString;

auto xamztime = curDateStr ~ "T" ~ curTimeStr ~ "Z";

string[string] empty;

SignableRequest r;
r.dateString = curDateStr;
r.timeStringUTC = curTimeStr;
r.region = zone;
r.service = service;
r.canonicalRequest = CanonicalRequest(
"GET",
"/",
["action" : "DescribeInstances", "version" : 
"2013-10-15"],
			["content-type" : "application/x-www-form-urlencoded; 
charset=utf-8",

 "host" : service ~ ".amazonaws.com",
 "x-amz-date" : xamztime],
 cast(ubyte[])"");

	auto qParm = 
canonicalQueryString(r.canonicalRequest.queryParameters);


auto sigHead = canonicalHeaders(r.canonicalRequest.headers);

auto sigStr = signableString(r);

auto sigKey = signingKey(secKey, curDateStr, zone, service);

	auto signature = sign(sigKey, 
cast(ubyte[])sigStr).toHexString().toLower();


writeln();  
writeln(qParm);
writeln();
writeln(sigHead);
writeln();
writeln(sigStr);
writeln();
writeln(signature);
writeln();

auto client = HTTP();
client.clearRequestHeaders;
	client.addRequestHeader("content-type:", 
"application/x-www-form-urlencoded; charset=utf-8");

client.addRequestHeader("host:", service ~ ".amazonaws.com");
client.addRequestHeader("x-amz-date:", xamztime);
	client.addRequestHeader("Authoryzation:", "AWS4-HMAC-SHA256" ~ " 
" ~ "Credential=" ~ accKey ~ "/" ~ xamztime ~ "/" ~ zone ~ "/" ~ 
service ~ "/" ~ "aws4_request" ~ ", " ~ "SignedHeaders=" ~ 
"content-type;host;x-amz-date" ~ ", " ~ "Signature=" ~ signature);


	auto url = service ~ ".amazonaws.com?" ~ 
"Action=DescribeInstances=2013-10-15";

writeln(url);
auto content = get(url, client);
writeln(content);
}

Everything is compiling but im getting 400 (Bad Request):

  [root@ultraxps aws]# ./header.d

action=DescribeRegions=2013-10-15

content-type: application/x-www-form-urlencoded; charset=utf-8
host: ec2.amazonaws.com
x-amz-date: 20151009T053800Z


AWS4-HMAC-SHA256
20151009T053800Z
20151009/us-east-1/ec2/aws4_request
888595748692147ceafafcae3941ec0d83ac42c97641e4d954d7447a00c56270

69b1e4c5212cc6b485569fdfb43f7dde94413b36c50393c55d4810ced47f167b

ec2.amazonaws.com?Action=DescribeRegions=2013-10-15
std.net.curl.CurlException@/usr/include/dlang/dmd/std/net/curl.d(824): HTTP 
request returned status code 400 (Bad Request)

/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(pure @safe 
bool std.exception.enforce!(std.net.curl.CurlException, bool).enforce(bool, 
lazy const(char)[], immutable(char)[], ulong)+0x65) [0x5004bd]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[] 
std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], const(void)[], 
std.net.curl.HTTP)+0x1a2) [0x4fc0ba]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[] 
std.net.curl.get!(std.net.curl.HTTP, char).get(const(char)[], 
std.net.curl.HTTP)+0x6a) [0x4fbec2]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_Dmain+0x840)
 [0x4f8f98]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv+0x1f)
 [0x533eab]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void 
rt.dmain2._d_run_main(int, char**, extern (C) int 
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void 
rt.dmain2._d_run_main(int, char**, extern (C) int 
function(char[][])*).runAll()+0x2b) [0x533e67]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void 
rt.dmain2._d_run_main(int, char**, extern (C) int 
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_d_run_main+0x1d2)
 [0x533d86]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(main+0x12) 
[0x52cb62]

Re: AWS API Dlang, hmac sha256 function.

[Laeeth Isharc via Digitalmars-d-learn]

On Friday, 9 October 2015 at 04:04:57 UTC, holo wrote:

	r.dateString = 	client.addRequestHeader("Authoryzation:", 
"AWS4-HMAC-SHA256" ~ " " ~ "Credential=" ~ accKey ~ "/" ~ 
xamztime ~ "/" ~ zone ~ "/" ~ service ~ "/" ~ "aws4_request" ~ 
", " ~ "SignedHeaders=" ~ "content-type;host;x-amz-date" ~ ", " 
~ "Signature=" ~ signature);


authorisation ??? (Or maybe authorization) ie a typo (check other 
fields carefully too, just in case)




	auto url = service ~ ".amazonaws.com?" ~ 
"Action=DescribeInstances=2013-10-15";

writeln(url);
auto content = get(url, client);
writeln(content);
}

Everything is compiling but im getting 400 (Bad Request):

  [root@ultraxps aws]# ./header.d

action=DescribeRegions=2013-10-15

content-type: application/x-www-form-urlencoded; charset=utf-8
host: ec2.amazonaws.com
x-amz-date: 20151009T053800Z


AWS4-HMAC-SHA256
20151009T053800Z
20151009/us-east-1/ec2/aws4_request
888595748692147ceafafcae3941ec0d83ac42c97641e4d954d7447a00c56270

69b1e4c5212cc6b485569fdfb43f7dde94413b36c50393c55d4810ced47f167b

ec2.amazonaws.com?Action=DescribeRegions=2013-10-15
std.net.curl.CurlException@/usr/include/dlang/dmd/std/net/curl.d(824): HTTP 
request returned status code 400 (Bad Request)

/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(pure @safe 
bool std.exception.enforce!(std.net.curl.CurlException, bool).enforce(bool, 
lazy const(char)[], immutable(char)[], ulong)+0x65) [0x5004bd]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[] 
std.net.curl._basicHTTP!(char)._basicHTTP(const(char)[], const(void)[], 
std.net.curl.HTTP)+0x1a2) [0x4fc0ba]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(char[] 
std.net.curl.get!(std.net.curl.HTTP, char).get(const(char)[], 
std.net.curl.HTTP)+0x6a) [0x4fbec2]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_Dmain+0x840)
 [0x4f8f98]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_D2rt6dmain211_d_run_mainUiPPaPUAAaZiZ6runAllMFZ9__lambda1MFZv+0x1f)
 [0x533eab]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void 
rt.dmain2._d_run_main(int, char**, extern (C) int 
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void 
rt.dmain2._d_run_main(int, char**, extern (C) int 
function(char[][])*).runAll()+0x2b) [0x533e67]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(void 
rt.dmain2._d_run_main(int, char**, extern (C) int 
function(char[][])*).tryExec(scope void delegate())+0x2a) [0x533e06]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(_d_run_main+0x1d2)
 [0x533d86]
/tmp/.rdmd-0/rdmd-header.d-54C0A2BD6BD71C27D9AC7319D786A6F3/header(main+0x12) 
[0x52cb62]
/usr/lib/libc.so.6(__libc_start_main+0xf0) [0x7f9c39f71610]
[root@ultraxps aws]#

What am i doing wrong? What is strange when when i do same with 
curl its responding normally of course is not authenticated).


[root@ultraxps aws]# curl 
ec2.amazonaws.com?Action=DescribeInstances=2013-10-15

[1] 20390
[root@ultraxps aws]# 
MissingParameterThe request must contain the parameter 
AWSAccessKeyIde1352781-c2b4-4e74-ade3-80d655efd0ac

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

Congrats on getting it working!


@Rikki Thanks :)

I was trying to write my own lib from beginning based on examples 
but after some time i resign from that idea (will back to it when 
i will have some more experience) and right now im trying to 
customize that one from link which yawniek paste:


https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d

I removed import from vibe.d library and copy/paste missed 
functions formEncode and filterURLEncode (BTW: what that "(R)" 
mean in it? filterURLEncode(R)(ref R dst, ..., ..) ).


Next thing what i did was to replace hmac function to use hmac 
module from newest library. Now whole code looks like this:


module sigv4;

import std.array;
import std.algorithm;
import std.digest.sha;
import std.range;
import std.stdio;
import std.string;
import std.format;
import std.digest.hmac;


const algorithm = "AWS4-HMAC-SHA256";


void filterURLEncode(R)(ref R dst, string str, string 
allowed_chars = null, bool form_encoding = false)

{
while( str.length > 0 ) {
switch(str[0]) {
case ' ':
if (form_encoding) {
dst.put('+');
break;
}
goto default;
case 'A': .. case 'Z':
case 'a': .. case 'z':
case '0': .. case '9':
case '-': case '_': case '.': case '~':
dst.put(str[0]);
break;
default:
if (allowed_chars.canFind(str[0])) 
dst.put(str[0]);
else formattedWrite(dst, "%%%02X", str[0]);
}
str = str[1 .. $];
}
}

string formEncode(string str, string allowed_chars = null)
@safe {
auto dst = appender!string();
dst.reserve(str.length);
filterURLEncode(dst, str, allowed_chars, true);
return dst.data;
}

struct CanonicalRequest
{
string method;
string uri;
string[string] queryParameters;
string[string] headers;
ubyte[] payload;
}

string canonicalQueryString(string[string] queryParameters)
{
alias encode = formEncode;

string[string] encoded;
foreach (p; queryParameters.keys())
{
encoded[encode(p)] = encode(queryParameters[p]);
}
string[] keys = encoded.keys();
sort(keys);
return keys.map!(k => k ~ "=" ~ encoded[k]).join("&");
}

string canonicalHeaders(string[string] headers)
{
string[string] trimmed;
foreach (h; headers.keys())
{
trimmed[h.toLower().strip()] = headers[h].strip();
}
string[] keys = trimmed.keys();
sort(keys);
return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join("");
}

string signedHeaders(string[string] headers)
{
string[] keys = headers.keys().map!(k => k.toLower()).array();
sort(keys);
return keys.join(";");
}

string hash(T)(T payload)
{
auto hash = sha256Of(payload);
return hash.toHexString().toLower();
}

string makeCRSigV4(CanonicalRequest r)
{
auto cr =
r.method.toUpper() ~ "\n" ~
(r.uri.empty ? "/" : r.uri) ~ "\n" ~
canonicalQueryString(r.queryParameters) ~ "\n" ~
canonicalHeaders(r.headers) ~ "\n" ~
signedHeaders(r.headers) ~ "\n" ~
hash(r.payload);

return hash(cr);
}

unittest {
string[string] empty;

auto r = CanonicalRequest(
"POST",
"/",
empty,
["content-type": "application/x-www-form-urlencoded; 
charset=utf-8",

 "host": "iam.amazonaws.com",
 "x-amz-date": "20110909T233600Z"],
cast(ubyte[])"Action=ListUsers=2010-05-08");

auto sig = makeCRSigV4(r);

assert(sig == 
"3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2");

}

struct SignableRequest
{
string dateString;
string timeStringUTC;
string region;
string service;
CanonicalRequest canonicalRequest;
}

string signableString(SignableRequest r) {
return algorithm ~ "\n" ~
r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~
r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~ 
"/aws4_request\n" ~

makeCRSigV4(r.canonicalRequest);
}

unittest {
string[string] empty;

SignableRequest r;
r.dateString = "20110909";
r.timeStringUTC = "233600";
r.region = "us-east-1";
r.service = "iam";
r.canonicalRequest = CanonicalRequest(
"POST",
"/",
empty,
["content-type": "application/x-www-form-urlencoded; 
charset=utf-8",

 "host": "iam.amazonaws.com",
 "x-amz-date": "20110909T233600Z"],
cast(ubyte[])"Action=ListUsers=2010-05-08");

auto sampleString =
algorithm ~ "\n" ~
"20110909T233600Z\n" ~

Re: AWS API Dlang, hmac sha256 function.

[Rikki Cattermole via Digitalmars-d-learn]

On 07/10/15 3:18 PM, holo wrote:

Congrats on getting it working!


@Rikki Thanks :)

I was trying to write my own lib from beginning based on examples but
after some time i resign from that idea (will back to it when i will
have some more experience) and right now im trying to customize that one
from link which yawniek paste:

https://github.com/yannick/vibe-aws/blob/master/source/vibe/aws/sigv4.d

I removed import from vibe.d library and copy/paste missed functions
formEncode and filterURLEncode (BTW: what that "(R)" mean in it?
filterURLEncode(R)(ref R dst, ..., ..) ).


If you see a template argument (which R is) it is typically meant for a 
range. In this case an output range. Although there should be 
isOutputRange!R in an template if condition, to check this.



Next thing what i did was to replace hmac function to use hmac module
from newest library. Now whole code looks like this:

module sigv4;

import std.array;
import std.algorithm;
import std.digest.sha;
import std.range;
import std.stdio;
import std.string;
import std.format;
import std.digest.hmac;


const algorithm = "AWS4-HMAC-SHA256";


void filterURLEncode(R)(ref R dst, string str, string allowed_chars =
null, bool form_encoding = false)
{
 while( str.length > 0 ) {
 switch(str[0]) {
 case ' ':
 if (form_encoding) {
 dst.put('+');
 break;
 }
 goto default;
 case 'A': .. case 'Z':
 case 'a': .. case 'z':
 case '0': .. case '9':
 case '-': case '_': case '.': case '~':
 dst.put(str[0]);
 break;
 default:
 if (allowed_chars.canFind(str[0])) dst.put(str[0]);
 else formattedWrite(dst, "%%%02X", str[0]);
 }
 str = str[1 .. $];
 }
}

string formEncode(string str, string allowed_chars = null)
@safe {
 auto dst = appender!string();
 dst.reserve(str.length);
 filterURLEncode(dst, str, allowed_chars, true);
 return dst.data;
}

struct CanonicalRequest
{
 string method;
 string uri;
 string[string] queryParameters;
 string[string] headers;
 ubyte[] payload;
}

string canonicalQueryString(string[string] queryParameters)
{
 alias encode = formEncode;

 string[string] encoded;
 foreach (p; queryParameters.keys())
 {
 encoded[encode(p)] = encode(queryParameters[p]);
 }
 string[] keys = encoded.keys();
 sort(keys);
 return keys.map!(k => k ~ "=" ~ encoded[k]).join("&");
}

string canonicalHeaders(string[string] headers)
{
 string[string] trimmed;
 foreach (h; headers.keys())
 {
 trimmed[h.toLower().strip()] = headers[h].strip();
 }
 string[] keys = trimmed.keys();
 sort(keys);
 return keys.map!(k => k ~ ":" ~ trimmed[k] ~ "\n").join("");
}

string signedHeaders(string[string] headers)
{
 string[] keys = headers.keys().map!(k => k.toLower()).array();
 sort(keys);
 return keys.join(";");
}

string hash(T)(T payload)
{
 auto hash = sha256Of(payload);
 return hash.toHexString().toLower();
}

string makeCRSigV4(CanonicalRequest r)
{
 auto cr =
 r.method.toUpper() ~ "\n" ~
 (r.uri.empty ? "/" : r.uri) ~ "\n" ~
 canonicalQueryString(r.queryParameters) ~ "\n" ~
 canonicalHeaders(r.headers) ~ "\n" ~
 signedHeaders(r.headers) ~ "\n" ~
 hash(r.payload);

 return hash(cr);
}

unittest {
 string[string] empty;

 auto r = CanonicalRequest(
 "POST",
 "/",
 empty,
 ["content-type": "application/x-www-form-urlencoded;
charset=utf-8",
  "host": "iam.amazonaws.com",
  "x-amz-date": "20110909T233600Z"],
 cast(ubyte[])"Action=ListUsers=2010-05-08");

 auto sig = makeCRSigV4(r);

 assert(sig ==
"3511de7e95d28ecd39e9513b642aee07e54f4941150d8df8bf94b328ef7e55e2");
}

struct SignableRequest
{
 string dateString;
 string timeStringUTC;
 string region;
 string service;
 CanonicalRequest canonicalRequest;
}

string signableString(SignableRequest r) {
 return algorithm ~ "\n" ~
 r.dateString ~ "T" ~ r.timeStringUTC ~ "Z\n" ~
 r.dateString ~ "/" ~ r.region ~ "/" ~ r.service ~
"/aws4_request\n" ~
 makeCRSigV4(r.canonicalRequest);
}

unittest {
 string[string] empty;

 SignableRequest r;
 r.dateString = "20110909";
 r.timeStringUTC = "233600";
 r.region = "us-east-1";
 r.service = "iam";
 r.canonicalRequest = CanonicalRequest(
 "POST",
 "/",
 empty,
 ["content-type": "application/x-www-form-urlencoded;
charset=utf-8",
  "host": "iam.amazonaws.com",
  "x-amz-date": "20110909T233600Z"],
 cast(ubyte[])"Action=ListUsers=2010-05-08");

 auto sampleString =
 algorithm ~ "\n" ~
 

Re: AWS API Dlang, hmac sha256 function.

[Rikki Cattermole via Digitalmars-d-learn]

On 06/10/15 11:26 AM, holo wrote:

On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev wrote:

On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:

@Vladimir where can i check or do you know when next version of
phobos will be available?


You can use Digger to get the latest version of D:

https://github.com/CyberShadow/Digger


Thank you, now it is working as expected. Hope i wont miss upgrade in my
system to uninstall digger version before.

[holo@ultraxps workplace]$ cat ./hmac.d
#!/usr/bin/rdmd

import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
 auto hmac = hmac!SHA256("secretkey".representation);
 hmac.put("texttohash".representation);
 auto digest = hmac.finish();

 writeln(digest);
}

[holo@ultraxps workplace]$ ./hmac.d
[28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 148, 84,
170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 252, 26, 124,
215, 6]
[holo@ultraxps workplace]$ dmd --version
DMD64 D Compiler v2.069-devel-5e73c30
Copyright (c) 1999-2015 by Digital Mars written by Walter Bright
[holo@ultraxps workplace]$

Now need try to write some code which will connect me to AWS api based
on examples.


Congrats on getting it working!

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

On Saturday, 3 October 2015 at 23:58:39 UTC, Rikki Cattermole 
wrote:

On 04/10/15 2:31 AM, holo wrote:
On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole 
wrote:

On 04/10/15 1:49 AM, holo wrote:
On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki 
Cattermole wrote:

On 04/10/15 1:09 AM, holo wrote:

[...]


By the looks of things the problem is with SHA256, I'm 
guessing it

doesn't have its block size defined.
https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394


You'll also need to update sha.d as well. Otherwise it 
looks good.


I updated sha.d and now im getting following error:

$ ./app.d
/usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure 
function
'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call 
impure

function 'core.cpuid.ssse3'
/usr/include/dlang/dmd/std/digest/digest.d(285): Error: 
template
instance std.range.primitives.isOutputRange!(SHA!(512u, 
160u),

const(ubyte)[]) error instantiating
/usr/include/dlang/dmd/std/digest/digest.d(851): 
instantiated from here:

isDigest!(SHA!(512u, 160u))
/usr/include/dlang/dmd/std/digest/sha.d(1179):while 
looking for

match for WrapperDigest!(SHA!(512u, 160u))
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]


And one more file (ssse3.d) needs to be updated.
You are going to be playing whack a mole I think for a while.


I downloaded whole master branch phobos and tried to use it 
(changed
path in dmd.conf), but there are missing much more files (i 
think that
master branch is not ready yet). So i get back to my original 
library
and tried to updated file you mention. But i can't find it 
(ssse3.d),
its not appearing in my stable lib and even in that 
master-branch from

zip file.


Apologies, I didn't see this till after I got to bed. It's 
core.cpuid not ssse3.d. It will be in druntime, not Phobos.


Sorry i was busy at weekend. Thank you all for answers. I will 
try to write some solution basing on yawniek example and will 
back if some troubles appears.


@Vladimir where can i check or do you know when next version of 
phobos will be available?


@Rikki i downloaded both master zip files for runtime and phobos 
librarys all dependencies looks are satisfied right now but still 
getting errors:


[holo@ultraxps workplace]$ dmd hmac.d
hmac.o: In function 
`_D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':

hmac.d:(.text._D3std6digest4hmac58__T4hmacTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4hmacFNaNbNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x18):
 undefined reference to `_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6__initZ'
hmac.o: In function 
`_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':

hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x1b):
 undefined reference to 
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x31):
 undefined reference to 
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC6__ctorMFNaNbNcNiNfMAxhZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x3e):
 undefined reference to 
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA6finishMFNaNbNiNeZG20h'
hmac.o: In function 
`_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':

hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0x8d):
 undefined reference to 
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA5startMFNaNbNiNfZv'
hmac.d:(.text._D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC5startMFNaNbNcNiNjNfZS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC+0xa4):
 undefined reference to 
`_D3std6digest3sha20__T3SHAVki512Vki160Z3SHA3putMFNaNbNiNeMAxhXv'
hmac.o: In function 
`_D3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC3putMFNaNbNcNiNjNfxAhXS3std6digest4hmac58__T4HMACTS3std6digest3sha20__T3SHAVki512Vki160Z3SHAVmi512Z4HMAC':

Re: AWS API Dlang, hmac sha256 function.

[Vladimir Panteleev via Digitalmars-d-learn]

On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
@Vladimir where can i check or do you know when next version of 
phobos will be available?


You can use Digger to get the latest version of D:

https://github.com/CyberShadow/Digger

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

On Monday, 5 October 2015 at 21:00:38 UTC, Vladimir Panteleev 
wrote:

On Monday, 5 October 2015 at 19:43:39 UTC, holo wrote:
@Vladimir where can i check or do you know when next version 
of phobos will be available?


You can use Digger to get the latest version of D:

https://github.com/CyberShadow/Digger


Thank you, now it is working as expected. Hope i wont miss 
upgrade in my system to uninstall digger version before.


[holo@ultraxps workplace]$ cat ./hmac.d
#!/usr/bin/rdmd

import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
auto hmac = hmac!SHA256("secretkey".representation);
hmac.put("texttohash".representation);
auto digest = hmac.finish();

writeln(digest);
}

[holo@ultraxps workplace]$ ./hmac.d
[28, 169, 142, 131, 133, 104, 149, 47, 205, 215, 20, 154, 170, 
148, 84, 170, 252, 36, 10, 119, 18, 25, 10, 145, 183, 133, 135, 
252, 26, 124, 215, 6]

[holo@ultraxps workplace]$ dmd --version
DMD64 D Compiler v2.069-devel-5e73c30
Copyright (c) 1999-2015 by Digital Mars written by Walter Bright
[holo@ultraxps workplace]$

Now need try to write some code which will connect me to AWS api 
based on examples.

Re: AWS API Dlang, hmac sha256 function.

[Rikki Cattermole via Digitalmars-d-learn]

On 04/10/15 1:09 AM, holo wrote:

On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole wrote:

On 03/10/15 6:01 PM, Rikki Cattermole wrote:

On 03/10/15 4:54 PM, holo wrote:

On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole wrote:


You could implement it yourself, (it looks pretty easy).
Or go the route of Botan:
https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d






Thank you for answer. I used dub to fetch botan, after that i created
project:

dub init projectname botan

and tried to add botan module:

$ cat app.d
#!/usr/bin/rdmd

import std.stdio;
import botan.mac.hmac;

void main()
{
 writeln("Edit source/app.d to start your project.");
}

but when im trying to run my simple app i get:

$ ./app.d
./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which
cannot be read
import path[0] = .
import path[1] = /usr/include/dlang/dmd
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

What am i doing wrong? How to add library with dub package manager? Im
really beginner in D (basically in programming) sorry if im asking
obvious questions.


By the looks of that error message with paths, botan isn't actually
added as a dependency. Can you please paste the dub file?


O wait nevermind.
Use dub to compile/run your program. You are using the shebang line!

$ dub run


According to my other question are that functions equivalent to that
python one? If they are not doing exact that same i could not create
auth string which is needed to connect to AWS api.


I can't say for certain if it is not exact. The reason I did not answer
it is because I am not familiar with environment or any of what you are
doing.
I can unfortunately only point you in the right direction.


Thank you for explaining how to use dub.

I find out in master branch of phobos on git hub is included hmac.d
module which will be added in next release of phobos so i think that
will be best solution for me to stick with standard library for beginning.

https://github.com/D-Programming-Language/phobos/tree/master/std

I downloaded hmac.d and new version of digest.d file and place them in
my phobos library. I think it exactly contains what i need but when im
trying to test it i have such error:

$ ./app.d
./app.d(9): Error: template std.digest.hmac.hmac cannot deduce function
from argument types !(SHA!(512, 256))(immutable(ubyte)[]), candidates are:
/usr/include/dlang/dmd/std/digest/hmac.d(202): std.digest.hmac.hmac(H)
if (isDigest!H && hasBlockSize!H)
/usr/include/dlang/dmd/std/digest/hmac.d(208): std.digest.hmac.hmac(H,
ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H)
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]



My test code (based on unint test from module) looks like that:

#!/usr/bin/rdmd

import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
 auto hmac = hmac!SHA256("secretkey".representation);
 auto digest = hmac.put("texttohash".representation).finish;
 writeln(digest);
}

Am i using it correctly?


By the looks of things the problem is with SHA256, I'm guessing it 
doesn't have its block size defined.

https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394
You'll also need to update sha.d as well. Otherwise it looks good.

Re: AWS API Dlang, hmac sha256 function.

[Rikki Cattermole via Digitalmars-d-learn]

On 04/10/15 1:49 AM, holo wrote:

On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote:

On 04/10/15 1:09 AM, holo wrote:

On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole wrote:

On 03/10/15 6:01 PM, Rikki Cattermole wrote:

On 03/10/15 4:54 PM, holo wrote:

[...]


By the looks of that error message with paths, botan isn't actually
added as a dependency. Can you please paste the dub file?


O wait nevermind.
Use dub to compile/run your program. You are using the shebang line!

$ dub run


[...]


I can't say for certain if it is not exact. The reason I did not
answer
it is because I am not familiar with environment or any of what you
are
doing.
I can unfortunately only point you in the right direction.


Thank you for explaining how to use dub.

I find out in master branch of phobos on git hub is included hmac.d
module which will be added in next release of phobos so i think that
will be best solution for me to stick with standard library for
beginning.

https://github.com/D-Programming-Language/phobos/tree/master/std

I downloaded hmac.d and new version of digest.d file and place them in
my phobos library. I think it exactly contains what i need but when im
trying to test it i have such error:

$ ./app.d
./app.d(9): Error: template std.digest.hmac.hmac cannot deduce function
from argument types !(SHA!(512, 256))(immutable(ubyte)[]), candidates
are:
/usr/include/dlang/dmd/std/digest/hmac.d(202): std.digest.hmac.hmac(H)
if (isDigest!H && hasBlockSize!H)
/usr/include/dlang/dmd/std/digest/hmac.d(208): std.digest.hmac.hmac(H,
ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H)
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]



My test code (based on unint test from module) looks like that:

#!/usr/bin/rdmd

import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
 auto hmac = hmac!SHA256("secretkey".representation);
 auto digest = hmac.put("texttohash".representation).finish;
 writeln(digest);
}

Am i using it correctly?


By the looks of things the problem is with SHA256, I'm guessing it
doesn't have its block size defined.
https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394

You'll also need to update sha.d as well. Otherwise it looks good.


I updated sha.d and now im getting following error:

$ ./app.d
/usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function
'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure
function 'core.cpuid.ssse3'
/usr/include/dlang/dmd/std/digest/digest.d(285): Error: template
instance std.range.primitives.isOutputRange!(SHA!(512u, 160u),
const(ubyte)[]) error instantiating
/usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here:
isDigest!(SHA!(512u, 160u))
/usr/include/dlang/dmd/std/digest/sha.d(1179):while looking for
match for WrapperDigest!(SHA!(512u, 160u))
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]


And one more file (ssse3.d) needs to be updated.
You are going to be playing whack a mole I think for a while.

Re: AWS API Dlang, hmac sha256 function.

[yawniek via Digitalmars-d-learn]

On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote:

Hello

I'm trying to contact AWS API with D according to documentation:

[...]


check https://github.com/yannick/vibe-aws
it has v4 implemented

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole 
wrote:

On 04/10/15 1:09 AM, holo wrote:
On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole 
wrote:

On 03/10/15 6:01 PM, Rikki Cattermole wrote:

On 03/10/15 4:54 PM, holo wrote:

[...]


By the looks of that error message with paths, botan isn't 
actually

added as a dependency. Can you please paste the dub file?


O wait nevermind.
Use dub to compile/run your program. You are using the 
shebang line!


$ dub run


[...]


I can't say for certain if it is not exact. The reason I did 
not answer
it is because I am not familiar with environment or any of 
what you are

doing.
I can unfortunately only point you in the right direction.


Thank you for explaining how to use dub.

I find out in master branch of phobos on git hub is included 
hmac.d
module which will be added in next release of phobos so i 
think that
will be best solution for me to stick with standard library 
for beginning.


https://github.com/D-Programming-Language/phobos/tree/master/std

I downloaded hmac.d and new version of digest.d file and place 
them in
my phobos library. I think it exactly contains what i need but 
when im

trying to test it i have such error:

$ ./app.d
./app.d(9): Error: template std.digest.hmac.hmac cannot deduce 
function
from argument types !(SHA!(512, 256))(immutable(ubyte)[]), 
candidates are:
/usr/include/dlang/dmd/std/digest/hmac.d(202): 
std.digest.hmac.hmac(H)

if (isDigest!H && hasBlockSize!H)
/usr/include/dlang/dmd/std/digest/hmac.d(208): 
std.digest.hmac.hmac(H,

ulong blockSize)(scope const(ubyte)[] secret) if (isDigest!H)
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]



My test code (based on unint test from module) looks like that:

#!/usr/bin/rdmd

import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
 auto hmac = hmac!SHA256("secretkey".representation);
 auto digest = 
hmac.put("texttohash".representation).finish;

 writeln(digest);
}

Am i using it correctly?


By the looks of things the problem is with SHA256, I'm guessing 
it doesn't have its block size defined.

https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394
You'll also need to update sha.d as well. Otherwise it looks 
good.


I updated sha.d and now im getting following error:

$ ./app.d
/usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure 
function 'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot 
call impure function 'core.cpuid.ssse3'
/usr/include/dlang/dmd/std/digest/digest.d(285): Error: template 
instance std.range.primitives.isOutputRange!(SHA!(512u, 160u), 
const(ubyte)[]) error instantiating
/usr/include/dlang/dmd/std/digest/digest.d(851):
instantiated from here: isDigest!(SHA!(512u, 160u))
/usr/include/dlang/dmd/std/digest/sha.d(1179):while 
looking for match for WrapperDigest!(SHA!(512u, 160u))

Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole 
wrote:

On 04/10/15 1:49 AM, holo wrote:
On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole 
wrote:

On 04/10/15 1:09 AM, holo wrote:

[...]


By the looks of things the problem is with SHA256, I'm 
guessing it

doesn't have its block size defined.
https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394

You'll also need to update sha.d as well. Otherwise it looks 
good.


I updated sha.d and now im getting following error:

$ ./app.d
/usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure 
function
'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call 
impure

function 'core.cpuid.ssse3'
/usr/include/dlang/dmd/std/digest/digest.d(285): Error: 
template

instance std.range.primitives.isOutputRange!(SHA!(512u, 160u),
const(ubyte)[]) error instantiating
/usr/include/dlang/dmd/std/digest/digest.d(851): instantiated 
from here:

isDigest!(SHA!(512u, 160u))
/usr/include/dlang/dmd/std/digest/sha.d(1179):while 
looking for

match for WrapperDigest!(SHA!(512u, 160u))
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]


And one more file (ssse3.d) needs to be updated.
You are going to be playing whack a mole I think for a while.


I downloaded whole master branch phobos and tried to use it 
(changed path in dmd.conf), but there are missing much more files 
(i think that master branch is not ready yet). So i get back to 
my original library and tried to updated file you mention. But i 
can't find it (ssse3.d), its not appearing in my stable lib and 
even in that master-branch from zip file.

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

On Saturday, 3 October 2015 at 05:02:58 UTC, Rikki Cattermole 
wrote:

On 03/10/15 6:01 PM, Rikki Cattermole wrote:

On 03/10/15 4:54 PM, holo wrote:
On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole 
wrote:


You could implement it yourself, (it looks pretty easy).
Or go the route of Botan:
https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d





Thank you for answer. I used dub to fetch botan, after that i 
created

project:

dub init projectname botan

and tried to add botan module:

$ cat app.d
#!/usr/bin/rdmd

import std.stdio;
import botan.mac.hmac;

void main()
{
 writeln("Edit source/app.d to start your project.");
}

but when im trying to run my simple app i get:

$ ./app.d
./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' 
which

cannot be read
import path[0] = .
import path[1] = /usr/include/dlang/dmd
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

What am i doing wrong? How to add library with dub package 
manager? Im
really beginner in D (basically in programming) sorry if im 
asking

obvious questions.


By the looks of that error message with paths, botan isn't 
actually

added as a dependency. Can you please paste the dub file?


O wait nevermind.
Use dub to compile/run your program. You are using the shebang 
line!


$ dub run

According to my other question are that functions equivalent 
to that
python one? If they are not doing exact that same i could not 
create

auth string which is needed to connect to AWS api.


I can't say for certain if it is not exact. The reason I did 
not answer
it is because I am not familiar with environment or any of 
what you are

doing.
I can unfortunately only point you in the right direction.


Thank you for explaining how to use dub.

I find out in master branch of phobos on git hub is included 
hmac.d module which will be added in next release of phobos so i 
think that will be best solution for me to stick with standard 
library for beginning.


https://github.com/D-Programming-Language/phobos/tree/master/std

I downloaded hmac.d and new version of digest.d file and place 
them in my phobos library. I think it exactly contains what i 
need but when im trying to test it i have such error:


$ ./app.d
./app.d(9): Error: template std.digest.hmac.hmac cannot deduce 
function from argument types !(SHA!(512, 
256))(immutable(ubyte)[]), candidates are:
/usr/include/dlang/dmd/std/digest/hmac.d(202):
std.digest.hmac.hmac(H) if (isDigest!H && hasBlockSize!H)
/usr/include/dlang/dmd/std/digest/hmac.d(208):
std.digest.hmac.hmac(H, ulong blockSize)(scope const(ubyte)[] 
secret) if (isDigest!H)

Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]



My test code (based on unint test from module) looks like that:

#!/usr/bin/rdmd

import std.stdio;
import std.digest.sha, std.digest.hmac;
import std.string : representation;

void main()
{
auto hmac = hmac!SHA256("secretkey".representation);
auto digest = hmac.put("texttohash".representation).finish;
writeln(digest);
}

Am i using it correctly?

Re: AWS API Dlang, hmac sha256 function.

[Vladimir Panteleev via Digitalmars-d-learn]

On Saturday, 3 October 2015 at 03:11:06 UTC, holo wrote:

Last but not least, how to write such function in D:

def sign(key, msg):
return hmac.new(key, msg.encode("utf-8"), 
hashlib.sha256).digest()


?

I can't find in standard libraryt hmac function, is it existing?


The next version of D will have a std.digest.hmac module:

http://dlang.org/phobos-prerelease/std_digest_hmac.html

Re: AWS API Dlang, hmac sha256 function.

[Rikki Cattermole via Digitalmars-d-learn]

On 04/10/15 2:31 AM, holo wrote:

On Saturday, 3 October 2015 at 12:50:58 UTC, Rikki Cattermole wrote:

On 04/10/15 1:49 AM, holo wrote:

On Saturday, 3 October 2015 at 12:22:11 UTC, Rikki Cattermole wrote:

On 04/10/15 1:09 AM, holo wrote:

[...]


By the looks of things the problem is with SHA256, I'm guessing it
doesn't have its block size defined.
https://github.com/D-Programming-Language/phobos/blob/master/std/digest/digest.d#L394


You'll also need to update sha.d as well. Otherwise it looks good.


I updated sha.d and now im getting following error:

$ ./app.d
/usr/include/dlang/dmd/std/digest/sha.d(225): Error: pure function
'std.digest.sha.SHA!(512u, 160u).SHA.transform' cannot call impure
function 'core.cpuid.ssse3'
/usr/include/dlang/dmd/std/digest/digest.d(285): Error: template
instance std.range.primitives.isOutputRange!(SHA!(512u, 160u),
const(ubyte)[]) error instantiating
/usr/include/dlang/dmd/std/digest/digest.d(851): instantiated from here:
isDigest!(SHA!(512u, 160u))
/usr/include/dlang/dmd/std/digest/sha.d(1179):while looking for
match for WrapperDigest!(SHA!(512u, 160u))
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]


And one more file (ssse3.d) needs to be updated.
You are going to be playing whack a mole I think for a while.


I downloaded whole master branch phobos and tried to use it (changed
path in dmd.conf), but there are missing much more files (i think that
master branch is not ready yet). So i get back to my original library
and tried to updated file you mention. But i can't find it (ssse3.d),
its not appearing in my stable lib and even in that master-branch from
zip file.


Apologies, I didn't see this till after I got to bed. It's core.cpuid 
not ssse3.d. It will be in druntime, not Phobos.

Re: AWS API Dlang, hmac sha256 function.

[Rikki Cattermole via Digitalmars-d-learn]

On 03/10/15 6:01 PM, Rikki Cattermole wrote:

On 03/10/15 4:54 PM, holo wrote:

On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole wrote:


You could implement it yourself, (it looks pretty easy).
Or go the route of Botan:
https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d





Thank you for answer. I used dub to fetch botan, after that i created
project:

dub init projectname botan

and tried to add botan module:

$ cat app.d
#!/usr/bin/rdmd

import std.stdio;
import botan.mac.hmac;

void main()
{
 writeln("Edit source/app.d to start your project.");
}

but when im trying to run my simple app i get:

$ ./app.d
./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which
cannot be read
import path[0] = .
import path[1] = /usr/include/dlang/dmd
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

What am i doing wrong? How to add library with dub package manager? Im
really beginner in D (basically in programming) sorry if im asking
obvious questions.


By the looks of that error message with paths, botan isn't actually
added as a dependency. Can you please paste the dub file?


O wait nevermind.
Use dub to compile/run your program. You are using the shebang line!

$ dub run


According to my other question are that functions equivalent to that
python one? If they are not doing exact that same i could not create
auth string which is needed to connect to AWS api.


I can't say for certain if it is not exact. The reason I did not answer
it is because I am not familiar with environment or any of what you are
doing.
I can unfortunately only point you in the right direction.

Re: AWS API Dlang, hmac sha256 function.

[Rikki Cattermole via Digitalmars-d-learn]

On 03/10/15 4:54 PM, holo wrote:

On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole wrote:


You could implement it yourself, (it looks pretty easy).
Or go the route of Botan:
https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d




Thank you for answer. I used dub to fetch botan, after that i created
project:

dub init projectname botan

and tried to add botan module:

$ cat app.d
#!/usr/bin/rdmd

import std.stdio;
import botan.mac.hmac;

void main()
{
 writeln("Edit source/app.d to start your project.");
}

but when im trying to run my simple app i get:

$ ./app.d
./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' which
cannot be read
import path[0] = .
import path[1] = /usr/include/dlang/dmd
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

What am i doing wrong? How to add library with dub package manager? Im
really beginner in D (basically in programming) sorry if im asking
obvious questions.


By the looks of that error message with paths, botan isn't actually 
added as a dependency. Can you please paste the dub file?



According to my other question are that functions equivalent to that
python one? If they are not doing exact that same i could not create
auth string which is needed to connect to AWS api.


I can't say for certain if it is not exact. The reason I did not answer 
it is because I am not familiar with environment or any of what you are 
doing.

I can unfortunately only point you in the right direction.

AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

Hello

I'm trying to contact AWS API with D according to documentation:

http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html

there is written example in python which i want to rewrite to D:

http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html

I want to ask about such 2 things:

Is it equivalent in D:

hexDigest!SHA256("sometexttohash");

for such python method:

hashlib.sha256('sometexttohash').hexdigest()

?

Maybe is there some prettier solution in D eg:

toHexString(sha256Of("sometexttohash"));

or

SHA256 sha;
sha.start();
string data = "sometexttohash";
sha.put(data);
toHexString(sha.finish())

If those are equivalents to python method which should i choose?

Last but not least, how to write such function in D:

def sign(key, msg):
return hmac.new(key, msg.encode("utf-8"), 
hashlib.sha256).digest()


?

I can't find in standard libraryt hmac function, is it existing?

//holo

Re: AWS API Dlang, hmac sha256 function.

[Rikki Cattermole via Digitalmars-d-learn]

On 03/10/15 4:11 PM, holo wrote:

Hello

I'm trying to contact AWS API with D according to documentation:

http://docs.aws.amazon.com/general/latest/gr/sigv4_signing.html

there is written example in python which i want to rewrite to D:

http://docs.aws.amazon.com/general/latest/gr/sigv4-signed-request-examples.html


I want to ask about such 2 things:

Is it equivalent in D:

hexDigest!SHA256("sometexttohash");

for such python method:

hashlib.sha256('sometexttohash').hexdigest()

?

Maybe is there some prettier solution in D eg:

toHexString(sha256Of("sometexttohash"));

or

SHA256 sha;
sha.start();
string data = "sometexttohash";
sha.put(data);
toHexString(sha.finish())

If those are equivalents to python method which should i choose?

Last but not least, how to write such function in D:

def sign(key, msg):
 return hmac.new(key, msg.encode("utf-8"), hashlib.sha256).digest()

?

I can't find in standard libraryt hmac function, is it existing?

//holo


You could implement it yourself, (it looks pretty easy).
Or go the route of Botan: 
https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d

Re: AWS API Dlang, hmac sha256 function.

[holo via Digitalmars-d-learn]

On Saturday, 3 October 2015 at 03:15:21 UTC, Rikki Cattermole 
wrote:


You could implement it yourself, (it looks pretty easy).
Or go the route of Botan: 
https://github.com/etcimon/botan/blob/3bdc835d5b9bad7523deaa86fe98b1fbb2f09d6b/source/botan/mac/hmac.d



Thank you for answer. I used dub to fetch botan, after that i 
created project:


dub init projectname botan

and tried to add botan module:

$ cat app.d
#!/usr/bin/rdmd

import std.stdio;
import botan.mac.hmac;

void main()
{
writeln("Edit source/app.d to start your project.");
}

but when im trying to run my simple app i get:

$ ./app.d
./app.d(4): Error: module hmac is in file 'botan/mac/hmac.d' 
which cannot be read

import path[0] = .
import path[1] = /usr/include/dlang/dmd
Failed: ["dmd", "-v", "-o-", "./app.d", "-I."]

What am i doing wrong? How to add library with dub package 
manager? Im really beginner in D (basically in programming) sorry 
if im asking obvious questions.


According to my other question are that functions equivalent to 
that python one? If they are not doing exact that same i could 
not create auth string which is needed to connect to AWS api.