[389-devel] 389 DS nightly 2019-05-16 - 92% PASS
https://fedorapeople.org/groups/389ds/ci/nightly/2019/05/16/report-389-ds-base-1.4.1.2-20190515git41c30fd.fc29.x86_64.html ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Logging future direction and ideas.
Replying to both Simon and Ludwig in the one mail ... To just comment to one point from Ludwig: > This may sound too negative, I agree that logging deserves attention and > improvement, but we need to agree on what we want to achieve. I don't think this is negative, at all and I think it's exactly the kind of feedback I wanted. :) So thank you! > Hi William, > I am more than interested! > > I'd like to learn it one day anyway. > So if there will be no objections and we'll go forward now, > I think, it is important to agree on some points of decreasing a bus factor: > > - As you said, it should have very detailed comments on all of the > language features and technics you'll use. > - I think it will be nice to have an additional documentation which will > describe the basic setup for the development you use. All the toold you > need to develop, compile, test and debug. > - Also, some nice links for the basic tutorials on Rust types, concepts, etc. > - I think, we should have detailed unit tests. It will help to > understand the code better and we will have less bugs (hopefully). So I'm writing a document for the wiki to cover this *and* rust supports tests as part of the system - that will be all be included. Is that acceptable? > > And the final and big point I wanted to mention: > > - We should be prepared for a slow review process because we have quite > limited > recources in the team and a lot of work should be done (WebUI still has to > be refactored to React, > and it is only a small piece of the workload). > Also, I think, it makes sense to have the smallest Rust PRs that can be put > together as an independent unit. That's totally reasonable. I'll try to keep it to small parts and will build up as we go. See below for ideas on how I'll work toward this. > > But everything is just my opinion and I don't know what others think and > if everyone is prepared to join it. I am feeling excited though :) > > Thanks, > Simon > > P.S. check the contribution guide please. Espesially a part about > 'rebase-force-push'. I think it is nice not to force push during > the review process (and rebase-squash only after you got an ACK). Yep, I saw this change. I'll keep it in mind :) > On 15 May 2019, at 19:37, Ludwig wrote: > > Hi William, > > this is a good starting point to discuss and decide how to move forward with > logging (although it looks like you are already some steps ahead). > > I have no strong ties to existing logging format and if we do it in rust or > not I don't care, but I do not want to use existing functionality for the > sake af a new unified approach. > > First we need to define what is the purpose and usage of our logs, what do we > want to keep or extend. I see these main area: > > - statistics > > - auditing (authentications, modifications) > > . debugging (for my usage the most important aspect) > > Next: what are the real problems > > - performance: yes, it would be nice to have less logging impact on > performance, but I haven't seen real complaints and doing it differently does > not guarantee better performance, we have debug levels (like tracing) which > are not usable because they slow down everything, I do not think this will be > resolved by just replacing the library > > - information: we continue to add information to the logs, and there are > still open requests > > > So: If I look at your suggestions I do like some and have concerns with > others, and I am not sure if the priorities are right. > > You list a couple of tickets related to logging, but forgot others, eg: > > https://pagure.io/389-ds-base/issue/47822 - improve logging of ACL decisions > > https://pagure.io/389-ds-base/issue/48680 - enable logging for 3rd party libs > > https://pagure.io/389-ds-base/issue/50002 - improve password policy logging > > https://pagure.io/389-ds-base/issue/50187 - improve replication logging These are all things that you you are correct, that I missed. So I want to focus on your point about what we want from logs, IE stats, auditing, debugging. I think this is a great summary of what we want, but to make it more detailed: * Stats about operations from start to finish * Auditing of client operations * Debugging of client operations * Debugging of internal server machinery and state * Reporting of errors outside of the normal operation system. Today we current have: access - auditing of operations at a highlevel audit - auditing of modifications and writes error - debugging of internal server machinery (we tend not to enable the levels of debugging operations ...) So I think there is room to improve. Now I think that performance so far is only a barrier in terms of preventing us from adding more detail, because it slows down operations. This is why it's pretty high on the list of "to fix". > > > these requests for improvements of logging exist for quite some time, they > were accepted as useful and good to
[389-devel] Re: Use of rust for new logging backend
Hi I will also help from QA team(Test and automation ) , i started learning "rust" 6 month ago (not regular ). I know 2% (may be). Regards Anuj Borah On Wed, May 15, 2019 at 8:20 PM Simon Pichugin wrote: > On Tue, May 14, 2019 at 01:21:28PM +1000, William Brown wrote: > > Hi all, > > > > So it's been a while since I pushed this, but again, I think I want to > bring this up. I'd like to write the new logging backend in Rust. > > > > I'll start with a summary: > > > > Pros: > > * Faster development time due to stricter code correctness guarantees > > * Modern and safe language to lower number of bugs in implementation > > * Much better libraries for interacting with things like json and thread > safety > > > > Cons: > > * Another language in the codebase ... > > * We need to learn it to work on it > > * Vendors need to be ready to build with the toolchain > > > > I've talked through some of these thoughts with Mark a bit, and I really > think it's time we did this - or gave up on pursuing it. I have been > running the rust enabled version of ds for a few years now with no issues. > SUSE is also in a position where they are able to and ready to build DS > with Rust (I'm assuming RH could easily also be brought to parity here). > > > > So the main barrier becomes us: the most common argument is we don't > know enough or have enough experience. However I think this argument is > flawed, in the fact that there are many parts of the code where we already > only have 1 or 2 experts in that area - often when we look into a bug or a > feature, we always have to learn new things, we have to read the code, > understand even the unique styles of how that was written for the time, and > then do work. This would be no different. I think we are all capable of > learning and working with these tools. > > > > By this point, Gnome is looking into Rust as a mainline part of the > desktop, Samba has started to look into it, Firefox is already shipping > Rust components. I think that Rust is here to stay, and is not some passing > trend. > > > > > > So what do we think? I think if there are no major objections I would > like to do this in Rust. I'll also commit to providing C-Rust FFI > documentation for the team, resources to help understand what's going on, > and like always, I will be sure to comment all my code thoroughly as part > of this improvement. > Hi William, > I am more than interested! > > I'd like to learn it one day anyway. > So if there will be no objections and we'll go forward now, > I think, it is important to agree on some points of decreasing a bus > factor: > > - As you said, it should have very detailed comments on all of the > language features and technics you'll use. > - I think it will be nice to have an additional documentation which will > describe the basic setup for the development you use. All the toold you > need to develop, compile, test and debug. > - Also, some nice links for the basic tutorials on Rust types, concepts, > etc. > - I think, we should have detailed unit tests. It will help to > understand the code better and we will have less bugs (hopefully). > > And the final and big point I wanted to mention: > > - We should be prepared for a slow review process because we have quite > limited > recources in the team and a lot of work should be done (WebUI still has > to be refactored to React, > and it is only a small piece of the workload). > Also, I think, it makes sense to have the smallest Rust PRs that can be > put together as an independent unit. > > But everything is just my opinion and I don't know what others think and > if everyone is prepared to join it. I am feeling excited though :) > > Thanks, > Simon > > P.S. check the contribution guide please. Espesially a part about > 'rebase-force-push'. I think it is nice not to force push during > the review process (and rebase-squash only after you got an ACK). > > > > > > > > — > > Sincerely, > > > > William Brown > > > > Senior Software Engineer, 389 Directory Server > > SUSE Labs > > ___ > > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org > ___ 389-devel mailing list --
[389-devel] Re: Use of rust for new logging backend
On Tue, May 14, 2019 at 01:21:28PM +1000, William Brown wrote: > Hi all, > > So it's been a while since I pushed this, but again, I think I want to bring > this up. I'd like to write the new logging backend in Rust. > > I'll start with a summary: > > Pros: > * Faster development time due to stricter code correctness guarantees > * Modern and safe language to lower number of bugs in implementation > * Much better libraries for interacting with things like json and thread > safety > > Cons: > * Another language in the codebase ... > * We need to learn it to work on it > * Vendors need to be ready to build with the toolchain > > I've talked through some of these thoughts with Mark a bit, and I really > think it's time we did this - or gave up on pursuing it. I have been running > the rust enabled version of ds for a few years now with no issues. SUSE is > also in a position where they are able to and ready to build DS with Rust > (I'm assuming RH could easily also be brought to parity here). > > So the main barrier becomes us: the most common argument is we don't know > enough or have enough experience. However I think this argument is flawed, in > the fact that there are many parts of the code where we already only have 1 > or 2 experts in that area - often when we look into a bug or a feature, we > always have to learn new things, we have to read the code, understand even > the unique styles of how that was written for the time, and then do work. > This would be no different. I think we are all capable of learning and > working with these tools. > > By this point, Gnome is looking into Rust as a mainline part of the desktop, > Samba has started to look into it, Firefox is already shipping Rust > components. I think that Rust is here to stay, and is not some passing trend. > > > So what do we think? I think if there are no major objections I would like to > do this in Rust. I'll also commit to providing C-Rust FFI documentation for > the team, resources to help understand what's going on, and like always, I > will be sure to comment all my code thoroughly as part of this improvement. Hi William, I am more than interested! I'd like to learn it one day anyway. So if there will be no objections and we'll go forward now, I think, it is important to agree on some points of decreasing a bus factor: - As you said, it should have very detailed comments on all of the language features and technics you'll use. - I think it will be nice to have an additional documentation which will describe the basic setup for the development you use. All the toold you need to develop, compile, test and debug. - Also, some nice links for the basic tutorials on Rust types, concepts, etc. - I think, we should have detailed unit tests. It will help to understand the code better and we will have less bugs (hopefully). And the final and big point I wanted to mention: - We should be prepared for a slow review process because we have quite limited recources in the team and a lot of work should be done (WebUI still has to be refactored to React, and it is only a small piece of the workload). Also, I think, it makes sense to have the smallest Rust PRs that can be put together as an independent unit. But everything is just my opinion and I don't know what others think and if everyone is prepared to join it. I am feeling excited though :) Thanks, Simon P.S. check the contribution guide please. Espesially a part about 'rebase-force-push'. I think it is nice not to force push during the review process (and rebase-squash only after you got an ACK). > > > — > Sincerely, > > William Brown > > Senior Software Engineer, 389 Directory Server > SUSE Labs > ___ > 389-devel mailing list -- 389-devel@lists.fedoraproject.org > To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org signature.asc Description: PGP signature ___ 389-devel mailing list -- 389-devel@lists.fedoraproject.org To unsubscribe send an email to 389-devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
[389-devel] Re: Logging future direction and ideas.
Hi William, this is a good starting point to discuss and decide how to move forward with logging (although it looks like you are already some steps ahead). I have no strong ties to existing logging format and if we do it in rust or not I don't care, but I do not want to use existing functionality for the sake af a new unified approach. First we need to define what is the purpose and usage of our logs, what do we want to keep or extend. I see these main area: - statistics - auditing (authentications, modifications) . debugging (for my usage the most important aspect) Next: what are the real problems - performance: yes, it would be nice to have less logging impact on performance, but I haven't seen real complaints and doing it differently does not guarantee better performance, we have debug levels (like tracing) which are not usable because they slow down everything, I do not think this will be resolved by just replacing the library - information: we continue to add information to the logs, and there are still open requests So: If I look at your suggestions I do like some and have concerns with others, and I am not sure if the priorities are right. You list a couple of tickets related to logging, but forgot others, eg: https://pagure.io/389-ds-base/issue/47822 - improve logging of ACL decisions https://pagure.io/389-ds-base/issue/48680 - enable logging for 3rd party libs https://pagure.io/389-ds-base/issue/50002 - improve password policy logging https://pagure.io/389-ds-base/issue/50187 - improve replication logging these requests for improvements of logging exist for quite some time, they were accepted as useful and good to have, but we didn't have the capacity to work on them. The work is tedious, going thru ACL or replication code and decide what and how to log better requires good knowledge of the areas and is not fancy - but I wonder if the effort wouldn't be spent better to improve the "what" and not the "how". Now: some comments on your suggestions. - perf: I think everything we can do to reduce locking in logging is a good thing, if we do logging more asynchronous it is ok for me as long as we can see the "real time" of the event and get the logs ordered. If replacing the "log" component by a rust implemention helps it is good. We could in a first step keep the slapi_log_* api and only do it more efficient - merging of logs: I do not like the idea to have everything in one log file, if you say we can provide tools to split, we can also provide tools to merge. The logs have a different format and for good reasons, eg the audit log is ldif like and can easily be used to replay changes, why force it into another format. Also, in my opinion, the different logs have a different life time. You might want to keep all audit logs, keep the error log for the main events for some longer time than the fast growing access log, defining log rotation based on the log type is a benfit in my opinion - operation orientation: there are logging events which are independent of operations, especially in replicatio. I would not want to artificially enforce a "operation context" eg for changelog processing, tombstone purging,. - format: for machines it might be good that logs are machine readable, but I did spend a lot of times reading logs and lokkinng for patterns and clues, I think as soon as they are in json I will stop doing this. This may sound too negative, I agree that logging deserves attention and improvement, but we need to agree on what we want to achieve. Best regards, Ludwig On 05/10/2019 05:13 AM, William Brown wrote: Hi all, So I think it's time for me to write some logging code to improve the situation. Relevant links before we start: https://pagure.io/389-ds-base/issue/49415 http://www.port389.org/docs/389ds/design/logging-performance-improvement.html https://pagure.io/389-ds-base/issue/50350 https://pagure.io/389-ds-base/issue/50361 All of these links touch on issues around logging, and I think they all combine to create three important points: * The performance of logging should be improved * The amount of details (fine grain) and information in logs should improve * The structure of the log content should be improved to aid interaction (possibly even machine parsable) I will turn this into a design document, but there are some questions I would like some input to help answer as part of this process to help set the direction and tasks to achieve. -- Should our logs as they exist today, continue to exist? I think that my view on this is "no". I think if we make something better, we have little need to continue to support our legacy interfaces. Of course, this would be a large change and it may not sit comfortably with people. A large part of this thinking is that the "new" log interface I want to add is focused on *operations* rather than auditing accesses or changes, or over looking at errors. The
