Re: [RFC PATCH v2 0/7] ndctl: nvdimmd: notify/monitor the feathers of over threshold event
On Thu, Dec 7, 2017 at 1:58 AM, Qi, Fuliwrote: [..] >> Ok, I will make a sample hook up to Fluentd. >> >> > I tried to hook up the current json format output to Fluentd, > Fluentd parsed each line in the output as a new event. > The current json format output cannot be consumed by Fluentd without > any plugins. This is undesired. > So I changed the multiline json format output to one-line like the following > sample, then Fluentd parsed it well. > > {"datetime":"2017/11/28 > 11:00:12","pid":7013,"dev":"nmem1","health":{"health_state":"non-critical","temperature_celsius":23,"spares_percentage":75","life_used_percentage":5,"shutdown_state":"clean"}} > > I think one-line output will get consumed more easily by other data collect > services besides Fluentd. All the output items should be placed in json block, > such as datetime, eventid. > > So I would like to change current multi-line json format output to one line > and move all items into json block. > This is just a small matter of using the JSON_C_TO_STRING_PLAIN flag instead of JSON_C_TO_STRING_PRETTY when emitting json (json_object_to_json_string_ext()) for Fluentd or another service to consume. Also, when creating a json record for Fluentd I think the standard record that ndctl list reports should be wrapped as a sub-object i.e: { "dev":"nmem3", "id":"cdab-0a-07e0-feff", "handle":1, "phys_id":1, "health":{ "health_state":"non-critical", "temperature_celsius":23, "spares_percentage":75, "alarm_temperature":true, "alarm_spares":true, "temperature_threshold":80.125, "spares_threshold":128, "life_used_percentage":5, "shutdown_state":"clean" } } ...becomes: { "timestamp":"/MM/DD HH:mm:ss", "pid":1234, { "dev":"nmem3", "id":"cdab-0a-07e0-feff", "handle":1, "phys_id":1, "health":{ "health_state":"non-critical", "temperature_celsius":23, "spares_percentage":75, "alarm_temperature":true, "alarm_spares":true, "temperature_threshold":80.125, "spares_threshold":128, "life_used_percentage":5, "shutdown_state":"clean" } } ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v2 0/7] ndctl: nvdimmd: notify/monitor the feathers of over threshold event
On 2017/11/21 10:59, Qi, Fuli wrote: On 2017/11/21 1:13, Dan Williams wrote: On Mon, Nov 20, 2017 at 2:07 AM, Yasunori Gotowrote: On Mon, Nov 13, 2017 at 11:46 PM, QI Fuli wrote: Hi, here is my second version of nvdimm daemon, It would be appreciated if you could check it. Change log since v1: - Adding a config file(/etc/nvdimmd/nvdimmd.conf) - Using struct log_ctx instead of syslog() - Using log_syslog() to save the notify messages to syslog - Using log_file() to save the notify messages to special file - Adding LOG_NOTICE level into log_priority - Using automake instead of Makefile - Adding a new util file(nvdimmd/util.c) including helper functions needed for nvdimm daemon. - Adding nvdimmd_test program --- This is a patch set of nvdimmd, a tiny daemon to monitor the features of over threshold events. When an over thershold event fires, nvdimmd will output the notification including dimm health status to syslog or a special file to users' configuration. Users can choose the output format to be structured json or text. Here are out put samples. - json format: 2017/11/10 11:15:03 [28065] log_notify: nvdimm dimm over threshold notify { "dev":"nmem1", "id":"cdab-0a-07e0-feff", "handle":1, "phys_id":1, "health":{ "health_state":"non-critical", "temperature_celsius":23, "spares_percentage":75, "alarm_temperature":true, "alarm_spares":true, "temperature_threshold":40, "spares_threshold":5, "life_used_percentage":5, "shutdown_state":"clean" } } - text format: 2017/11/10 16:21:53 [12479] log_notify: nvdimm dimm over threshold notify dev: nmem1 health_state: non-critical spares_percentage: 75 Are these formats that services like Scribe and Fluent can consume? I'd like to see a sample hook up to those applications. I'm also wondering that since this will be a long running service that may gain more functionality over time, if we should write it in a language that does not have the security sharp edges of C. I've been looking for an excuse to use the Go language on a project, and this seems a good opportunity. Hm, This seems to be big change of direction of nvdimmd. Certainly, go-lang is good and interesting language in my impression. But, if nvdimmd is re-created by go-lang, I suppose nvdimmd will need to discard many functions of ndctl libraries, and need to re-create them by golang. (Though golang may be able to link C-lang libraries, then its security will be simlar with C-lang level, not go-lang level, I think.) Do you intent that we should re-create many functions for golang? In addition, do you include source code of go-lang nvdimmd in ndctl source code? Yes, but let's continue with C for now, I took a look and the impact is not quite as straightforward as I thought. The next step is to build a demonstration of Scribe or Fluentd consuming nvdimmd events. Ok, I will make a sample hook up to Fluentd. I tried to hook up the current json format output to Fluentd, Fluentd parsed each line in the output as a new event. The current json format output cannot be consumed by Fluentd without any plugins. This is undesired. So I changed the multiline json format output to one-line like the following sample, then Fluentd parsed it well. {"datetime":"2017/11/28 11:00:12","pid":7013,"dev":"nmem1","health":{"health_state":"non-critical","temperature_celsius":23,"spares_percentage":75","life_used_percentage":5,"shutdown_state":"clean"}} I think one-line output will get consumed more easily by other data collect services besides Fluentd. All the output items should be placed in json block, such as datetime, eventid. So I would like to change current multi-line json format output to one line and move all items into json block. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v2 0/7] ndctl: nvdimmd: notify/monitor the feathers of over threshold event
On 2017/11/21 1:13, Dan Williams wrote: On Mon, Nov 20, 2017 at 2:07 AM, Yasunori Gotowrote: On Mon, Nov 13, 2017 at 11:46 PM, QI Fuli wrote: Hi, here is my second version of nvdimm daemon, It would be appreciated if you could check it. Change log since v1: - Adding a config file(/etc/nvdimmd/nvdimmd.conf) - Using struct log_ctx instead of syslog() - Using log_syslog() to save the notify messages to syslog - Using log_file() to save the notify messages to special file - Adding LOG_NOTICE level into log_priority - Using automake instead of Makefile - Adding a new util file(nvdimmd/util.c) including helper functions needed for nvdimm daemon. - Adding nvdimmd_test program --- This is a patch set of nvdimmd, a tiny daemon to monitor the features of over threshold events. When an over thershold event fires, nvdimmd will output the notification including dimm health status to syslog or a special file to users' configuration. Users can choose the output format to be structured json or text. Here are out put samples. - json format: 2017/11/10 11:15:03 [28065] log_notify: nvdimm dimm over threshold notify { "dev":"nmem1", "id":"cdab-0a-07e0-feff", "handle":1, "phys_id":1, "health":{ "health_state":"non-critical", "temperature_celsius":23, "spares_percentage":75, "alarm_temperature":true, "alarm_spares":true, "temperature_threshold":40, "spares_threshold":5, "life_used_percentage":5, "shutdown_state":"clean" } } - text format: 2017/11/10 16:21:53 [12479] log_notify: nvdimm dimm over threshold notify dev: nmem1 health_state: non-critical spares_percentage: 75 Are these formats that services like Scribe and Fluent can consume? I'd like to see a sample hook up to those applications. I'm also wondering that since this will be a long running service that may gain more functionality over time, if we should write it in a language that does not have the security sharp edges of C. I've been looking for an excuse to use the Go language on a project, and this seems a good opportunity. Hm, This seems to be big change of direction of nvdimmd. Certainly, go-lang is good and interesting language in my impression. But, if nvdimmd is re-created by go-lang, I suppose nvdimmd will need to discard many functions of ndctl libraries, and need to re-create them by golang. (Though golang may be able to link C-lang libraries, then its security will be simlar with C-lang level, not go-lang level, I think.) Do you intent that we should re-create many functions for golang? In addition, do you include source code of go-lang nvdimmd in ndctl source code? Yes, but let's continue with C for now, I took a look and the impact is not quite as straightforward as I thought. The next step is to build a demonstration of Scribe or Fluentd consuming nvdimmd events. Ok, I will make a sample hook up to Fluentd. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v2 0/7] ndctl: nvdimmd: notify/monitor the feathers of over threshold event
On Mon, Nov 20, 2017 at 2:07 AM, Yasunori Gotowrote: >> On Mon, Nov 13, 2017 at 11:46 PM, QI Fuli wrote: >> > Hi, here is my second version of nvdimm daemon, It would be appreciated >> > if you could check it. >> > >> > Change log since v1: >> > - Adding a config file(/etc/nvdimmd/nvdimmd.conf) >> > - Using struct log_ctx instead of syslog() >> >- Using log_syslog() to save the notify messages to syslog >> >- Using log_file() to save the notify messages to special file >> > - Adding LOG_NOTICE level into log_priority >> > - Using automake instead of Makefile >> > - Adding a new util file(nvdimmd/util.c) including helper functions needed >> >for nvdimm daemon. >> > - Adding nvdimmd_test program >> > >> > --- >> > This is a patch set of nvdimmd, a tiny daemon to monitor the features of >> > over >> > threshold events. When an over thershold event fires, nvdimmd will output >> > the >> > notification including dimm health status to syslog or a special file to >> > users' configuration. Users can choose the output format to be structured >> > json >> > or text. >> > >> > Here are out put samples. >> > - json format: >> > 2017/11/10 11:15:03 [28065] log_notify: nvdimm dimm over threshold notify >> > { >> > "dev":"nmem1", >> > "id":"cdab-0a-07e0-feff", >> > "handle":1, >> > "phys_id":1, >> > "health":{ >> > "health_state":"non-critical", >> > "temperature_celsius":23, >> > "spares_percentage":75, >> > "alarm_temperature":true, >> > "alarm_spares":true, >> > "temperature_threshold":40, >> > "spares_threshold":5, >> > "life_used_percentage":5, >> > "shutdown_state":"clean" >> > } >> > } >> > - text format: >> > 2017/11/10 16:21:53 [12479] log_notify: nvdimm dimm over threshold notify >> > dev: nmem1 >> > health_state: non-critical >> > spares_percentage: 75 >> >> Are these formats that services like Scribe and Fluent can consume? >> I'd like to see a sample hook up to those applications. >> >> I'm also wondering that since this will be a long running service that >> may gain more functionality over time, if we should write it in a >> language that does not have the security sharp edges of C. >> >> I've been looking for an excuse to use the Go language on a project, >> and this seems a good opportunity. > > Hm, This seems to be big change of direction of nvdimmd. > > Certainly, go-lang is good and interesting language in my impression. > But, if nvdimmd is re-created by go-lang, I suppose nvdimmd will need to > discard many functions of ndctl libraries, and need to re-create them by > golang. > > (Though golang may be able to link C-lang libraries, then its security > will be simlar with C-lang level, not go-lang level, I think.) > > Do you intent that we should re-create many functions for golang? > > In addition, do you include source code of go-lang nvdimmd in ndctl source > code? Yes, but let's continue with C for now, I took a look and the impact is not quite as straightforward as I thought. The next step is to build a demonstration of Scribe or Fluentd consuming nvdimmd events. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v2 0/7] ndctl: nvdimmd: notify/monitor the feathers of over threshold event
> On Mon, Nov 13, 2017 at 11:46 PM, QI Fuliwrote: > > Hi, here is my second version of nvdimm daemon, It would be appreciated > > if you could check it. > > > > Change log since v1: > > - Adding a config file(/etc/nvdimmd/nvdimmd.conf) > > - Using struct log_ctx instead of syslog() > >- Using log_syslog() to save the notify messages to syslog > >- Using log_file() to save the notify messages to special file > > - Adding LOG_NOTICE level into log_priority > > - Using automake instead of Makefile > > - Adding a new util file(nvdimmd/util.c) including helper functions needed > >for nvdimm daemon. > > - Adding nvdimmd_test program > > > > --- > > This is a patch set of nvdimmd, a tiny daemon to monitor the features of > > over > > threshold events. When an over thershold event fires, nvdimmd will output > > the > > notification including dimm health status to syslog or a special file to > > users' configuration. Users can choose the output format to be structured > > json > > or text. > > > > Here are out put samples. > > - json format: > > 2017/11/10 11:15:03 [28065] log_notify: nvdimm dimm over threshold notify > > { > > "dev":"nmem1", > > "id":"cdab-0a-07e0-feff", > > "handle":1, > > "phys_id":1, > > "health":{ > > "health_state":"non-critical", > > "temperature_celsius":23, > > "spares_percentage":75, > > "alarm_temperature":true, > > "alarm_spares":true, > > "temperature_threshold":40, > > "spares_threshold":5, > > "life_used_percentage":5, > > "shutdown_state":"clean" > > } > > } > > - text format: > > 2017/11/10 16:21:53 [12479] log_notify: nvdimm dimm over threshold notify > > dev: nmem1 > > health_state: non-critical > > spares_percentage: 75 > > Are these formats that services like Scribe and Fluent can consume? > I'd like to see a sample hook up to those applications. > > I'm also wondering that since this will be a long running service that > may gain more functionality over time, if we should write it in a > language that does not have the security sharp edges of C. > > I've been looking for an excuse to use the Go language on a project, > and this seems a good opportunity. Hm, This seems to be big change of direction of nvdimmd. Certainly, go-lang is good and interesting language in my impression. But, if nvdimmd is re-created by go-lang, I suppose nvdimmd will need to discard many functions of ndctl libraries, and need to re-create them by golang. (Though golang may be able to link C-lang libraries, then its security will be simlar with C-lang level, not go-lang level, I think.) Do you intent that we should re-create many functions for golang? In addition, do you include source code of go-lang nvdimmd in ndctl source code? Thanks, > > > > > TODO list: > > - The dimms to monitor should be filtered by namespace and region > > Yes, I'd like to make this equivalent to 'ndctl list' in terms filtering. > > > - Add more information into the notify message > > - Make nvdimmd_test an ndctl command or an option of ndctl inject-error > > Yes, the test events should originate from "ndctl inject-error". > ___ > Linux-nvdimm mailing list > Linux-nvdimm@lists.01.org > https://lists.01.org/mailman/listinfo/linux-nvdimm ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC PATCH v2 0/7] ndctl: nvdimmd: notify/monitor the feathers of over threshold event
On Mon, Nov 13, 2017 at 11:46 PM, QI Fuliwrote: > Hi, here is my second version of nvdimm daemon, It would be appreciated > if you could check it. > > Change log since v1: > - Adding a config file(/etc/nvdimmd/nvdimmd.conf) > - Using struct log_ctx instead of syslog() >- Using log_syslog() to save the notify messages to syslog >- Using log_file() to save the notify messages to special file > - Adding LOG_NOTICE level into log_priority > - Using automake instead of Makefile > - Adding a new util file(nvdimmd/util.c) including helper functions needed >for nvdimm daemon. > - Adding nvdimmd_test program > > --- > This is a patch set of nvdimmd, a tiny daemon to monitor the features of over > threshold events. When an over thershold event fires, nvdimmd will output the > notification including dimm health status to syslog or a special file to > users' configuration. Users can choose the output format to be structured json > or text. > > Here are out put samples. > - json format: > 2017/11/10 11:15:03 [28065] log_notify: nvdimm dimm over threshold notify > { > "dev":"nmem1", > "id":"cdab-0a-07e0-feff", > "handle":1, > "phys_id":1, > "health":{ > "health_state":"non-critical", > "temperature_celsius":23, > "spares_percentage":75, > "alarm_temperature":true, > "alarm_spares":true, > "temperature_threshold":40, > "spares_threshold":5, > "life_used_percentage":5, > "shutdown_state":"clean" > } > } > - text format: > 2017/11/10 16:21:53 [12479] log_notify: nvdimm dimm over threshold notify > dev: nmem1 > health_state: non-critical > spares_percentage: 75 Are these formats that services like Scribe and Fluent can consume? I'd like to see a sample hook up to those applications. I'm also wondering that since this will be a long running service that may gain more functionality over time, if we should write it in a language that does not have the security sharp edges of C. I've been looking for an excuse to use the Go language on a project, and this seems a good opportunity. > > TODO list: > - The dimms to monitor should be filtered by namespace and region Yes, I'd like to make this equivalent to 'ndctl list' in terms filtering. > - Add more information into the notify message > - Make nvdimmd_test an ndctl command or an option of ndctl inject-error Yes, the test events should originate from "ndctl inject-error". ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
Re: [RFC patch v2 0/7] ndctl: nvdimmd: notify/monitor the feathers of over threshold event
I am sorry that Imadea mistake encoding this mail. I havesent a new one. Please ignore this mail. ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
[RFC PATCH v2 0/7] ndctl: nvdimmd: notify/monitor the feathers of over threshold event
Hi, here is my second version of nvdimm daemon, It would be appreciated if you could check it. Change log since v1: - Adding a config file(/etc/nvdimmd/nvdimmd.conf) - Using struct log_ctx instead of syslog() - Using log_syslog() to save the notify messages to syslog - Using log_file() to save the notify messages to special file - Adding LOG_NOTICE level into log_priority - Using automake instead of Makefile - Adding a new util file(nvdimmd/util.c) including helper functions needed for nvdimm daemon. - Adding nvdimmd_test program --- This is a patch set of nvdimmd, a tiny daemon to monitor the features of over threshold events. When an over thershold event fires, nvdimmd will output the notification including dimm health status to syslog or a special file to users' configuration. Users can choose the output format to be structured json or text. Here are out put samples. - json format: 2017/11/10 11:15:03 [28065] log_notify: nvdimm dimm over threshold notify { "dev":"nmem1", "id":"cdab-0a-07e0-feff", "handle":1, "phys_id":1, "health":{ "health_state":"non-critical", "temperature_celsius":23, "spares_percentage":75, "alarm_temperature":true, "alarm_spares":true, "temperature_threshold":40, "spares_threshold":5, "life_used_percentage":5, "shutdown_state":"clean" } } - text format: 2017/11/10 16:21:53 [12479] log_notify: nvdimm dimm over threshold notify dev: nmem1 health_state: non-critical spares_percentage: 75 TODO list: - The dimms to monitor should be filtered by namespace and region - Add more information into the notify message - Make nvdimmd_test an ndctl command or an option of ndctl inject-error QI Fuli (7): ndctl: nvdimmd: add LOG_NOTICE level into log_priority ndctl: nvdimmd: add nvdimmd necessary util functions ndctl: nvdimmd: add nvdimmd necessary functions ndctl: nvdimmd: add body file of nvdimm daemon ndctl: nvdimmd: add nvdimmd config file ndctl: nvdimmd: add the unit file of systemd for nvdimmd service ndctl: nvdimmd: add a temporary test for nvdimm daemon Makefile.am | 2 +- configure.ac| 1 + nvdimmd/Makefile.am | 47 nvdimmd/libnvdimmd.c| 315 nvdimmd/libnvdimmd.h| 53 nvdimmd/nvdimmd.c | 112 + nvdimmd/nvdimmd.conf| 25 nvdimmd/nvdimmd.service | 7 ++ nvdimmd/nvdimmd_test.c | 142 ++ nvdimmd/util.c | 80 nvdimmd/util.h | 33 + util/log.c | 2 + util/log.h | 3 + 13 files changed, 821 insertions(+), 1 deletion(-) create mode 100644 nvdimmd/Makefile.am create mode 100644 nvdimmd/libnvdimmd.c create mode 100644 nvdimmd/libnvdimmd.h create mode 100644 nvdimmd/nvdimmd.c create mode 100644 nvdimmd/nvdimmd.conf create mode 100644 nvdimmd/nvdimmd.service create mode 100644 nvdimmd/nvdimmd_test.c create mode 100644 nvdimmd/util.c create mode 100644 nvdimmd/util.h -- 2.9.5 ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm
[RFC patch v2 0/7] ndctl: nvdimmd: notify/monitor the feathers of over threshold event
Hi, here is my second version of nvdimm daemon, It would be appreciated if you could check it. Change log since v1: - Adding a config file(/etc/nvdimmd/nvdimmd.conf) - Using struct log_ctx instead of syslog() - Using log_syslog() to save the notify messages to syslog - Using log_file() to save the notify messages to special file - Adding LOG_NOTICE level into log_priority - Using automake instead of Makefile - Adding a new util file(nvdimmd/util.c) including helper functions needed for nvdimm daemon. - Adding nvdimmd_test program --- This is a patch set of nvdimmd, a tiny daemon to monitor the features of over threshold events. When an over thershold event fires, nvdimmd will output the notification including dimm health status to syslog or a special file to users' configuration. Users can choose the output format to be structured json or text. Here are out put samples. - json format: 2017/11/10 11:15:03 [28065] log_notify: nvdimm dimm over threshold notify { "dev":"nmem1", "id":"cdab-0a-07e0-feff", "handle":1, "phys_id":1, "health":{ "health_state":"non-critical", "temperature_celsius":23, "spares_percentage":75, "alarm_temperature":true, "alarm_spares":true, "temperature_threshold":40, "spares_threshold":5, "life_used_percentage":5, "shutdown_state":"clean" } } - text format: 2017/11/10 16:21:53 [12479] log_notify: nvdimm dimm over threshold notify dev: nmem1 health_state: non-critical spares_percentage: 75 TODO list: - The dimms to monitor should be filtered by namespace and region - Add more information into the notify message - Make nvdimmd_test an ndctl command or an option of ndctl inject-error Makefile.am | 2 +- configure.ac | 1 + nvdimmd/Makefile.am | 47 nvdimmd/libnvdimmd.c | 315 nvdimmd/libnvdimmd.h | 53 nvdimmd/nvdimmd | 228 +++ nvdimmd/nvdimmd.c | 112 + nvdimmd/nvdimmd.conf | 25 nvdimmd/nvdimmd.service | 7 ++ nvdimmd/nvdimmd_test.c | 142 ++ nvdimmd/util.c | 80 nvdimmd/util.h | 33 + util/log.c | 2 + util/log.h | 3 + 14 files changed, 1049 insertions(+), 1 deletion(-) ___ Linux-nvdimm mailing list Linux-nvdimm@lists.01.org https://lists.01.org/mailman/listinfo/linux-nvdimm