Re: [go-nuts] Persistent memory support for Go
And no need to be sorry. You didn’t do anything wrong at all. > On Apr 3, 2019, at 6:32 PM, 'Mohit Verma' via golang-nuts > wrote: > > Hi Robert, > > Sorry for the confusion. Our changes are to make byte-addressable persistent > memory support available to Go community. You can find more information about > this technology here: > 1. http://pmem.io/ > 2. > https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html > > >>> I think the project needs to be distributed as a patch to the Go codebase > >>> - too much to review/maintain for security controls. > We eventually plan to distribute our changes as patches to Go codebase, but > before we do that we wanted to gather feedback here > > >>> Also, I’ve used pmem for a very long time (I have a bg in retail point of > >>> sale systems where it is very common) and we always had a driver/library > >>> on top of the physical device to multiplex across consumers, and we used > >>> it from Java, and never needed specialized JVM to use it efficiently. > >>> Again, just curious, maybe the use cases have changed. > The recent technological changes have put this technology close to the access > speeds of DRAM, while retaining the persistence property. Quoting from the > blog post in Jerrin’s original e-mail: > “Ad hoc libraries often provide a difficult programming interface. As memory > management for RAM has increasingly become a part of programming languages, > we believe managing persistent memory should be addressed in the same manner. > And so, our programming model includes changes to Go itself ”. > > There has been a lot of work around this technology in the past few years, > including efforts to change C and Java, and adhoc libraries for C/C++, Java > and Python. These changes are our efforts to make persistent memory > accessible to Go developers. > > Thanks! > Mohit > > From: Robert Engels > Date: Wednesday, April 3, 2019 at 4:00 PM > To: Jerrin Shaji George > Cc: "golang-nuts@googlegroups.com" , Mohit > Verma , Pratap Subrahmanyam , Rajesh > Venkatasubramanian > Subject: Re: [go-nuts] Persistent memory support for Go > > Also, I’ve used pmem for a very long time (I have a bg in retail point of > sale systems where it is very common) and we always had a driver/library on > top of the physical device to multiplex across consumers, and we used it from > Java, and never needed specialized JVM to use it efficiently. Again, just > curious, maybe the use cases have changed. > > On Apr 3, 2019, at 5:47 PM, Robert Engels wrote: > > I think the project needs to be distributed as a patch to the Go codebase - > too much to review/maintain for security controls. > > Also, I’m curious about the Api. In reviewing the example, it looks no > different than any ORM - just flatter -and these don’t need a customized > runtime. > > What would be the advantage of using the Api directly rather than using Redis > (with the support there)? > > No criticism, just not obvious to me. Maybe a link describing the persistent > memory capabilities in detail? > > On Apr 3, 2019, at 5:31 PM, 'Jerrin Shaji George' via golang-nuts > wrote: > > Hi, > > I am part of a small team at VMware working on projects related to persistent > memory (others in CC). We have recently been working on adding persistent > memory > support to the Go programming language, and I wanted to spread the word about > couple of these projects. > > 1) Go-pmem-transaction > The go-pmem-transaction project introduces a new programming model for > developing applications in Go for persistent memory. It consists of two > packages > - pmem and transaction. > > The pmem package provides methods to initialize persistent memory and an > interface to set and retrieve objects in persistent memory. The transaction > package provides undo and redo transaction logging APIs to support > crash-consistent updates to persistent memory data. > > Project page - https://github.com/vmware/go-pmem-transaction > > 2) Go-pmem > The Go-pmem project adds native persistent memory support to Go. > Some of the features of the persistent memory support added to Go are: > * Support for persistent memory allocations > * Garbage collector now collects objects from persistent heap > and volatile > heap > * Runtime automatically swizzles pointers if the memory > mapping address > changes on an application restart > * The persistent memory heap is dynamical
Re: [go-nuts] Persistent memory support for Go
Hi Robert, Sorry for the confusion. Our changes are to make byte-addressable persistent memory support available to Go community. You can find more information about this technology here: 1. http://pmem.io/ 2. https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html >>> I think the project needs to be distributed as a patch to the Go codebase - >>> too much to review/maintain for security controls. We eventually plan to distribute our changes as patches to Go codebase, but before we do that we wanted to gather feedback here >>> Also, I’ve used pmem for a very long time (I have a bg in retail point of >>> sale systems where it is very common) and we always had a driver/library on >>> top of the physical device to multiplex across consumers, and we used it >>> from Java, and never needed specialized JVM to use it efficiently. Again, >>> just curious, maybe the use cases have changed. The recent technological changes have put this technology close to the access speeds of DRAM, while retaining the persistence property. Quoting from the blog<https://blogs.vmware.com/opensource/2019/04/03/persistent-memory-with-go/> post in Jerrin’s original e-mail: “Ad hoc libraries often provide a difficult programming interface. As memory management for RAM has increasingly become a part of programming languages, we believe managing persistent memory should be addressed in the same manner. And so, our programming model includes changes to Go itself ”. There has been a lot of work around this technology in the past few years, including efforts to change C<https://csmd.ornl.gov/pub/nvl-c-static-analysis-techniques-efficient-correct-programming-non-volatile-main-memory-systems> and Java<https://arxiv.org/abs/1710.09968>, and adhoc libraries for C/C++<https://github.com/pmem/pmdk>, Java<https://github.com/pmem/pcj> and Python<https://github.com/pmem/pynvm>. These changes are our efforts to make persistent memory accessible to Go developers. Thanks! Mohit From: Robert Engels Date: Wednesday, April 3, 2019 at 4:00 PM To: Jerrin Shaji George Cc: "golang-nuts@googlegroups.com" , Mohit Verma , Pratap Subrahmanyam , Rajesh Venkatasubramanian Subject: Re: [go-nuts] Persistent memory support for Go Also, I’ve used pmem for a very long time (I have a bg in retail point of sale systems where it is very common) and we always had a driver/library on top of the physical device to multiplex across consumers, and we used it from Java, and never needed specialized JVM to use it efficiently. Again, just curious, maybe the use cases have changed. On Apr 3, 2019, at 5:47 PM, Robert Engels mailto:reng...@ix.netcom.com>> wrote: I think the project needs to be distributed as a patch to the Go codebase - too much to review/maintain for security controls. Also, I’m curious about the Api. In reviewing the example, it looks no different than any ORM - just flatter -and these don’t need a customized runtime. What would be the advantage of using the Api directly rather than using Redis (with the support there)? No criticism, just not obvious to me. Maybe a link describing the persistent memory capabilities in detail? On Apr 3, 2019, at 5:31 PM, 'Jerrin Shaji George' via golang-nuts mailto:golang-nuts@googlegroups.com>> wrote: Hi, I am part of a small team at VMware working on projects related to persistent memory (others in CC). We have recently been working on adding persistent memory support to the Go programming language, and I wanted to spread the word about couple of these projects. 1) Go-pmem-transaction The go-pmem-transaction project introduces a new programming model for developing applications in Go for persistent memory. It consists of two packages - pmem and transaction. The pmem package provides methods to initialize persistent memory and an interface to set and retrieve objects in persistent memory. The transaction package provides undo and redo transaction logging APIs to support crash-consistent updates to persistent memory data. Project page - https://github.com/vmware/go-pmem-transaction<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fvmware%2Fgo-pmem-transaction=02%7C01%7Cmohitv%40vmware.com%7C2670b6a89afd4161f26508d6b888349d%7Cb39138ca3cee4b4aa4d6cd83d9dd62f0%7C0%7C0%7C636899292469143173=OcqYdUEQYnxFQi8UgPPuScJVo%2BeeFL7BYJTP4uTy8Do%3D=0> 2) Go-pmem The Go-pmem project adds native persistent memory support to Go. Some of the features of the persistent memory support added to Go are: * Support for persistent memory allocations * Garbage collector now collects objects from persistent heap and volatile heap * Runtime automatically swizzles pointers if the memory mapping address
Re: [go-nuts] Persistent memory support for Go
Thanks for the info! > On Apr 3, 2019, at 6:32 PM, Mohit Verma wrote: > > Hi Robert, > > Sorry for the confusion. Our changes are to make byte-addressable persistent > memory support available to Go community. You can find more information about > this technology here: > 1. http://pmem.io/ > 2. > https://www.intel.com/content/www/us/en/architecture-and-technology/optane-dc-persistent-memory.html > > >>> I think the project needs to be distributed as a patch to the Go codebase > >>> - too much to review/maintain for security controls. > We eventually plan to distribute our changes as patches to Go codebase, but > before we do that we wanted to gather feedback here > > >>> Also, I’ve used pmem for a very long time (I have a bg in retail point of > >>> sale systems where it is very common) and we always had a driver/library > >>> on top of the physical device to multiplex across consumers, and we used > >>> it from Java, and never needed specialized JVM to use it efficiently. > >>> Again, just curious, maybe the use cases have changed. > The recent technological changes have put this technology close to the access > speeds of DRAM, while retaining the persistence property. Quoting from the > blog post in Jerrin’s original e-mail: > “Ad hoc libraries often provide a difficult programming interface. As memory > management for RAM has increasingly become a part of programming languages, > we believe managing persistent memory should be addressed in the same manner. > And so, our programming model includes changes to Go itself ”. > > There has been a lot of work around this technology in the past few years, > including efforts to change C and Java, and adhoc libraries for C/C++, Java > and Python. These changes are our efforts to make persistent memory > accessible to Go developers. > > Thanks! > Mohit > > From: Robert Engels > Date: Wednesday, April 3, 2019 at 4:00 PM > To: Jerrin Shaji George > Cc: "golang-nuts@googlegroups.com" , Mohit > Verma , Pratap Subrahmanyam , Rajesh > Venkatasubramanian > Subject: Re: [go-nuts] Persistent memory support for Go > > Also, I’ve used pmem for a very long time (I have a bg in retail point of > sale systems where it is very common) and we always had a driver/library on > top of the physical device to multiplex across consumers, and we used it from > Java, and never needed specialized JVM to use it efficiently. Again, just > curious, maybe the use cases have changed. > > On Apr 3, 2019, at 5:47 PM, Robert Engels wrote: > > I think the project needs to be distributed as a patch to the Go codebase - > too much to review/maintain for security controls. > > Also, I’m curious about the Api. In reviewing the example, it looks no > different than any ORM - just flatter -and these don’t need a customized > runtime. > > What would be the advantage of using the Api directly rather than using Redis > (with the support there)? > > No criticism, just not obvious to me. Maybe a link describing the persistent > memory capabilities in detail? > > On Apr 3, 2019, at 5:31 PM, 'Jerrin Shaji George' via golang-nuts > wrote: > > Hi, > > I am part of a small team at VMware working on projects related to persistent > memory (others in CC). We have recently been working on adding persistent > memory > support to the Go programming language, and I wanted to spread the word about > couple of these projects. > > 1) Go-pmem-transaction > The go-pmem-transaction project introduces a new programming model for > developing applications in Go for persistent memory. It consists of two > packages > - pmem and transaction. > > The pmem package provides methods to initialize persistent memory and an > interface to set and retrieve objects in persistent memory. The transaction > package provides undo and redo transaction logging APIs to support > crash-consistent updates to persistent memory data. > > Project page - https://github.com/vmware/go-pmem-transaction > > 2) Go-pmem > The Go-pmem project adds native persistent memory support to Go. > Some of the features of the persistent memory support added to Go are: > * Support for persistent memory allocations > * Garbage collector now collects objects from persistent heap > and volatile > heap > * Runtime automatically swizzles pointers if the memory > mapping address > changes on an application restart > * The persistent memory heap is dynamically sized and > supports automatic > heap growth dep
Re: [go-nuts] Persistent memory support for Go
Also, I’ve used pmem for a very long time (I have a bg in retail point of sale systems where it is very common) and we always had a driver/library on top of the physical device to multiplex across consumers, and we used it from Java, and never needed specialized JVM to use it efficiently. Again, just curious, maybe the use cases have changed. > On Apr 3, 2019, at 5:47 PM, Robert Engels wrote: > > I think the project needs to be distributed as a patch to the Go codebase - > too much to review/maintain for security controls. > > Also, I’m curious about the Api. In reviewing the example, it looks no > different than any ORM - just flatter -and these don’t need a customized > runtime. > > What would be the advantage of using the Api directly rather than using Redis > (with the support there)? > > No criticism, just not obvious to me. Maybe a link describing the persistent > memory capabilities in detail? > >> On Apr 3, 2019, at 5:31 PM, 'Jerrin Shaji George' via golang-nuts >> wrote: >> >> Hi, >> >> I am part of a small team at VMware working on projects related to persistent >> memory (others in CC). We have recently been working on adding persistent >> memory >> support to the Go programming language, and I wanted to spread the word about >> couple of these projects. >> >> 1) Go-pmem-transaction >> The go-pmem-transaction project introduces a new programming model for >> developing applications in Go for persistent memory. It consists of two >> packages >> - pmem and transaction. >> >> The pmem package provides methods to initialize persistent memory and an >> interface to set and retrieve objects in persistent memory. The transaction >> package provides undo and redo transaction logging APIs to support >> crash-consistent updates to persistent memory data. >> >> Project page - https://github.com/vmware/go-pmem-transaction >> >> 2) Go-pmem >> The Go-pmem project adds native persistent memory support to Go. >> Some of the features of the persistent memory support added to Go are: >> * Support for persistent memory allocations >> * Garbage collector now collects objects from persistent >> heap and volatile >> heap >> * Runtime automatically swizzles pointers if the memory >> mapping address >> changes on an application restart >> * The persistent memory heap is dynamically sized and >> supports automatic >> heap growth depending on memory demand >> >> Project page - https://github.com/jerrinsg/go-pmem >> >> The project pages contains links to further documentation. We welcome the >> community to try out these projects and send any feedback our way! >> >> Also see the blog post at >> https://blogs.vmware.com/opensource/2019/04/03/persistent-memory-with-go/ >> >> Thanks, >> Jerrin >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Persistent memory support for Go
Never mind on the “what is pmem” as I found the VMware docs. I am curious though why you need specialized runtime support rather than mmap on a block device? I am assuming that the api is slightly easier to use from a standard Go perspective but given the variable performance characteristics of pmem I don’t see why it shouldn’t be treated like any other external resource. > On Apr 3, 2019, at 5:47 PM, Robert Engels wrote: > > I think the project needs to be distributed as a patch to the Go codebase - > too much to review/maintain for security controls. > > Also, I’m curious about the Api. In reviewing the example, it looks no > different than any ORM - just flatter -and these don’t need a customized > runtime. > > What would be the advantage of using the Api directly rather than using Redis > (with the support there)? > > No criticism, just not obvious to me. Maybe a link describing the persistent > memory capabilities in detail? > >> On Apr 3, 2019, at 5:31 PM, 'Jerrin Shaji George' via golang-nuts >> wrote: >> >> Hi, >> >> I am part of a small team at VMware working on projects related to persistent >> memory (others in CC). We have recently been working on adding persistent >> memory >> support to the Go programming language, and I wanted to spread the word about >> couple of these projects. >> >> 1) Go-pmem-transaction >> The go-pmem-transaction project introduces a new programming model for >> developing applications in Go for persistent memory. It consists of two >> packages >> - pmem and transaction. >> >> The pmem package provides methods to initialize persistent memory and an >> interface to set and retrieve objects in persistent memory. The transaction >> package provides undo and redo transaction logging APIs to support >> crash-consistent updates to persistent memory data. >> >> Project page - https://github.com/vmware/go-pmem-transaction >> >> 2) Go-pmem >> The Go-pmem project adds native persistent memory support to Go. >> Some of the features of the persistent memory support added to Go are: >> * Support for persistent memory allocations >> * Garbage collector now collects objects from persistent >> heap and volatile >> heap >> * Runtime automatically swizzles pointers if the memory >> mapping address >> changes on an application restart >> * The persistent memory heap is dynamically sized and >> supports automatic >> heap growth depending on memory demand >> >> Project page - https://github.com/jerrinsg/go-pmem >> >> The project pages contains links to further documentation. We welcome the >> community to try out these projects and send any feedback our way! >> >> Also see the blog post at >> https://blogs.vmware.com/opensource/2019/04/03/persistent-memory-with-go/ >> >> Thanks, >> Jerrin >> -- >> You received this message because you are subscribed to the Google Groups >> "golang-nuts" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to golang-nuts+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: [go-nuts] Persistent memory support for Go
I think the project needs to be distributed as a patch to the Go codebase - too much to review/maintain for security controls. Also, I’m curious about the Api. In reviewing the example, it looks no different than any ORM - just flatter -and these don’t need a customized runtime. What would be the advantage of using the Api directly rather than using Redis (with the support there)? No criticism, just not obvious to me. Maybe a link describing the persistent memory capabilities in detail? > On Apr 3, 2019, at 5:31 PM, 'Jerrin Shaji George' via golang-nuts > wrote: > > Hi, > > I am part of a small team at VMware working on projects related to persistent > memory (others in CC). We have recently been working on adding persistent > memory > support to the Go programming language, and I wanted to spread the word about > couple of these projects. > > 1) Go-pmem-transaction > The go-pmem-transaction project introduces a new programming model for > developing applications in Go for persistent memory. It consists of two > packages > - pmem and transaction. > > The pmem package provides methods to initialize persistent memory and an > interface to set and retrieve objects in persistent memory. The transaction > package provides undo and redo transaction logging APIs to support > crash-consistent updates to persistent memory data. > > Project page - https://github.com/vmware/go-pmem-transaction > > 2) Go-pmem > The Go-pmem project adds native persistent memory support to Go. > Some of the features of the persistent memory support added to Go are: > * Support for persistent memory allocations > * Garbage collector now collects objects from persistent heap > and volatile > heap > * Runtime automatically swizzles pointers if the memory > mapping address > changes on an application restart > * The persistent memory heap is dynamically sized and > supports automatic > heap growth depending on memory demand > > Project page - https://github.com/jerrinsg/go-pmem > > The project pages contains links to further documentation. We welcome the > community to try out these projects and send any feedback our way! > > Also see the blog post at > https://blogs.vmware.com/opensource/2019/04/03/persistent-memory-with-go/ > > Thanks, > Jerrin > -- > You received this message because you are subscribed to the Google Groups > "golang-nuts" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to golang-nuts+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "golang-nuts" group. To unsubscribe from this group and stop receiving emails from it, send an email to golang-nuts+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
