Re: [U-Boot] dfu: dfu and UBI Volumes
On Wed, May 29, 2013 at 06:35:27AM +0200, Heiko Schocher wrote: Hello Tom, Am 28.05.2013 23:16, schrieb Tom Rini: On Tue, May 28, 2013 at 11:01:09PM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130528172309.GF5829@bill-the-cat you wrote: Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images. Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your I don't think so. To write a UBI volume on an existing UBI device, you would use the ubi write command. This translates into a call of ubi_volume_write(char *volume, void *buf, size_t size) which means the size must be known before you start writing; as far as I can tell ubi_volume_write() does not support incremental write operations of individual parts of a volume. OK. A very quick read of ubi_volume_write leads into the guts of the write being in drivers/mtd/ubi/upd.c and it feels like you could actually do the volume write in chunks with ubi_start_update() and ubi_more_update_data() (and maybe some other housekeeping bits). It might take a bit more work since it looks like looks like both functions rely on knowing the size at the start, but that's just to make sure the size will fit in the volume. Yes, I think so too ... seems some work, but not unsolveable ... Hmm.. is it possible to get the filesize over dfu on startup? (I hope so, as it makes no sense to transfer a file, if it does not fit in the partition ... ) There is not, but that's OK. Even if you think you had the room for the whole image you could find a new bad block and not fit, so it's just behaving like when we do a raw NAND write, get the data chunk, make sure we think we still have room, write if we do, error if we don't. -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hello Wolfgang, Am 28.05.2013 07:58, schrieb Wolfgang Denk: Dear Heiko, In message 51a42ccd.1020...@denx.de you wrote: I would imagine, but testing and implementation might show a better way, we do UBI as name ubi ubiN volume-name, ie: rootfs ubi ubi0 rootfs user ubi ubi0 user-data and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge. Yes, I think, thats the way to go ... I doubt that dfu_nand.c is the right place for this. What if I start using DFU on NOR flash? The decisions which device type (NAND, MMC, NOR, USB mass storage, ...) to habdle on one side, and which partition type / image type (raw, UBI volume, file, ...) on the other side are fully orthogonal to each other. They should be handled by independent code, and not one of them repeated for all implementations of the other. Yes, exactly ... bye, Heiko -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130527233735.GZ17119@bill-the-cat you wrote: Where exactly is this 8 MB limit coming into play? In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today. Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system? I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be. I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says I'm about to send you a blob of X bytes, so we cannot know at the start how much data is coming our way. Today we solve this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE. -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hi Tom, On May 28, 2013, at 6:01 PM, Tom Rini wrote: On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130527233735.GZ17119@bill-the-cat you wrote: Where exactly is this 8 MB limit coming into play? In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today. Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system? I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be. I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says I'm about to send you a blob of X bytes, so we cannot know at the start how much data is coming our way. Today we solve this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE. Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen. The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer. Upon start of a dfu transfer we can allocate the buffer, and do our thing. -- Tom Regards -- Pantelis ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
On Tue, May 28, 2013 at 07:55:03AM +0200, Wolfgang Denk wrote: Dear Heiko, In message 51a427a8.8090...@denx.de you wrote: Where exactly is this 8 MB limit coming into play? You find this in drivers/dfu/dfu.c: static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE) dfu_buf[DFU_DATA_BUF_SIZE]; Ah, so it is a DFU restriction! [snip] drivers/dfu/dfu_mmc.c use (another?, why?) buffer: static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE) dfu_file_buf[CONFIG_SYS_DFU_MAX_FILE_SIZE]; ... and use this buffer for not raw partitions ... and this buffer gets flushed, only if the complete file is transfered, as the README states: CONFIG_SYS_DFU_MAX_FILE_SIZE When updating files rather than the raw storage device, we use a static buffer to copy the file into and then write the buffer once we've been given the whole file. Define this to the maximum filesize (in bytes) for the buffer. Default is 4 MiB if undefined. This makes very little sense to me. Why do we need another (and even smaller) buffer when we already have one? Per my other email, the intention and implementation didn't quite match-up. The intent of the README should be (but isn't) reflected) in the code. And perhaps we can come up with something better than a big static allocation. Perhaps. -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
On Tue, May 28, 2013 at 08:24:06AM +0200, Heiko Schocher wrote: Hello Wolfgang, Am 28.05.2013 07:58, schrieb Wolfgang Denk: Dear Heiko, In message 51a42ccd.1020...@denx.de you wrote: I would imagine, but testing and implementation might show a better way, we do UBI as name ubi ubiN volume-name, ie: rootfs ubi ubi0 rootfs user ubi ubi0 user-data and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge. Yes, I think, thats the way to go ... I doubt that dfu_nand.c is the right place for this. What if I start using DFU on NOR flash? The decisions which device type (NAND, MMC, NOR, USB mass storage, ...) to habdle on one side, and which partition type / image type (raw, UBI volume, file, ...) on the other side are fully orthogonal to each other. They should be handled by independent code, and not one of them repeated for all implementations of the other. Yes, exactly ... We can see about what re-org makes sense once we've got another implementation here. We might be able to share the filesystem-based write code, but that in part might cause some screams since it'll depend on our continued (ab)use of run_command. The device-specific code does end up being the device-specific API part. Unless we're adding ubifs in addition to ubi volume support to DFU, we don't have real shared filesystem stuff, yet. -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Dear Pantelis Antoniou, On Tuesday, May 28, 2013 5:05:12 PM, Pantelis Antoniou wrote: Hi Tom, On May 28, 2013, at 6:01 PM, Tom Rini wrote: On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130527233735.GZ17119@bill-the-cat you wrote: Where exactly is this 8 MB limit coming into play? In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today. Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system? I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be. I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says I'm about to send you a blob of X bytes, so we cannot know at the start how much data is coming our way. Today we solve this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE. Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen. The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer. Upon start of a dfu transfer we can allocate the buffer, and do our thing. I don't know the details of the DFU implementation in U-Boot, but the specification leaves the choice between programming the firmware on-the-fly during the download, and later during the manifestation phase (or a mix of both). Hence, there is not need for a global firmware buffer if U-Boot goes for the on-the-fly programming strategy. The only buffer constraint would be wTransferSize (chosen by U-Boot for the control endpoint) in that case. See 7. Manifestation Phase on page 26 here: http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images. Best regards, Benoît ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hi Benoît On May 28, 2013, at 7:31 PM, Benoît Thébaudeau wrote: Dear Pantelis Antoniou, On Tuesday, May 28, 2013 5:05:12 PM, Pantelis Antoniou wrote: Hi Tom, On May 28, 2013, at 6:01 PM, Tom Rini wrote: On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130527233735.GZ17119@bill-the-cat you wrote: Where exactly is this 8 MB limit coming into play? In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today. Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system? I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be. I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says I'm about to send you a blob of X bytes, so we cannot know at the start how much data is coming our way. Today we solve this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE. Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen. The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer. Upon start of a dfu transfer we can allocate the buffer, and do our thing. I don't know the details of the DFU implementation in U-Boot, but the specification leaves the choice between programming the firmware on-the-fly during the download, and later during the manifestation phase (or a mix of both). Hence, there is not need for a global firmware buffer if U-Boot goes for the on-the-fly programming strategy. The only buffer constraint would be wTransferSize (chosen by U-Boot for the control endpoint) in that case. See 7. Manifestation Phase on page 26 here: http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf The problem is not DFU TBH, it's that since we don't have an option to append to a file, we have to have the whole file transferred in RAM and written in one go. The raw medium dfu methods in u-boot don't have a problem. Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images. It already happens for raw memory partitions, it's the UBI images being discussed. Best regards, Benoît Regards -- Pantelis ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hi Pantelis, On Tuesday, May 28, 2013 6:43:06 PM, Pantelis Antoniou wrote: Hi Benoît On May 28, 2013, at 7:31 PM, Benoît Thébaudeau wrote: Dear Pantelis Antoniou, On Tuesday, May 28, 2013 5:05:12 PM, Pantelis Antoniou wrote: Hi Tom, On May 28, 2013, at 6:01 PM, Tom Rini wrote: On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130527233735.GZ17119@bill-the-cat you wrote: Where exactly is this 8 MB limit coming into play? In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today. Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system? I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be. I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says I'm about to send you a blob of X bytes, so we cannot know at the start how much data is coming our way. Today we solve this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE. Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen. The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer. Upon start of a dfu transfer we can allocate the buffer, and do our thing. I don't know the details of the DFU implementation in U-Boot, but the specification leaves the choice between programming the firmware on-the-fly during the download, and later during the manifestation phase (or a mix of both). Hence, there is not need for a global firmware buffer if U-Boot goes for the on-the-fly programming strategy. The only buffer constraint would be wTransferSize (chosen by U-Boot for the control endpoint) in that case. See 7. Manifestation Phase on page 26 here: http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf The problem is not DFU TBH, it's that since we don't have an option to append to a file, we have to have the whole file transferred in RAM and written in one go. The raw medium dfu methods in u-boot don't have a problem. Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images. It already happens for raw memory partitions, it's the UBI images being discussed. But what does appending to a file has to do with programming a UBI image, which is a memory partition containing a whole file system? This is what I don't get in this discussion. Is it because of a restriction of the DFU API in U-Boot? Best regards, Benoît ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hi On May 28, 2013, at 7:43 PM, Benoît Thébaudeau wrote: Hi Pantelis, On Tuesday, May 28, 2013 6:43:06 PM, Pantelis Antoniou wrote: Hi Benoît On May 28, 2013, at 7:31 PM, Benoît Thébaudeau wrote: Dear Pantelis Antoniou, On Tuesday, May 28, 2013 5:05:12 PM, Pantelis Antoniou wrote: Hi Tom, On May 28, 2013, at 6:01 PM, Tom Rini wrote: On Tue, May 28, 2013 at 07:50:46AM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130527233735.GZ17119@bill-the-cat you wrote: Where exactly is this 8 MB limit coming into play? In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today. Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system? I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be. I thought I might not have been clear about where this limit comes from, after I sent the email. The problem we have, and this is only for writing to a filesystem (_not_ writing of a filesystem) is that we do not have the API for appending to files, only create/overwrite. So we must read the whole file into memory, and then write it out. The DFU protocol doesn't have (I would swear anyhow) a part where it says I'm about to send you a blob of X bytes, so we cannot know at the start how much data is coming our way. Today we solve this with a statically defined CONFIG_SYS_DFU_MAX_FILE_SIZE. Looking at things again, I think this is buggy right now in that we need to also whack DFU_DATA_BUF_SIZE to also be that same value. Going forward, we may be able to switch this to (and both of these are off the top of my head) a getenv to see how much space to malloc, or just making it a malloc and adding some compile-time check to ensure that the malloc area is at least as big as CONFIG_SYS_DFU_MAX_FILE_SIZE. Correct, the DFU protocol doesn't have a method to inform you before hand about the size of the transfer about to happen. The only possible solution I see at this point is to have an environment variable, i.e. dfubuf that controls the size of the buffer. Upon start of a dfu transfer we can allocate the buffer, and do our thing. I don't know the details of the DFU implementation in U-Boot, but the specification leaves the choice between programming the firmware on-the-fly during the download, and later during the manifestation phase (or a mix of both). Hence, there is not need for a global firmware buffer if U-Boot goes for the on-the-fly programming strategy. The only buffer constraint would be wTransferSize (chosen by U-Boot for the control endpoint) in that case. See 7. Manifestation Phase on page 26 here: http://www.usb.org/developers/devclass_docs/DFU_1.1.pdf The problem is not DFU TBH, it's that since we don't have an option to append to a file, we have to have the whole file transferred in RAM and written in one go. The raw medium dfu methods in u-boot don't have a problem. Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images. It already happens for raw memory partitions, it's the UBI images being discussed. But what does appending to a file has to do with programming a UBI image, which is a memory partition containing a whole file system? This is what I don't get in this discussion. Is it because of a restriction of the DFU API in U-Boot? Don't expect a discussion on a mailing list to stay on topic for long :) We sort of drifted from UBI to the fixed sized buffer. Best regards, Benoît Regards -- Pantelis ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
On Tue, May 28, 2013 at 06:31:41PM +0200, Beno??t Th??baudeau wrote: [snip] Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images. Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your kernel in UBI or ...) will depend on what the UBI API provides us today. Modifying files on UBIFS (say replacing the kernel that's in UBIFS rather than a UBI volume itself) is another wrinkle, due to a lack of filesystem append. -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Dear Tom, In message 20130528172309.GF5829@bill-the-cat you wrote: Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images. Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your I don't think so. To write a UBI volume on an existing UBI device, you would use the ubi write command. This translates into a call of ubi_volume_write(char *volume, void *buf, size_t size) which means the size must be known before you start writing; as far as I can tell ubi_volume_write() does not support incremental write operations of individual parts of a volume. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de Its always easier short term to pee in the pond than install a toilet - it's just not a good long term plan. - Alan Cox in 20100101145701.6432e...@lxorguk.ukuu.org.uk ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
On Tue, May 28, 2013 at 11:01:09PM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130528172309.GF5829@bill-the-cat you wrote: Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images. Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your I don't think so. To write a UBI volume on an existing UBI device, you would use the ubi write command. This translates into a call of ubi_volume_write(char *volume, void *buf, size_t size) which means the size must be known before you start writing; as far as I can tell ubi_volume_write() does not support incremental write operations of individual parts of a volume. OK. A very quick read of ubi_volume_write leads into the guts of the write being in drivers/mtd/ubi/upd.c and it feels like you could actually do the volume write in chunks with ubi_start_update() and ubi_more_update_data() (and maybe some other housekeeping bits). It might take a bit more work since it looks like looks like both functions rely on knowing the size at the start, but that's just to make sure the size will fit in the volume. -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hello Tom, Am 28.05.2013 23:16, schrieb Tom Rini: On Tue, May 28, 2013 at 11:01:09PM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130528172309.GF5829@bill-the-cat you wrote: Of course this can't yet apply to writing files on file systems since the current API in U-Boot misses the append feature, but this could be applied to program raw memory partitions, including UBI images. Correct. We can write a whole UBI image, today, of NAND size, regardless of DDR size. But modifying UBI volumes (so UBIFS or your I don't think so. To write a UBI volume on an existing UBI device, you would use the ubi write command. This translates into a call of ubi_volume_write(char *volume, void *buf, size_t size) which means the size must be known before you start writing; as far as I can tell ubi_volume_write() does not support incremental write operations of individual parts of a volume. OK. A very quick read of ubi_volume_write leads into the guts of the write being in drivers/mtd/ubi/upd.c and it feels like you could actually do the volume write in chunks with ubi_start_update() and ubi_more_update_data() (and maybe some other housekeeping bits). It might take a bit more work since it looks like looks like both functions rely on knowing the size at the start, but that's just to make sure the size will fit in the volume. Yes, I think so too ... seems some work, but not unsolveable ... Hmm.. is it possible to get the filesize over dfu on startup? (I hope so, as it makes no sense to transfer a file, if it does not fit in the partition ... ) bye, Heiko -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hi Heiko,
Hello Tom,
Am 24.05.2013 19:12, schrieb Tom Rini:
On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote:
Hi Heiko,
On May 24, 2013, at 7:39 PM, Heiko Schocher wrote:
Hello,
just digging in DFU support in U-Boot for an upcoming board
support based on an AM335x. This board support uses for example a
rootfs in an UBI Volume on a NAND flash, and this should be
updated with dfu ...
How To do this? Current state on this board is to erase the rootfs
mtd partition with a nand erase and write the new image using
dfu_nand.c ... which calls in the end nand_write ... which is ...
lets say ... not the prefered way on an UBI volume ...
How to solve this? Any ideas?
Well, what would you like ideally to do? Why is nand_write not
ideal for a UBI volume.
Note that dfu will skip over the bad blocks...
Presumably because they want to replace say ubi0:rootfs (and leave
ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than
write in a new ubi container of everything.
I would suggest that, so long as our existing UBI infrastructure
allows this, you add a new method, dfu_ubi which takes care of
programming things. This shouldn't be too bad to write as I've
heard the existing infrastucture was easily expanded for SPI (and
patches are pending a little more clean up prior to posting).
This sounds easy ... but they have also raw nand partitions, for
example spl partitions on one nand flash ... for which dfu_nand.c
fits perfectly ... is it possible to use dfu_nand.c and another
dfu_xxx.c at the same time?
I'm not so familiar with nand devices handling, but in my opinion we
shall create dfu_ubi.c file.
I think that, nand part of dfu handling shall be separated from ubi,
even if UBI itself is layed on nand.
However, Tom and Pantelis shall give their opinion, since they spent a
lot of time on forcing dfu_nand.c to work.
I would say no, if I understand the code right:
- starting dfu with dfu interface={nand or mmc} ...
and after that, you can not switch anymore between nand or mmc, right?
You must press Ctrl-C to end dfu and start it again for switching
to another interface ...
Or should I look to add ubi support to drivers/dfu/dfu_nand.c?
Something like:
Add in dfu_fill_entity_nand a else if (!strcmp(st, ubipart)) {
(Did not looked deeper in this, if this is a possible way... and
thinking about it ... it is not the correct way, ubi should be
seperate, because it maybe runs also on nor flash ...)
So, the best way would be, to switch in dfu nand mode between
subinterfaces ... like raw, part, ubi, ...
Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu-layout
for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...
After all ... should we add a DFU_UBI and add this to
drivers/dfu/dfu_nand.c?
bye,
Heiko
--
Best regards,
Lukasz Majewski
Samsung RD Poland (SRPOL) | Linux Platform Group
___
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hello Lukasz, Am 27.05.2013 09:02, schrieb Lukasz Majewski: Hi Heiko, Hello Tom, Am 24.05.2013 19:12, schrieb Tom Rini: On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote: Hi Heiko, On May 24, 2013, at 7:39 PM, Heiko Schocher wrote: Hello, just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ... How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ... How to solve this? Any ideas? Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume. Note that dfu will skip over the bad blocks... Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything. I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting). This sounds easy ... but they have also raw nand partitions, for example spl partitions on one nand flash ... for which dfu_nand.c fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c at the same time? I'm not so familiar with nand devices handling, but in my opinion we shall create dfu_ubi.c file. I think that, nand part of dfu handling shall be separated from ubi, even if UBI itself is layed on nand. Yes, I tend also to this .. but not as a separte dfu interface .. dfu nand .. should stay, and each partition should know, if it is a raw nand, or UBI, or jffs2?,... as like in dfu_mmc.c ... but this should not be hardcoded in every dfu_xxx.c, instead it should be something like a subinterface ... if it is possible. However, Tom and Pantelis shall give their opinion, since they spent a lot of time on forcing dfu_nand.c to work. Yep, that would be great, as I am a dfu beginner ;-) bye, Heiko -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hi Guys, On May 27, 2013, at 10:28 AM, Heiko Schocher wrote: Hello Lukasz, Am 27.05.2013 09:02, schrieb Lukasz Majewski: Hi Heiko, Hello Tom, Am 24.05.2013 19:12, schrieb Tom Rini: On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote: Hi Heiko, On May 24, 2013, at 7:39 PM, Heiko Schocher wrote: Hello, just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ... How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ... How to solve this? Any ideas? Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume. Note that dfu will skip over the bad blocks... Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything. I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting). This sounds easy ... but they have also raw nand partitions, for example spl partitions on one nand flash ... for which dfu_nand.c fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c at the same time? I'm not so familiar with nand devices handling, but in my opinion we shall create dfu_ubi.c file. I think that, nand part of dfu handling shall be separated from ubi, even if UBI itself is layed on nand. Yes, I tend also to this .. but not as a separte dfu interface .. dfu nand .. should stay, and each partition should know, if it is a raw nand, or UBI, or jffs2?,... as like in dfu_mmc.c ... but this should not be hardcoded in every dfu_xxx.c, instead it should be something like a subinterface ... if it is possible. I'm not completely up to speed with UBI, but dfu_ubi seems to be the way to go for me. Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters). I don't know how smart you have to be with UBI version; be very careful when the binary format of UBI changes. However, Tom and Pantelis shall give their opinion, since they spent a lot of time on forcing dfu_nand.c to work. Yep, that would be great, as I am a dfu beginner ;-) bye, Heiko -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Regards -- Pantelis ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hello, Am 27.05.2013 09:35, schrieb Pantelis Antoniou: Hi Guys, On May 27, 2013, at 10:28 AM, Heiko Schocher wrote: Hello Lukasz, Am 27.05.2013 09:02, schrieb Lukasz Majewski: Hi Heiko, Hello Tom, Am 24.05.2013 19:12, schrieb Tom Rini: On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote: Hi Heiko, On May 24, 2013, at 7:39 PM, Heiko Schocher wrote: Hello, just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ... How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ... How to solve this? Any ideas? Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume. Note that dfu will skip over the bad blocks... Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything. I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting). This sounds easy ... but they have also raw nand partitions, for example spl partitions on one nand flash ... for which dfu_nand.c fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c at the same time? I'm not so familiar with nand devices handling, but in my opinion we shall create dfu_ubi.c file. I think that, nand part of dfu handling shall be separated from ubi, even if UBI itself is layed on nand. Yes, I tend also to this .. but not as a separte dfu interface .. dfu nand .. should stay, and each partition should know, if it is a raw nand, or UBI, or jffs2?,... as like in dfu_mmc.c ... but this should not be hardcoded in every dfu_xxx.c, instead it should be something like a subinterface ... if it is possible. I'm not completely up to speed with UBI, but dfu_ubi seems to be the way to go for me. But how to handle a raw nand partition and a ubi partition on one nand? If ubi is a new dfu interface, somebody must start dfu on u-boot with dfu nand .. or dfu ubi .. dependent on which partition has to be updated ... before using dfu-util on the host side ... and start dfu-util for the correct partition... This seems not really userfriendly to me ... if I have to use the u-boot shell, why using dfu-util on a host pc instead executing tftp and update my images within u-boot? Is ubi really a interface as nand or mmc ... ? Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters). Yep, or load the complete image in ram, and write it with ubi write ... I don't know how smart you have to be with UBI version; be very careful when the binary format of UBI changes. Thanks! bye, Heiko -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Dear Heiko Schocher, In message 51a30f34.7030...@denx.de you wrote: But how to handle a raw nand partition and a ubi partition on one nand? If ubi is a new dfu interface, somebody must start dfu on u-boot with dfu nand .. or dfu ubi .. dependent on which partition has to be updated ... before using dfu-util on the host side ... and start dfu-util for the correct partition... This seems not really userfriendly to me ... if I have to use the Indeed, this makes no sense and breaks the whole concept of DFU to be able to download a sinlge firmware image with one, simple command. Is ubi really a interface as nand or mmc ... ? No, it is not. It could be considered a partition type at best. Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters). Yep, or load the complete image in ram, and write it with ubi write ... Not or. When dealing with UBI volumes, then ubi write (or the equivalent C API) is the way to go. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de All a hacker needs is a tight PUSHJ, a loose pair of UUOs, and a warm place to shift. ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hi On May 27, 2013, at 7:25 PM, Wolfgang Denk wrote: Dear Heiko Schocher, In message 51a30f34.7030...@denx.de you wrote: But how to handle a raw nand partition and a ubi partition on one nand? If ubi is a new dfu interface, somebody must start dfu on u-boot with dfu nand .. or dfu ubi .. dependent on which partition has to be updated ... before using dfu-util on the host side ... and start dfu-util for the correct partition... This seems not really userfriendly to me ... if I have to use the Indeed, this makes no sense and breaks the whole concept of DFU to be able to download a sinlge firmware image with one, simple command. Is ubi really a interface as nand or mmc ... ? No, it is not. It could be considered a partition type at best. Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters). Yep, or load the complete image in ram, and write it with ubi write ... Not or. When dealing with UBI volumes, then ubi write (or the equivalent C API) is the way to go. I pretty much agree. UBI looks like it's partition type. BTW, the whole point of DFU is not to store your image in RAM at all. There are very few systems that have that much RAM. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de All a hacker needs is a tight PUSHJ, a loose pair of UUOs, and a warm place to shift. Regards -- Pantelis ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
On Sun, May 26, 2013 at 09:09:22AM +0200, Heiko Schocher wrote: [snip] Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu-layout for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4... After all ... should we add a DFU_UBI and add this to drivers/dfu/dfu_nand.c? Yes, I think we should be able to adapt dfu_nand to support raw (current) and UBI (which will need a little further handling so you can update per UBI container). In MMC (and there's examples in trats and am335x_evm) we say name ext4/fat/part/raw device#/start_blk part#/end_blk. I would imagine, but testing and implementation might show a better way, we do UBI as name ubi ubiN volume-name, ie: rootfs ubi ubi0 rootfs user ubi ubi0 user-data and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge. -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
On Mon, May 27, 2013 at 07:29:02PM +0300, Pantelis Antoniou wrote: Hi On May 27, 2013, at 7:25 PM, Wolfgang Denk wrote: Dear Heiko Schocher, In message 51a30f34.7030...@denx.de you wrote: But how to handle a raw nand partition and a ubi partition on one nand? If ubi is a new dfu interface, somebody must start dfu on u-boot with dfu nand .. or dfu ubi .. dependent on which partition has to be updated ... before using dfu-util on the host side ... and start dfu-util for the correct partition... This seems not really userfriendly to me ... if I have to use the Indeed, this makes no sense and breaks the whole concept of DFU to be able to download a sinlge firmware image with one, simple command. Is ubi really a interface as nand or mmc ... ? No, it is not. It could be considered a partition type at best. Looks like it's simple enough; erase (but don't step over the wear counters) , write (but skip over the wear counters). Yep, or load the complete image in ram, and write it with ubi write ... Not or. When dealing with UBI volumes, then ubi write (or the equivalent C API) is the way to go. I pretty much agree. UBI looks like it's partition type. BTW, the whole point of DFU is not to store your image in RAM at all. There are very few systems that have that much RAM. This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know. -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Dear Tom, In message 20130527204127.GY17119@bill-the-cat you wrote: This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for Where exactly is this 8 MB limit coming into play? the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know. I thought the only size limitation for images we can load is available system RAM? Is this not the case? Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de PoB = Prisoner of Bill -- those held captive, unwillingly or other- wise, by the contemptible Microsoft monopoly. -- Tom Christiansen in 6abo45$3lc$2...@csnews.cs.colorado.edu ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
On Mon, May 27, 2013 at 11:25:52PM +0200, Wolfgang Denk wrote: Dear Tom, In message 20130527204127.GY17119@bill-the-cat you wrote: This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for Where exactly is this 8 MB limit coming into play? In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today. the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know. I thought the only size limitation for images we can load is available system RAM? Is this not the case? It is the case for raw media and not the case for filesystems. If it will be the case for UBI depends on what API we have available to us in the UBI layer. -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hello Wolfgang, Am 27.05.2013 23:25, schrieb Wolfgang Denk: Dear Tom, In message 20130527204127.GY17119@bill-the-cat you wrote: This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for Where exactly is this 8 MB limit coming into play? You find this in drivers/dfu/dfu.c: static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE) dfu_buf[DFU_DATA_BUF_SIZE]; I had just a patch in my workqueue, where I made this value at least configurable ... but this limits not the transfer file size, see below... the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know. I thought the only size limitation for images we can load is available system RAM? Is this not the case? As I read in code: Raw nand partitions fill the above buffer, if it is full - flush to nand, and go on ... so no limit for them. drivers/dfu/dfu_mmc.c use (another?, why?) buffer: static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE) dfu_file_buf[CONFIG_SYS_DFU_MAX_FILE_SIZE]; and use this buffer for not raw partitions ... and this buffer gets flushed, only if the complete file is transfered, as the README states: CONFIG_SYS_DFU_MAX_FILE_SIZE When updating files rather than the raw storage device, we use a static buffer to copy the file into and then write the buffer once we've been given the whole file. Define this to the maximum filesize (in bytes) for the buffer. Default is 4 MiB if undefined. bye, Heiko -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hello Tom, Am 27.05.2013 21:21, schrieb Tom Rini: On Sun, May 26, 2013 at 09:09:22AM +0200, Heiko Schocher wrote: [snip] Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu-layout for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4... After all ... should we add a DFU_UBI and add this to drivers/dfu/dfu_nand.c? Yes, I think we should be able to adapt dfu_nand to support raw (current) and UBI (which will need a little further handling so you can update per UBI container). In MMC (and there's examples in trats and am335x_evm) we say name ext4/fat/part/raw device#/start_blk part#/end_blk. I would imagine, but testing and implementation might show a better way, we do UBI as name ubi ubiN volume-name, ie: rootfs ubi ubi0 rootfs user ubi ubi0 user-data and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge. Yes, I think, thats the way to go ... bye, Heiko -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hello Tom, Am 27.05.2013 22:41, schrieb Tom Rini: On Mon, May 27, 2013 at 07:29:02PM +0300, Pantelis Antoniou wrote: Hi On May 27, 2013, at 7:25 PM, Wolfgang Denk wrote: Dear Heiko Schocher, In message 51a30f34.7030...@denx.de you wrote: [...] I pretty much agree. UBI looks like it's partition type. BTW, the whole point of DFU is not to store your image in RAM at all. There are very few systems that have that much RAM. This _may_ be the hard part for UBI. When doing raw block writes for NAND/MMC, we're able to write them out quickly and thus support images larger than RAM. But for filesystems we don't support that notion in general for write and so limit ourselves to 8MiB or so files. Fine for the most part, but not fine for UBI. It's possible that we can support this on UBI easier than we can on filesystems, but I just don't know. I also do not know this at the moment :-( Maybe a UBI expert can help here? bye, Heiko -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Dear Tom, In message 20130527233735.GZ17119@bill-the-cat you wrote: Where exactly is this 8 MB limit coming into play? In buffering the data. We cannot write a chunk of a file to a filesystem and then append to it, we don't have the API today. Sorry, I still don't get it. Assuming I have a GiB of RAM, why can I not load a 256 MiB file to RAM, and then write it to a file system? I have definitely dealt with images and files bigger than 8 MiB in thepast, so I really don't see where any buffer problem could be. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de No question is too silly to ask. Of course, some questions are too silly to to answer... - L. Wall R. L. Schwartz, _Programming Perl_ ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Dear Heiko, In message 51a427a8.8090...@denx.de you wrote: Where exactly is this 8 MB limit coming into play? You find this in drivers/dfu/dfu.c: static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE) dfu_buf[DFU_DATA_BUF_SIZE]; Ah, so it is a DFU restriction! Well, this is a design problem that needs to be fixed. As I read in code: Raw nand partitions fill the above buffer, if it is full - flush to nand, and go on ... so no limit for them. So this boils down to the question whether there is some incremental ubi write method. Hello UBI experts? drivers/dfu/dfu_mmc.c use (another?, why?) buffer: static unsigned char __aligned(CONFIG_SYS_CACHELINE_SIZE) dfu_file_buf[CONFIG_SYS_DFU_MAX_FILE_SIZE]; ... and use this buffer for not raw partitions ... and this buffer gets flushed, only if the complete file is transfered, as the README states: CONFIG_SYS_DFU_MAX_FILE_SIZE When updating files rather than the raw storage device, we use a static buffer to copy the file into and then write the buffer once we've been given the whole file. Define this to the maximum filesize (in bytes) for the buffer. Default is 4 MiB if undefined. This makes very little sense to me. Why do we need another (and even smaller) buffer when we already have one? Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de We don't have to protect the environment -- the Second Coming is at hand. - James Watt ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Dear Heiko, In message 51a42ccd.1020...@denx.de you wrote: I would imagine, but testing and implementation might show a better way, we do UBI as name ubi ubiN volume-name, ie: rootfs ubi ubi0 rootfs user ubi ubi0 user-data and so forth, and augment dfu_nand.c with UBI knowledge, ala dfu_mmc and fat/ext knowledge. Yes, I think, thats the way to go ... I doubt that dfu_nand.c is the right place for this. What if I start using DFU on NOR flash? The decisions which device type (NAND, MMC, NOR, USB mass storage, ...) to habdle on one side, and which partition type / image type (raw, UBI volume, file, ...) on the other side are fully orthogonal to each other. They should be handled by independent code, and not one of them repeated for all implementations of the other. Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de A mouse is an elephant built by the Japanese. ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hello Tom,
Am 24.05.2013 19:12, schrieb Tom Rini:
On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote:
Hi Heiko,
On May 24, 2013, at 7:39 PM, Heiko Schocher wrote:
Hello,
just digging in DFU support in U-Boot for an upcoming board support
based on an AM335x. This board support uses for example a rootfs in
an UBI Volume on a NAND flash, and this should be updated with dfu ...
How To do this? Current state on this board is to erase the rootfs
mtd partition with a nand erase and write the new image using
dfu_nand.c ... which calls in the end nand_write ... which is ...
lets say ... not the prefered way on an UBI volume ...
How to solve this? Any ideas?
Well, what would you like ideally to do? Why is nand_write not ideal for
a UBI volume.
Note that dfu will skip over the bad blocks...
Presumably because they want to replace say ubi0:rootfs (and leave
ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write
in a new ubi container of everything.
I would suggest that, so long as our existing UBI infrastructure allows
this, you add a new method, dfu_ubi which takes care of programming
things. This shouldn't be too bad to write as I've heard the existing
infrastucture was easily expanded for SPI (and patches are pending a
little more clean up prior to posting).
This sounds easy ... but they have also raw nand partitions, for
example spl partitions on one nand flash ... for which dfu_nand.c
fits perfectly ... is it possible to use dfu_nand.c and another dfu_xxx.c
at the same time?
I would say no, if I understand the code right:
- starting dfu with dfu interface={nand or mmc} ...
and after that, you can not switch anymore between nand or mmc, right?
You must press Ctrl-C to end dfu and start it again for switching
to another interface ...
Or should I look to add ubi support to drivers/dfu/dfu_nand.c?
Something like:
Add in dfu_fill_entity_nand a else if (!strcmp(st, ubipart)) {
(Did not looked deeper in this, if this is a possible way... and
thinking about it ... it is not the correct way, ubi should be
seperate, because it maybe runs also on nor flash ...)
So, the best way would be, to switch in dfu nand mode between
subinterfaces ... like raw, part, ubi, ...
Ah, looking in drivers/dfu/dfu_mmc.c, they use dfu-layout
for switching between DFU_RAW_ADDR, DFU_FS_FAT, DFU_FS_EXT4...
After all ... should we add a DFU_UBI and add this to
drivers/dfu/dfu_nand.c?
bye,
Heiko
--
DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
___
U-Boot mailing list
U-Boot@lists.denx.de
http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Hi Heiko, On May 24, 2013, at 7:39 PM, Heiko Schocher wrote: Hello, just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ... How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ... How to solve this? Any ideas? Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume. Note that dfu will skip over the bad blocks... bye, Heiko -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Regards -- Pantelis ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
On Fri, May 24, 2013 at 07:42:01PM +0300, Pantelis Antoniou wrote: Hi Heiko, On May 24, 2013, at 7:39 PM, Heiko Schocher wrote: Hello, just digging in DFU support in U-Boot for an upcoming board support based on an AM335x. This board support uses for example a rootfs in an UBI Volume on a NAND flash, and this should be updated with dfu ... How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ... How to solve this? Any ideas? Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume. Note that dfu will skip over the bad blocks... Presumably because they want to replace say ubi0:rootfs (and leave ubi0:user-data and ubi0:u-boot-env and so forth alone) rather than write in a new ubi container of everything. I would suggest that, so long as our existing UBI infrastructure allows this, you add a new method, dfu_ubi which takes care of programming things. This shouldn't be too bad to write as I've heard the existing infrastucture was easily expanded for SPI (and patches are pending a little more clean up prior to posting). -- Tom signature.asc Description: Digital signature ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
Re: [U-Boot] dfu: dfu and UBI Volumes
Dear Pantelis Antoniou, In message 6ad958cb-3cfc-4362-b72d-511147d04...@antoniou-consulting.com you wrote: How To do this? Current state on this board is to erase the rootfs mtd partition with a nand erase and write the new image using dfu_nand.c ... which calls in the end nand_write ... which is ... lets say ... not the prefered way on an UBI volume ... How to solve this? Any ideas? Well, what would you like ideally to do? Why is nand_write not ideal for a UBI volume. The key problem is that nand_erase will destroy all UBI wear levelling and erase counters. When working with UBI, you NEVER want to do this. See for example [1] for reference. Note that dfu will skip over the bad blocks... Which is even worse here. DFU should be able to deal with UBI volumes, and using proper access routines (ubi write) to write the new volume(s). But never ever should any low level erase of the underlying flash device be needed nor used. [1] http://www.linux-mtd.infradead.org/doc/ubi.html#L_format Best regards, Wolfgang Denk -- DENX Software Engineering GmbH, MD: Wolfgang Denk Detlev Zundel HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-10 Fax: (+49)-8142-66989-80 Email: w...@denx.de What's the sound a name makes when it's dropped? ___ U-Boot mailing list U-Boot@lists.denx.de http://lists.denx.de/mailman/listinfo/u-boot
