kangkaisen commented on issue #1235: Support SSL authentication with Kafka in routine load job URL: https://github.com/apache/incubator-doris/pull/1235#issuecomment-498502628 > @kangkaisen > `Save the small file content to FE metadata is necessary? could we save the small file content in FE disk or BE disk?` > > I've been considered this problem, here are my thoughts: > > 1. Limit the size and num of files so that FE memory is not a problem. > 2. If files be saved on disk, we may need another file manager to manage files on disks. If files are damaged or missing, there is no where to download them again. And I just want this light small file manager to be a certain level of 'Stateless'. > 3. BE also has a file cache, all downloaded files are saved on BE disks, and if file is missing, BE will downloaded file from FE again. > 4. I also add an option which allow user to just save file url in SmallFileMgr(You can see it in `CreateFileStmt.java`, but because it is not fully implemented yet, I currently forbid this feature). With this option, file in Doris will be represented as a URL. And the validation of URL is guaranteed by user themselves. Ok. I See. Thank you.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
