Cassandra blob storage
Hi, I have been spending some time looking into whether large files(100mb) can be stores in Cassandra. As per Cassandra faq: *Currently Cassandra isn't optimized specifically for large file or BLOB storage. However, files of around 64Mb and smaller can be easily stored in the database without splitting them into smaller chunks. This is primarily due to the fact that Cassandra's public API is based on Thrift, which offers no streaming abilities; any value written or fetched has to fit in to memory.* Does the above statement still hold? Thrift supports framed data transport, does that change the above statement. If not, why does casssandra not adopt the Thrift framed data transfer support? Thanks
Re: Cassandra blob storage
You may want to look at: https://github.com/Netflix/astyanax/wiki/Chunked-Object-Store -brian --- Brian O'Neill Chief Technology Officer Health Market Science The Science of Better Results 2700 Horizon Drive King of Prussia, PA 19406 M: 215.588.6024 @boneill42 http://www.twitter.com/boneill42 healthmarketscience.com This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited. From: prem yadav ipremya...@gmail.com Reply-To: user@cassandra.apache.org Date: Tuesday, March 18, 2014 at 1:41 PM To: user@cassandra.apache.org Subject: Cassandra blob storage Hi, I have been spending some time looking into whether large files(100mb) can be stores in Cassandra. As per Cassandra faq: Currently Cassandra isn't optimized specifically for large file or BLOB storage. However, files of around 64Mb and smaller can be easily stored in the database without splitting them into smaller chunks. This is primarily due to the fact that Cassandra's public API is based on Thrift, which offers no streaming abilities; any value written or fetched has to fit in to memory. Does the above statement still hold? Thrift supports framed data transport, does that change the above statement. If not, why does casssandra not adopt the Thrift framed data transfer support? Thanks
Re: Cassandra blob storage
Thanks Brian, I have seen that. Its more of a workaround and a hack. Of course a great solution. But my question is more about why Cassandra itself can't support that. Give then Thrift supports frames. Thanks. On Tue, Mar 18, 2014 at 5:55 PM, Brian O'Neill b...@alumni.brown.eduwrote: You may want to look at: https://github.com/Netflix/astyanax/wiki/Chunked-Object-Store -brian --- Brian O'Neill Chief Technology Officer *Health Market Science* *The Science of Better Results* 2700 Horizon Drive * King of Prussia, PA * 19406 M: 215.588.6024 * @boneill42 http://www.twitter.com/boneill42 * healthmarketscience.com This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited. From: prem yadav ipremya...@gmail.com Reply-To: user@cassandra.apache.org Date: Tuesday, March 18, 2014 at 1:41 PM To: user@cassandra.apache.org Subject: Cassandra blob storage Hi, I have been spending some time looking into whether large files(100mb) can be stores in Cassandra. As per Cassandra faq: *Currently Cassandra isn't optimized specifically for large file or BLOB storage. However, files of around 64Mb and smaller can be easily stored in the database without splitting them into smaller chunks. This is primarily due to the fact that Cassandra's public API is based on Thrift, which offers no streaming abilities; any value written or fetched has to fit in to memory.* Does the above statement still hold? Thrift supports framed data transport, does that change the above statement. If not, why does casssandra not adopt the Thrift framed data transfer support? Thanks
Re: Cassandra blob storage
For large volume big data scenarios we don't recommend using Cassandra as a blob storage simply because of intensive IO involved during compation, repair etc. Cassandra store is only well suited for metadata type storage. However, if you are fairly low volume then it's a different story, but if you have low volume why use Cassandra :) On Tue, Mar 18, 2014 at 10:55 AM, Brian O'Neill b...@alumni.brown.eduwrote: You may want to look at: https://github.com/Netflix/astyanax/wiki/Chunked-Object-Store -brian --- Brian O'Neill Chief Technology Officer *Health Market Science* *The Science of Better Results* 2700 Horizon Drive * King of Prussia, PA * 19406 M: 215.588.6024 * @boneill42 http://www.twitter.com/boneill42 * healthmarketscience.com This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited. From: prem yadav ipremya...@gmail.com Reply-To: user@cassandra.apache.org Date: Tuesday, March 18, 2014 at 1:41 PM To: user@cassandra.apache.org Subject: Cassandra blob storage Hi, I have been spending some time looking into whether large files(100mb) can be stores in Cassandra. As per Cassandra faq: *Currently Cassandra isn't optimized specifically for large file or BLOB storage. However, files of around 64Mb and smaller can be easily stored in the database without splitting them into smaller chunks. This is primarily due to the fact that Cassandra's public API is based on Thrift, which offers no streaming abilities; any value written or fetched has to fit in to memory.* Does the above statement still hold? Thrift supports framed data transport, does that change the above statement. If not, why does casssandra not adopt the Thrift framed data transfer support? Thanks
Re: Cassandra blob storage
@Mohit Bit confused with your reply. For what use cases you find Cassandra useful then? -Vivek On Tue, Mar 18, 2014 at 11:41 PM, Mohit Anchlia mohitanch...@gmail.comwrote: For large volume big data scenarios we don't recommend using Cassandra as a blob storage simply because of intensive IO involved during compation, repair etc. Cassandra store is only well suited for metadata type storage. However, if you are fairly low volume then it's a different story, but if you have low volume why use Cassandra :) On Tue, Mar 18, 2014 at 10:55 AM, Brian O'Neill b...@alumni.brown.eduwrote: You may want to look at: https://github.com/Netflix/astyanax/wiki/Chunked-Object-Store -brian --- Brian O'Neill Chief Technology Officer *Health Market Science* *The Science of Better Results* 2700 Horizon Drive * King of Prussia, PA * 19406 M: 215.588.6024 * @boneill42 http://www.twitter.com/boneill42 * healthmarketscience.com This information transmitted in this email message is for the intended recipient only and may contain confidential and/or privileged material. If you received this email in error and are not the intended recipient, or the person responsible to deliver it to the intended recipient, please contact the sender at the email above and delete this email and any attachments and destroy any copies thereof. Any review, retransmission, dissemination, copying or other use of, or taking any action in reliance upon, this information by persons or entities other than the intended recipient is strictly prohibited. From: prem yadav ipremya...@gmail.com Reply-To: user@cassandra.apache.org Date: Tuesday, March 18, 2014 at 1:41 PM To: user@cassandra.apache.org Subject: Cassandra blob storage Hi, I have been spending some time looking into whether large files(100mb) can be stores in Cassandra. As per Cassandra faq: *Currently Cassandra isn't optimized specifically for large file or BLOB storage. However, files of around 64Mb and smaller can be easily stored in the database without splitting them into smaller chunks. This is primarily due to the fact that Cassandra's public API is based on Thrift, which offers no streaming abilities; any value written or fetched has to fit in to memory. * Does the above statement still hold? Thrift supports framed data transport, does that change the above statement. If not, why does casssandra not adopt the Thrift framed data transfer support? Thanks
Re: Cassandra blob storage
On Tue, Mar 18, 2014 at 10:41 AM, prem yadav ipremya...@gmail.com wrote: I have been spending some time looking into whether large files(100mb) can be stores in Cassandra. As per Cassandra faq: https://code.google.com/p/mogilefs/ Cassandra is not optimized for single values of this size. Leaving aside Thrift, trying to store 100mb in a single cell is not Cassandra's sweet spot. =Rob
Re: Cassandra blob storage
On Tue, Mar 18, 2014 at 10:41 AM, prem yadav ipremya...@gmail.com wrote: I have been spending some time looking into whether large files(100mb) can be stores in Cassandra. As per Cassandra faq: https://code.google.com/p/mogilefs/ Cassandra is not optimized for single values of this size. Leaving aside Thrift, trying to store 100mb in a single cell is not Cassandra's sweet spot. =Rob
