[
https://issues.apache.org/jira/browse/CASSANDRA-16222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Berragan updated CASSANDRA-16222:
---------------------------------------
Description:
*Description:*
This ticket introduces the Spark-Cassandra Bulk Reader: a Spark library able
to read and compact Cassandra raw sstables into SparkSQL along the principles
of streaming compaction. The full code is attached as a patch file and will be
submitted to a GitHub repo.
**
*Motivation*
For analytics or "select *" use cases at scale, the performance is
prohibitively expensive to read via the normal CQL read path - either using the
Java driver or the Open Source Spark connector. By reading the raw sstables,
the bulk reader is able to read with near-zero impact to a production cluster
at speeds many orders of magnitudes faster than alternatives. We have seen very
good performance results, exporting a 32TB table (~46bn CQL rows) to HDFS in
Parquet format in around 1h10m; a 20x reduction compared to previous solutions.
By reading from multiple replicas and ‘compacting’ duplicate data together, it
can achieve consistency at a user defined level (i.e. ONE, TWO, LOCAL_QUORUM
etc).
*Overview*
This library provides the core code for reading a set of SSTables into
SparkSQL through a DataLayer abstraction. The role of the DataLayer is to:
* return a SchemaStruct, mapping the Cassandra CQL table schema to the
SparkSQL schema.
* a list of sstables available for reading.
* a method to open an InputStream for any file component of an sstable (e.g.
data, compression, summary etc).
The PartitionedDataLayer abstraction builds on the DataLayer interface for
partitioning Spark workers across a Cassandra token ring - allowing the Spark
job to scale linearly - and reading from sufficient Cassandra replicas to
achieve a user-specified consistency level.
A simple example LocalDataLayer implementation is included for reading from a
local file system. Users of the library can build their own implementations to
read from wherever they wish e.g. reading from a backup in a cloud storage
system, or reading from the snapshot directory of a live Cassandra cluster.
At the core, the Bulk Reader uses the Apache Cassandra CompactionIterator to
perform the streaming compaction. As it iterates through the CompactionIterator
it deserializes the ByteBuffers, converts into the appropriate SparkSQL data
type and pivots each cell into a SparkSQL row.
Supporting this library is a robust property-based test framework for writing
Cassandra sstables with arbitrary schemas using the CQLSSTableWriter, and
reading back through SparkSQL to verify the library achieves both consistency
and correctness.
was:
*Description:*
This ticket introduces the Spark-Cassandra Bulk Reader: a Spark library able to
read and compact Cassandra raw sstables into SparkSQL along the principles of
streaming compaction. The full code is attached as a patch file and will be
submitted to a GitHub repo.
**
*Motivation*
For analytics or "select *" use cases at scale, the performance is
prohibitively expensive to read via the normal CQL read path - either using the
Java driver or the Open Source Spark connector. By reading the raw sstables,
the bulk reader is able to read with near-zero impact to a production cluster
at speeds many orders of magnitudes faster than alternatives. We have seen very
good performance results, exporting a 32TB table (~46bn CQL rows) to HDFS in
Parquet format in around 1h10m; a 20x reduction compared to previous solutions.
By reading from multiple replicas and ‘compacting’ duplicate data together, it
can achieve consistency at a user defined level (i.e. ONE, TWO, LOCAL_QUORUM
etc).
*Overview*
This library provides the core code for reading a set of SSTables into SparkSQL
through an a DataLayer abstraction. The role of the DataLayer is to:
* return a SchemaStruct, mapping the Cassandra CQL table schema to the
SparkSQL schema.
* a list of sstables available for reading.
* a method to open an InputStream for any file component of an sstable (e.g.
data, compression, summary etc).
The PartitionedDataLayer abstraction builds on the DataLayer interface for
partitioning Spark workers across a Cassandra token ring - allowing the Spark
job to scale linearly - and reading from sufficient Cassandra replicas to
achieve a user-specified consistency level.
A simple example LocalDataLayer implementation is included for reading from a
local file system. Users of the library can build their own implementations to
read from wherever they wish e.g. reading from a backup in a cloud storage
system, or reading from the snapshot directory of a live Cassandra cluster.
At the core, the Bulk Reader uses the Apache Cassandra CompactionIterator to
perform the streaming compaction. As it iterates through the CompactionIterator
it deserializes the ByteBuffers, converts into the appropriate SparkSQL data
type and pivots each cell into a SparkSQL row.
Supporting this library is a robust property-based test framework for writing
Cassandra sstables with arbitrary schemas using the CQLSSTableWriter, and
reading back through SparkSQL to verify the library achieves both consistency
and correctness.
> Spark-Cassandra Bulk Reader
> ---------------------------
>
> Key: CASSANDRA-16222
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16222
> Project: Cassandra
> Issue Type: New Feature
> Reporter: James Berragan
> Priority: Normal
> Attachments: sparkbulkreader.patch
>
>
> *Description:*
> This ticket introduces the Spark-Cassandra Bulk Reader: a Spark library able
> to read and compact Cassandra raw sstables into SparkSQL along the principles
> of streaming compaction. The full code is attached as a patch file and will
> be submitted to a GitHub repo.
> **
> *Motivation*
> For analytics or "select *" use cases at scale, the performance is
> prohibitively expensive to read via the normal CQL read path - either using
> the Java driver or the Open Source Spark connector. By reading the raw
> sstables, the bulk reader is able to read with near-zero impact to a
> production cluster at speeds many orders of magnitudes faster than
> alternatives. We have seen very good performance results, exporting a 32TB
> table (~46bn CQL rows) to HDFS in Parquet format in around 1h10m; a 20x
> reduction compared to previous solutions. By reading from multiple replicas
> and ‘compacting’ duplicate data together, it can achieve consistency at a
> user defined level (i.e. ONE, TWO, LOCAL_QUORUM etc).
> *Overview*
> This library provides the core code for reading a set of SSTables into
> SparkSQL through a DataLayer abstraction. The role of the DataLayer is to:
> * return a SchemaStruct, mapping the Cassandra CQL table schema to the
> SparkSQL schema.
> * a list of sstables available for reading.
> * a method to open an InputStream for any file component of an sstable (e.g.
> data, compression, summary etc).
> The PartitionedDataLayer abstraction builds on the DataLayer interface for
> partitioning Spark workers across a Cassandra token ring - allowing the Spark
> job to scale linearly - and reading from sufficient Cassandra replicas to
> achieve a user-specified consistency level.
> A simple example LocalDataLayer implementation is included for reading from a
> local file system. Users of the library can build their own implementations
> to read from wherever they wish e.g. reading from a backup in a cloud storage
> system, or reading from the snapshot directory of a live Cassandra cluster.
>
> At the core, the Bulk Reader uses the Apache Cassandra CompactionIterator to
> perform the streaming compaction. As it iterates through the
> CompactionIterator it deserializes the ByteBuffers, converts into the
> appropriate SparkSQL data type and pivots each cell into a SparkSQL row.
>
> Supporting this library is a robust property-based test framework for
> writing Cassandra sstables with arbitrary schemas using the CQLSSTableWriter,
> and reading back through SparkSQL to verify the library achieves both
> consistency and correctness.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
