This is an automated email from the ASF dual-hosted git repository.
sarankk pushed a change to branch mutation-tracking-support
in repository https://gitbox.apache.org/repos/asf/cassandra-analytics.git
omit 6064e02e CASSANALYTICS-160: Determine whether mutation tracking is
enabled for keyspace for bulk writes (#214)
add 30e69988 CASSANALYTICS-24: SSTable-version-based bridge determination
add c0aab1d4 CASSANALYTICS-184 : CdcState.ReplicaCountSerializer map-size
overflow corrupts persisted CDC state (#225)
add 839020f6 CASSANALYTICS-182 : CDC batch-write mixing a CDC-enabled and
CDC-disabled table drops the CDC table's mutation (#224)
add 39ce0567 CASSANALYTICS-26: Support vector data type Patch by Lukasz
Antoniak; reviewed by Shailaja Koppu, Yifan Cai for CASSANALYTICS-26
add be5080e3 CASSANALYTICS-179: Fix total timeout calculation for
getAllNodeSettings
add 5aca2e68 CASSANALYTICS-186: AbstractSchemaBuilder.build() drops the
cdc flag, silently disabling CDC on all tables (#227)
add 98ca34c9 CASSANALYTICS-187: Fix FastByteOperations Silently Falling
Back to Pure-Java Comparator (#228)
add 5efb86ca CASSANALYTICS-188: Expand list of architecture that supports
unaligned access in FastByteOperations (#229)
add c92b6517 CASSANALYTICS-189 Add CapturePublishedSchema metric to
SidecarCdcStats (#230)
add dd4d2563 CASSANALYTICS-191: CDC reader stats silently dropped in
SidecarCdcBuilder (#232)
add 20116e26 CASSANALYTICS-180: TokenPartitioner fails to detect range gap
in reader (#234)
add 2ded78f5 CASSANALYTICS-104: Eliminate redundant filesystem lookups in
SSTable direct streaming Patch by Lukasz Antoniak; reviewed by Yifan Cai for
CASSANALYTICS-104
add 9a235588 Add Cassandra 6.0 support
add 5e9e5e71 CASSANALYTICS-178: CDC logs NPE for deleted column values
(#242)
add 5ebb4a01 ninja Fix: Adding missing entries in CHANGES.txt
add 03040d2b CASSANALYTICS-198: Add publish script and workflow (#245)
new b0c400bf CASSANALYTICS-160: Determine whether mutation tracking is
enabled for keyspace for bulk writes (#214)
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6064e02e)
\
N -- N -- N refs/heads/mutation-tracking-support (b0c400bf)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.circleci/config.yml | 37 +-
.github/workflows/publish-snapshot.yaml | 166 +++++++
.github/workflows/test.yaml | 10 +-
CHANGES.txt | 15 +
DEV-README.md | 68 ++-
.../cassandra/sidecar/client/SidecarClient.java | 12 +
build.gradle | 11 +-
.../java/org/apache/cassandra/cdc/CdcLogMode.java | 10 +-
.../cdc/schemastore/CachingSchemaStore.java | 22 +-
cassandra-analytics-cdc-sidecar/build.gradle | 22 +
.../apache/cassandra/cdc/sidecar/SidecarCdc.java | 12 +-
.../cassandra/cdc/sidecar/SidecarCdcBuilder.java | 9 +
.../cassandra/cdc/sidecar/SidecarCdcStats.java | 7 +
.../cassandra/cdc/sidecar/SidecarCdcTest.java | 56 +++
cassandra-analytics-cdc/build.gradle | 30 +-
.../main/java/org/apache/cassandra/cdc/Cdc.java | 43 +-
.../java/org/apache/cassandra/cdc/CdcBuilder.java | 3 +-
.../apache/cassandra/cdc/api/SchemaSupplier.java | 25 +-
.../java/org/apache/cassandra/cdc/CdcTests.java | 232 +++++++++-
.../cdc/model/CdcKryoSerializationTests.java | 46 ++
.../cassandra/cdc/test/TestVersionSupplier.java | 4 +-
.../bridge/BridgeInitializationParameters.java | 13 +-
.../apache/cassandra/bridge/CassandraVersion.java | 157 ++++++-
.../cassandra/bridge/CassandraVersionFeatures.java | 49 +-
.../org/apache/cassandra/cdc/api/CommitLog.java | 16 +-
.../org/apache/cassandra/cdc/state/CdcState.java | 22 +-
.../cassandra/spark/data/CassandraTypes.java | 11 +
.../org/apache/cassandra/spark/data/CqlField.java | 8 +-
.../org/apache/cassandra/spark/data/CqlTable.java | 33 +-
.../spark/data/partitioner/TokenPartitioner.java | 12 +-
.../org/apache/cassandra/spark/utils/CqlUtils.java | 2 +-
.../cassandra/bridge/CassandraVersionTest.java | 163 +++++++
.../TokenPartitionerValidationTest.java | 111 +++++
.../spark/example/DirectWriteAndReadJob.java | 3 +-
cassandra-analytics-core/build.gradle | 28 ++
.../cassandra/bridge/SSTableVersionAnalyzer.java | 209 +++++++++
.../org/apache/cassandra/spark/KryoRegister.java | 46 +-
.../bulkwriter/AbstractBulkWriterContext.java | 23 +-
.../spark/bulkwriter/BroadcastableClusterInfo.java | 13 +-
.../bulkwriter/BroadcastableClusterInfoGroup.java | 19 +-
.../spark/bulkwriter/BroadcastableTableSchema.java | 13 +-
.../cassandra/spark/bulkwriter/BulkSparkConf.java | 14 +-
.../spark/bulkwriter/BulkWriterConfig.java | 23 +-
.../bulkwriter/CassandraBulkWriterContext.java | 2 +-
.../spark/bulkwriter/CassandraClusterInfo.java | 114 +++--
.../CassandraDirectDataTransportContext.java | 2 +-
.../cassandra/spark/bulkwriter/ClusterInfo.java | 3 +-
.../spark/bulkwriter/DirectStreamSession.java | 90 ++--
.../bulkwriter/IBroadcastableClusterInfo.java | 7 +-
.../spark/bulkwriter/SSTableWriterFactory.java | 4 +-
.../spark/bulkwriter/SortedSSTableWriter.java | 223 +++++++--
.../spark/bulkwriter/SqlToCqlTypeConverter.java | 2 +
.../cassandra/spark/bulkwriter/StreamSession.java | 13 -
.../cassandra/spark/bulkwriter/TableSchema.java | 19 +-
.../spark/bulkwriter/TokenPartitioner.java | 13 +-
.../cloudstorage/CloudStorageStreamSession.java | 21 +-
.../coordinated/CassandraClusterInfoGroup.java | 46 +-
.../CassandraCoordinatedBulkWriterContext.java | 2 +-
.../spark/bulkwriter/util/FastByteOperations.java | 147 +++---
.../cassandra/spark/data/CassandraDataLayer.java | 168 ++++++-
.../cassandra/spark/data/LocalDataLayer.java | 16 +-
.../bridge/BridgeVersionConsistencyTest.java | 64 +++
.../bridge/CassandraVersionFeaturesTest.java | 22 +
.../bridge/SSTableVersionAnalyzerTest.java | 190 ++++++++
.../apache/cassandra/spark/KryoRegisterTest.java | 82 ++++
.../cassandra/spark/KryoSerializationTests.java | 57 +++
.../java/org/apache/cassandra/spark/TestUtils.java | 5 +-
.../bulkwriter/AbstractBulkWriterContextTest.java | 133 ++++++
.../spark/bulkwriter/BulkSparkConfTest.java | 32 ++
.../BulkWriterConfigExtensibilityTest.java | 8 +-
.../CassandraClusterInfoBridgeVersionTest.java | 199 ++++++++
.../spark/bulkwriter/CassandraClusterInfoTest.java | 80 ++++
.../spark/bulkwriter/DirectStreamSessionTest.java | 2 +-
.../spark/bulkwriter/MockBulkWriterContext.java | 7 +-
.../spark/bulkwriter/RecordWriterTest.java | 2 +-
.../cassandra/spark/bulkwriter/SSTables.java | 2 +
.../spark/bulkwriter/SortedSSTableWriterTest.java | 25 +-
.../bulkwriter/StreamSessionConsistencyTest.java | 2 +-
.../spark/bulkwriter/TableSchemaTestCommon.java | 8 +-
.../spark/bulkwriter/TokenPartitionerTest.java | 100 ++++
.../coordinated/CassandraClusterInfoGroupTest.java | 97 ++--
.../data/CassandraDataLayerValidationTest.java | 360 ++++++++++++++
.../cassandra/spark/endtoend/DataTypeTests.java | 101 ++++
.../apache/cassandra/spark/endtoend/MiscTests.java | 16 +-
.../build.gradle | 4 +-
.../distributed/impl/CassandraCluster.java | 5 +
.../testing/SharedClusterIntegrationTestBase.java | 48 +-
.../cassandra/analytics/BulkReaderVectorTest.java | 115 +++++
...undtripBigSSTableVersionBridgeDisabledTest.java | 20 +-
...oundtripBigSSTableVersionBridgeEnabledTest.java | 20 +-
...undtripBtiSSTableVersionBridgeDisabledTest.java | 20 +-
...oundtripBtiSSTableVersionBridgeEnabledTest.java | 22 +-
.../BulkRoundtripSSTableVersionBridgeTestBase.java | 271 +++++++++++
.../cassandra/analytics/BulkWriteVectorTest.java | 115 +++++
.../analytics/MixedSSTableVersionTest.java | 222 +++++++++
.../cassandra/analytics/ResiliencyTestBase.java | 33 ++
.../SharedClusterSparkIntegrationTestBase.java | 76 +++
.../apache/cassandra/analytics/SparkTestUtils.java | 1 -
.../analytics/expansion/JoiningTestBase.java | 6 +
.../analytics/movement/NodeMovementTestBase.java | 6 +
.../replacement/HostReplacementTestBase.java | 6 +
.../analytics/shrink/LeavingTestBase.java | 6 +
cassandra-analytics-sidecar-client/build.gradle | 19 +
.../java/org/apache/cassandra/clients/Sidecar.java | 84 ++++
.../org/apache/cassandra/clients/SidecarTest.java | 193 ++++++++
.../SparkSqlTypeConverterImplementation.java | 4 +
.../build.gradle | 18 +-
.../apache/cassandra/bridge/CassandraBridge.java | 5 +
.../org/apache/cassandra/bridge/CdcBridge.java | 20 +
.../cassandra/spark/utils/test/TestSchema.java | 3 +-
.../cassandra/bridge/CassandraSchemaTests.java | 55 +++
.../io/sstable/format/bti/BtiReaderUtilsTest.java | 8 +-
.../data/converter/types/VectorTypeTests.java | 59 +++
.../cassandra/spark/reader/SummaryDbTests.java | 6 +-
.../bridge/CassandraTypesImplementation.java | 21 +-
.../org/apache/cassandra/bridge/SchemaUpdater.java | 35 ++
.../cassandra/spark/data/complex/CqlVector.java | 166 +++++++
.../cassandra/spark/reader/SchemaBuilder.java | 114 +++++
.../bridge/AbstractCdcBridgeImplementation.java | 6 +
.../bridge/SSTableWriterImplementation.java | 6 +-
.../db/commitlog/BufferingCommitLogReader.java | 53 ++-
.../spark/reader/AbstractStreamScanner.java | 2 +-
.../cassandra/bridge/CassandraSchemaTests.java | 55 +++
.../cassandra/spark/reader/SchemaBuilderTests.java | 50 ++
.../cassandra/bridge/AbstractCassandraTypes.java | 6 +
.../apache/cassandra/bridge/CassandraSchema.java | 167 +++++--
.../org/apache/cassandra/bridge/SchemaUpdater.java | 8 +
.../spark/data/complex/AbstractCqlList.java | 7 +-
.../spark/data/complex/CqlCollection.java | 17 +-
...hemaBuilder.java => AbstractSchemaBuilder.java} | 75 +--
.../cassandra/spark/reader/SchemaBuilder.java | 515 +--------------------
.../build.gradle | 18 +-
.../build.gradle | 74 ++-
.../bridge/AbstractCdcBridgeImplementation.java | 11 +
.../bridge/CassandraBridgeImplementation.java | 27 +-
.../cdc/scanner/CdcSortedStreamScanner.java | 5 +
.../main/java/org/apache/cassandra/db/DbUtils.java | 6 +-
.../io/sstable/format/bti/BtiReaderUtils.java | 6 +-
.../io/util/RebufferingChannelInputStream.java | 5 +-
.../spark/reader/AbstractStreamScanner.java | 4 +-
.../spark/reader/CompressionMetadata.java | 303 ++++++++++++
.../cassandra/spark/reader/SSTableCache.java | 42 +-
.../cassandra/spark/reader/SSTableReader.java | 170 +++++--
.../cassandra/bridge/CassandraSchemaTests.java | 58 ++-
.../bridge/SSTableWriterImplementationTest.java | 33 +-
.../spark/reader/CompressionDictionaryTests.java | 387 ++++++++++++++++
.../cassandra/spark/reader/SchemaBuilderTests.java | 33 +-
.../build.gradle | 16 +-
.../apache/cassandra/bridge/CassandraSchema.java | 205 +++++---
.../bridge/CassandraTypesImplementation.java | 39 +-
.../org/apache/cassandra/bridge/SchemaBridge.java | 6 +-
.../org/apache/cassandra/bridge/SchemaUpdater.java | 129 ++++++
.../org/apache/cassandra/spark/data/CqlType.java | 2 +-
.../cassandra/spark/data/complex/CqlTuple.java | 2 +-
.../cassandra/spark/data/complex/CqlUdt.java | 5 +
.../spark/reader/AbstractSchemaBuilder.java | 125 ++---
.../cassandra/spark/reader/ComplexTypeBuffer.java | 20 +-
.../build.gradle | 20 +-
gradle.properties | 3 +-
gradle/common/publishing-spark.gradle | 4 +-
gradlew | 6 +
scripts/build-dtest-jars.sh | 9 +-
scripts/publish-snapshot.sh | 58 +++
scripts/relocate-dtest-dependencies.pom | 1 +
settings.gradle | 5 +
165 files changed, 7514 insertions(+), 1410 deletions(-)
create mode 100644 .github/workflows/publish-snapshot.yaml
create mode 100644
cassandra-analytics-common/src/test/java/org/apache/cassandra/bridge/CassandraVersionTest.java
create mode 100644
cassandra-analytics-common/src/test/java/org/apache/cassandra/spark/data/partitioner/TokenPartitionerValidationTest.java
create mode 100644
cassandra-analytics-core/src/main/java/org/apache/cassandra/bridge/SSTableVersionAnalyzer.java
create mode 100644
cassandra-analytics-core/src/test/java/org/apache/cassandra/bridge/BridgeVersionConsistencyTest.java
create mode 100644
cassandra-analytics-core/src/test/java/org/apache/cassandra/bridge/SSTableVersionAnalyzerTest.java
create mode 100644
cassandra-analytics-core/src/test/java/org/apache/cassandra/spark/KryoRegisterTest.java
create mode 100644
cassandra-analytics-core/src/test/java/org/apache/cassandra/spark/bulkwriter/AbstractBulkWriterContextTest.java
create mode 100644
cassandra-analytics-core/src/test/java/org/apache/cassandra/spark/bulkwriter/CassandraClusterInfoBridgeVersionTest.java
create mode 100644
cassandra-analytics-core/src/test/java/org/apache/cassandra/spark/data/CassandraDataLayerValidationTest.java
create mode 100644
cassandra-analytics-integration-tests/src/test/java/org/apache/cassandra/analytics/BulkReaderVectorTest.java
copy
cassandra-four-zero-types/src/main/java/org/apache/cassandra/spark/data/types/Counter.java
=>
cassandra-analytics-integration-tests/src/test/java/org/apache/cassandra/analytics/BulkRoundtripBigSSTableVersionBridgeDisabledTest.java
(67%)
copy
cassandra-five-zero-types/src/main/java/org/apache/cassandra/spark/data/complex/CqlTuple.java
=>
cassandra-analytics-integration-tests/src/test/java/org/apache/cassandra/analytics/BulkRoundtripBigSSTableVersionBridgeEnabledTest.java
(67%)
copy
cassandra-four-zero-types/src/main/java/org/apache/cassandra/spark/data/types/Counter.java
=>
cassandra-analytics-integration-tests/src/test/java/org/apache/cassandra/analytics/BulkRoundtripBtiSSTableVersionBridgeDisabledTest.java
(65%)
copy
cassandra-analytics-core/src/main/java/org/apache/cassandra/spark/common/schema/IntegerType.java
=>
cassandra-analytics-integration-tests/src/test/java/org/apache/cassandra/analytics/BulkRoundtripBtiSSTableVersionBridgeEnabledTest.java
(65%)
create mode 100644
cassandra-analytics-integration-tests/src/test/java/org/apache/cassandra/analytics/BulkRoundtripSSTableVersionBridgeTestBase.java
create mode 100644
cassandra-analytics-integration-tests/src/test/java/org/apache/cassandra/analytics/BulkWriteVectorTest.java
create mode 100644
cassandra-analytics-integration-tests/src/test/java/org/apache/cassandra/analytics/MixedSSTableVersionTest.java
create mode 100644
cassandra-analytics-sidecar-client/src/test/java/org/apache/cassandra/clients/SidecarTest.java
copy {cassandra-analytics-spark-five-zero-converter =>
cassandra-analytics-spark-six-zero-converter}/build.gradle (75%)
create mode 100644
cassandra-five-zero-bridge/src/test/java/org/apache/cassandra/spark/data/converter/types/VectorTypeTests.java
create mode 100644
cassandra-five-zero-types/src/main/java/org/apache/cassandra/spark/data/complex/CqlVector.java
create mode 100644
cassandra-five-zero-types/src/main/java/org/apache/cassandra/spark/reader/SchemaBuilder.java
copy
cassandra-four-zero-types/src/main/java/org/apache/cassandra/spark/reader/{SchemaBuilder.java
=> AbstractSchemaBuilder.java} (91%)
copy {cassandra-five-zero-avro-converter =>
cassandra-six-zero-avro-converter}/build.gradle (74%)
copy {cassandra-five-zero-bridge => cassandra-six-zero-bridge}/build.gradle
(58%)
copy {cassandra-four-zero-bridge =>
cassandra-six-zero-bridge}/src/main/java/org/apache/cassandra/bridge/AbstractCdcBridgeImplementation.java
(96%)
copy {cassandra-five-zero-bridge =>
cassandra-six-zero-bridge}/src/main/java/org/apache/cassandra/bridge/CassandraBridgeImplementation.java
(96%)
copy {cassandra-four-zero-bridge =>
cassandra-six-zero-bridge}/src/main/java/org/apache/cassandra/cdc/scanner/CdcSortedStreamScanner.java
(97%)
copy {cassandra-five-zero-bridge =>
cassandra-six-zero-bridge}/src/main/java/org/apache/cassandra/db/DbUtils.java
(88%)
copy {cassandra-five-zero-bridge =>
cassandra-six-zero-bridge}/src/main/java/org/apache/cassandra/io/sstable/format/bti/BtiReaderUtils.java
(98%)
copy {cassandra-five-zero-bridge =>
cassandra-six-zero-bridge}/src/main/java/org/apache/cassandra/io/util/RebufferingChannelInputStream.java
(90%)
copy {cassandra-five-zero-bridge =>
cassandra-six-zero-bridge}/src/main/java/org/apache/cassandra/spark/reader/AbstractStreamScanner.java
(99%)
create mode 100644
cassandra-six-zero-bridge/src/main/java/org/apache/cassandra/spark/reader/CompressionMetadata.java
copy {cassandra-four-zero-bridge =>
cassandra-six-zero-bridge}/src/main/java/org/apache/cassandra/spark/reader/SSTableCache.java
(80%)
copy {cassandra-five-zero-bridge =>
cassandra-six-zero-bridge}/src/main/java/org/apache/cassandra/spark/reader/SSTableReader.java
(83%)
copy {cassandra-four-zero-bridge =>
cassandra-six-zero-bridge}/src/test/java/org/apache/cassandra/bridge/CassandraSchemaTests.java
(56%)
copy {cassandra-five-zero-bridge =>
cassandra-six-zero-bridge}/src/test/java/org/apache/cassandra/bridge/SSTableWriterImplementationTest.java
(86%)
create mode 100644
cassandra-six-zero-bridge/src/test/java/org/apache/cassandra/spark/reader/CompressionDictionaryTests.java
copy {cassandra-five-zero-bridge =>
cassandra-six-zero-bridge}/src/test/java/org/apache/cassandra/spark/reader/SchemaBuilderTests.java
(84%)
copy {cassandra-five-zero-types => cassandra-six-zero-types}/build.gradle (71%)
copy {cassandra-four-zero-types =>
cassandra-six-zero-types}/src/main/java/org/apache/cassandra/bridge/CassandraSchema.java
(61%)
copy {cassandra-five-zero-types =>
cassandra-six-zero-types}/src/main/java/org/apache/cassandra/bridge/CassandraTypesImplementation.java
(66%)
copy {cassandra-four-zero-types =>
cassandra-six-zero-types}/src/main/java/org/apache/cassandra/bridge/SchemaBridge.java
(88%)
create mode 100644
cassandra-six-zero-types/src/main/java/org/apache/cassandra/bridge/SchemaUpdater.java
copy {cassandra-five-zero-types =>
cassandra-six-zero-types}/src/main/java/org/apache/cassandra/spark/data/CqlType.java
(98%)
copy {cassandra-five-zero-types =>
cassandra-six-zero-types}/src/main/java/org/apache/cassandra/spark/data/complex/CqlTuple.java
(96%)
copy {cassandra-four-zero-types =>
cassandra-six-zero-types}/src/main/java/org/apache/cassandra/spark/data/complex/CqlUdt.java
(97%)
copy
cassandra-four-zero-types/src/main/java/org/apache/cassandra/spark/reader/SchemaBuilder.java
=>
cassandra-six-zero-types/src/main/java/org/apache/cassandra/spark/reader/AbstractSchemaBuilder.java
(84%)
copy {cassandra-five-zero-types =>
cassandra-six-zero-types}/src/main/java/org/apache/cassandra/spark/reader/ComplexTypeBuffer.java
(61%)
copy {cassandra-four-zero => cassandra-six-zero}/build.gradle (59%)
create mode 100755 scripts/publish-snapshot.sh
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
