cassandra git commit: fix logging contexts

2017-12-29 Thread dbrosius 
Repository: cassandra
Updated Branches:
  refs/heads/trunk 88c0e29ca -> 21be9d2f5


fix logging contexts


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/21be9d2f
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/21be9d2f
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/21be9d2f

Branch: refs/heads/trunk
Commit: 21be9d2f50cc6a6bdceff56389adc015f811d5d6
Parents: 88c0e29
Author: Dave Brosius 
Authored: Fri Dec 29 20:26:46 2017 -0500
Committer: Dave Brosius 
Committed: Fri Dec 29 20:26:46 2017 -0500

--
 src/java/org/apache/cassandra/db/view/ViewBuilder.java | 2 +-
 .../org/apache/cassandra/repair/asymmetric/RangeDenormalizer.java  | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/21be9d2f/src/java/org/apache/cassandra/db/view/ViewBuilder.java
--
diff --git a/src/java/org/apache/cassandra/db/view/ViewBuilder.java 
b/src/java/org/apache/cassandra/db/view/ViewBuilder.java
index 8187a57..c727f63 100644
--- a/src/java/org/apache/cassandra/db/view/ViewBuilder.java
+++ b/src/java/org/apache/cassandra/db/view/ViewBuilder.java
@@ -58,7 +58,7 @@ import static java.util.stream.Collectors.toList;
  */
 class ViewBuilder
 {
-private static final Logger logger = 
LoggerFactory.getLogger(ViewBuilderTask.class);
+private static final Logger logger = 
LoggerFactory.getLogger(ViewBuilder.class);
 
 private static final int NUM_TASKS = 
Runtime.getRuntime().availableProcessors() * 4;
 

http://git-wip-us.apache.org/repos/asf/cassandra/blob/21be9d2f/src/java/org/apache/cassandra/repair/asymmetric/RangeDenormalizer.java
--
diff --git 
a/src/java/org/apache/cassandra/repair/asymmetric/RangeDenormalizer.java 
b/src/java/org/apache/cassandra/repair/asymmetric/RangeDenormalizer.java
index a04d6d5..f692dd6 100644
--- a/src/java/org/apache/cassandra/repair/asymmetric/RangeDenormalizer.java
+++ b/src/java/org/apache/cassandra/repair/asymmetric/RangeDenormalizer.java
@@ -37,7 +37,7 @@ import org.apache.cassandra.dht.Token;
 
 public class RangeDenormalizer
 {
-private static final Logger logger = 
LoggerFactory.getLogger(IncomingRepairStreamTracker.class);
+private static final Logger logger = 
LoggerFactory.getLogger(RangeDenormalizer.class);
 
 /**
  * "Denormalizes" (kind of the opposite of what Range.normalize does) the 
ranges in the keys of {{incoming}}


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

cassandra git commit: fix logging parm markers, to parms count

2017-12-29 Thread dbrosius 
Repository: cassandra
Updated Branches:
  refs/heads/trunk e2e28a02e -> 88c0e29ca


fix logging parm markers, to parms count


Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/88c0e29c
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/88c0e29c
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/88c0e29c

Branch: refs/heads/trunk
Commit: 88c0e29caaffab41422adc673182bb9549e735ab
Parents: e2e28a0
Author: Dave Brosius 
Authored: Fri Dec 29 20:23:27 2017 -0500
Committer: Dave Brosius 
Committed: Fri Dec 29 20:23:27 2017 -0500

--
 src/java/org/apache/cassandra/schema/MigrationManager.java | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/cassandra/blob/88c0e29c/src/java/org/apache/cassandra/schema/MigrationManager.java
--
diff --git a/src/java/org/apache/cassandra/schema/MigrationManager.java 
b/src/java/org/apache/cassandra/schema/MigrationManager.java
index 50e5c28..ef19c25 100644
--- a/src/java/org/apache/cassandra/schema/MigrationManager.java
+++ b/src/java/org/apache/cassandra/schema/MigrationManager.java
@@ -95,7 +95,7 @@ public class MigrationManager
 if (Schema.instance.isEmpty() || runtimeMXBean.getUptime() < 
MIGRATION_DELAY_IN_MS)
 {
 // If we think we may be bootstrapping or have recently started, 
submit MigrationTask immediately
-logger.debug("Immediately submitting migration task for {} due to 
{}, " +
+logger.debug("Immediately submitting migration task for {}, " +
  "schema versions: local={}, remote={}",
  endpoint,
  
Schema.schemaVersionToString(Schema.instance.getVersion()),


-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-14134) Migrate dtests to use pytest and python3

2017-12-29 Thread Jeff Jirsa (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306621#comment-16306621
 ] 

Jeff Jirsa commented on CASSANDRA-14134:


cc [~philipthompson] 

> Migrate dtests to use pytest and python3
> 
>
> Key: CASSANDRA-14134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14134
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>
> h4. Get the C* dtests running on the pytest framework.
> C* DTests currently run using the python test framework nosetest. This 
> framework has been largely abandoned with no releases since 2015 and a 
> general strong consensus in the python community that pytest is the future.
> h4. Why should we do this.
> Currently (and historically) dtests have always been difficult to run, flaky 
> and unpredictable in CI environments, and almost impossible to debug.
> On November 28th, 2017, I proposed on the dev@ list that we move the dtests 
> from nosetests to pytests. I got replies from Jon Haddad, Philip Thompson, 
> and kurt greaves with really only "+1" like replies to the proposal.
> Since then I've been working pretty much non stop to complete the large 
> refactor of dtests to pytests. As part of this effort (and due to the 
> migration tools that exist require it) I also ported the code to python3 
> (from the current python 2.7 based code-base).
> h4. High-level summary of key changes, improvements, and new features.
> * Migrate dtests from executing using the nosetest framework to pytest
> * Port the entire code base from Python 2.7 to Python 3.6
> * Update run_dtests.py to work with pytest
> * Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
> list of all available collected tests
> * Update README.md for executing the dtests with pytest
> * Add new debugging tips section to README.md to help with some basics of 
> debugging python3 and pytest
> * Migrate all existing Enviornment Variable usage as a means to control dtest 
> operation modes to argparse command line options with documented help on each 
> toggles intended usage
> * Migration of old unitTest and nose based test structure to modern pytest 
> fixture approach
> * Automatic detection of physical system resources to automatically determine 
> if @pytest.mark.resource_intensive annotated tests should be collected and 
> run on the system where they are being executed
> * new pytest fixture replacements for @since and @pytest.mark.upgrade_test 
> annotations
> * Migration to python logging framework
> * Upgrade thrift bindings to latest version with full python3 compatibility
> * Remove deprecated cql and pycassa dependencies and migrate any remaining 
> tests to fully remove those dependencies
> * Fixed dozens of tests that would hang the pytest framework forever when run 
> in CI enviornments
> * Ran code nearly 300 times in CircleCI during the migration and to find, 
> identify, and fix any tests capable of hanging CI
> * Upgrade Tests do not yet run in CI and still need additional migration work 
> (although all upgrade test classes compile successfully)
> I started with the *nose2pytest* [https://github.com/pytest-dev/nose2pytest] 
> migration tool. As this required python 3 language support I found myself 
> down the 2to3 python migration path. While painful to do this, the benefits 
> of python3 over python2.7 are numerous and moving to python3 for the 
> additional debugging tools now available to use when fixing dtests makes the 
> effort worth it for that reason alone!
> After the automated tools did their thing I began what was a much longer and 
> tedious manual process than I ever could have expected due to the custom many 
> ways we did things in dtests (frequently to work around nosetest limitations 
> of missing features that thankfully are now all included with the pytest 
> framework). I've done nearly 300 test runs of my migration branch with 
> circleci.
> The latest CircleCI runs can be found at:
> (dtests without vnodes) [https://circleci.com/gh/mkjellman/cassandra/277]
> (dtests with vnodes) [https://circleci.com/gh/mkjellman/cassandra/278]
> With vnodes, there are currently only 6 remaining dtest test failures.
> Without vnodes, there are 12 remaining dtest failures.
> It turns out that after the dtests were moved to ASF Jenkins from cassci, the 
> jobs were misconfigured and we actually haven't been running the dtests in 
> the non-vnodes configuration. The current most recent trunk dtest job to 
> complete on ASF Jenkins (with vnodes) was 
> [https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-dtest/387/].
>  That test run had 36 test failures.
> There are great performance improvements so far with pytests. With a 
> parallism

[jira] [Updated] (CASSANDRA-14139) Acquire read lock before accessing CompactionStrategyManager fields

2017-12-29 Thread Paulo Motta (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-14139:

Status: Patch Available  (was: Open)

It seems we missed grabbing the read lock for some operations on 
CASSANDRA-13948, what can cause IndexOutOfBounds/NullPointers such the ones 
above during compaction strategy reload, so this patch basically gets the lock 
on the few methods were not doing it.

CI looks good. Trunk patch is slightly different because there are some 
additional methods due to CASSANDRA-9143. Can you take a look [~krummas]? 
Thanks!

||3.11||trunk||
|[branch|https://github.com/apache/cassandra/compare/cassandra-3.11...pauloricardomg:3.11-14139]|[branch|https://github.com/apache/cassandra/compare/trunk...pauloricardomg:trunk-14139]|
|[testall|https://issues.apache.org/jira/secure/attachment/12904036/3.11-14139-testall.png]|[testall|https://issues.apache.org/jira/secure/attachment/12904038/trunk-14139-testall.png]|
|[dtest|https://issues.apache.org/jira/secure/attachment/12904035/3.11-14139-dtest.png]|[dtest|https://issues.apache.org/jira/secure/attachment/12904037/trunk-14139-dtest.png]|


> Acquire read lock before accessing CompactionStrategyManager fields
> ---
>
> Key: CASSANDRA-14139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14139
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Attachments: 3.11-14139-dtest.png, 3.11-14139-testall.png, 
> trunk-14139-dtest.png, trunk-14139-testall.png
>
>
> There are a few methods in {{CompactionStrategyManager}} accessing the 
> repaired/unrepaired compaction strategy lists without using the read lock, 
> what could cause issues like the one below:
> {noformat}
> ERROR [CompactionExecutor:1] 2017-12-22 12:17:12,320 CassandraDaemon.java:141 
> - Exception in thread Thread[CompactionExecutor:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.supportsEarlyOpen(CompactionStrategyManager.java:1262)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.supportsEarlyOpen(ColumnFamilyStore.java:558)
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.construct(SSTableRewriter.java:119)
> at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.(CompactionAwareWriter.java:91)
> at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.(DefaultCompactionWriter.java:57)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.getCompactionAwareWriter(CompactionTask.java:293)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:90)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14139) Acquire read lock before accessing CompactionStrategyManager fields

2017-12-29 Thread Paulo Motta (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-14139:

Summary: Acquire read lock before accessing CompactionStrategyManager 
fields  (was: Avoid races during compaction strategy reload)

> Acquire read lock before accessing CompactionStrategyManager fields
> ---
>
> Key: CASSANDRA-14139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14139
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Attachments: 3.11-14139-dtest.png, 3.11-14139-testall.png, 
> trunk-14139-dtest.png, trunk-14139-testall.png
>
>
> There are a few methods in {{CompactionStrategyManager}} accessing the 
> repaired/unrepaired compaction strategy lists without using the read lock, 
> what could cause issues like the one below:
> {noformat}
> ERROR [CompactionExecutor:1] 2017-12-22 12:17:12,320 CassandraDaemon.java:141 
> - Exception in thread Thread[CompactionExecutor:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.supportsEarlyOpen(CompactionStrategyManager.java:1262)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.supportsEarlyOpen(ColumnFamilyStore.java:558)
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.construct(SSTableRewriter.java:119)
> at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.(CompactionAwareWriter.java:91)
> at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.(DefaultCompactionWriter.java:57)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.getCompactionAwareWriter(CompactionTask.java:293)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:90)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14139) Avoid races during compaction strategy reload

2017-12-29 Thread Paulo Motta (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Paulo Motta updated CASSANDRA-14139:

Attachment: trunk-14139-testall.png
trunk-14139-dtest.png
3.11-14139-testall.png
3.11-14139-dtest.png

> Avoid races during compaction strategy reload
> -
>
> Key: CASSANDRA-14139
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14139
> Project: Cassandra
>  Issue Type: Bug
>Reporter: Paulo Motta
>Assignee: Paulo Motta
> Attachments: 3.11-14139-dtest.png, 3.11-14139-testall.png, 
> trunk-14139-dtest.png, trunk-14139-testall.png
>
>
> There are a few methods in {{CompactionStrategyManager}} accessing the 
> repaired/unrepaired compaction strategy lists without using the read lock, 
> what could cause issues like the one below:
> {noformat}
> ERROR [CompactionExecutor:1] 2017-12-22 12:17:12,320 CassandraDaemon.java:141 
> - Exception in thread Thread[CompactionExecutor:1,5,main]
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at 
> org.apache.cassandra.db.compaction.CompactionStrategyManager.supportsEarlyOpen(CompactionStrategyManager.java:1262)
> at 
> org.apache.cassandra.db.ColumnFamilyStore.supportsEarlyOpen(ColumnFamilyStore.java:558)
> at 
> org.apache.cassandra.io.sstable.SSTableRewriter.construct(SSTableRewriter.java:119)
> at 
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.(CompactionAwareWriter.java:91)
> at 
> org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.(DefaultCompactionWriter.java:57)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.getCompactionAwareWriter(CompactionTask.java:293)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:200)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at 
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:90)
> at 
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:101)
> at 
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:310)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at 
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-13867) Make PartitionUpdate and Mutation immutable

2017-12-29 Thread Aleksey Yeschenko (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-13867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aleksey Yeschenko updated CASSANDRA-13867:
--
Status: Ready to Commit  (was: Patch Available)

> Make PartitionUpdate and Mutation immutable
> ---
>
> Key: CASSANDRA-13867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13867
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 4.x
>
>
> To avoid issues like CASSANDRA-13619 we should make PartitionUpdate and 
> Mutation immutable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Commented] (CASSANDRA-13867) Make PartitionUpdate and Mutation immutable

2017-12-29 Thread Aleksey Yeschenko (JIRA) 

[ 
https://issues.apache.org/jira/browse/CASSANDRA-13867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16306424#comment-16306424
 ] 

Aleksey Yeschenko commented on CASSANDRA-13867:
---


Nicely done. This will make the codebase a bit more sane and safe - +1 from me. 
Have 3 very minor notes and a few nits highlighted by IDEA:

1. {{BatchUpdatesCollector.toMutations()}} creates ands throws away an 
unnecessary list in each iteration of the loop. Could replace that for-loop 
body with something like {{ksMap.values().forEach(b -> ms.add(b.build()));}} 
instead.
2. {{SingleTableUpdatesCollector.toMutations()}} validates indexed columns, but 
{{BatchUpdatesCollector.toMutations()}} doesn’t. Also, I don’t feel like that 
validation code completely belongs to {{UpdatesCollector}} - at least the 
nested calls. Maybe should put {{validateIndexedColumns()}} in 
{{PartitionUpdate}} and in {{Mutation}}, encapsulated?
3. There are a few places duplicating the logic for {{cdcEnabled}} calculation. 
I feel like it might be better to move it back to {{Mutation}} constructor 
instead. 

The nits in no particular order:
- {{update}} variable in {{RowUpdateBuilder.buildUpdate()}} is redundant, IDEA 
is nagging about it
- {{BatchUpdatesCollector.IMutationBuilder}} has some javadoc problems that 
IDEA doesn’t like
- {{BatchUpdatesCollector.MutationBuilder.add()}} calls default impl of 
{{PartitionUpdate.Builder.toString()}} on duplicate add() - should override 
{{toString()}} to I guess, or just log {{prev.metadata().id}} or something 
instead.
- {{BatchUpdatesCollector.CounterMutationBuilder}} can also be private, like 
{{BatchUpdatesCollector.MutationBuilder}} already is
- {{SingleTableUpdatesCollector}} has a bunch of unused imports

> Make PartitionUpdate and Mutation immutable
> ---
>
> Key: CASSANDRA-13867
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13867
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Core
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 4.x
>
>
> To avoid issues like CASSANDRA-13619 we should make PartitionUpdate and 
> Mutation immutable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

[jira] [Updated] (CASSANDRA-14134) Migrate dtests to use pytest and python3

2017-12-29 Thread Michael Kjellman (JIRA) 

 [ 
https://issues.apache.org/jira/browse/CASSANDRA-14134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Kjellman updated CASSANDRA-14134:
-
Status: Patch Available  (was: Open)

> Migrate dtests to use pytest and python3
> 
>
> Key: CASSANDRA-14134
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14134
> Project: Cassandra
>  Issue Type: Improvement
>  Components: Testing
>Reporter: Michael Kjellman
>Assignee: Michael Kjellman
>
> h4. Get the C* dtests running on the pytest framework.
> C* DTests currently run using the python test framework nosetest. This 
> framework has been largely abandoned with no releases since 2015 and a 
> general strong consensus in the python community that pytest is the future.
> h4. Why should we do this.
> Currently (and historically) dtests have always been difficult to run, flaky 
> and unpredictable in CI environments, and almost impossible to debug.
> On November 28th, 2017, I proposed on the dev@ list that we move the dtests 
> from nosetests to pytests. I got replies from Jon Haddad, Philip Thompson, 
> and kurt greaves with really only "+1" like replies to the proposal.
> Since then I've been working pretty much non stop to complete the large 
> refactor of dtests to pytests. As part of this effort (and due to the 
> migration tools that exist require it) I also ported the code to python3 
> (from the current python 2.7 based code-base).
> h4. High-level summary of key changes, improvements, and new features.
> * Migrate dtests from executing using the nosetest framework to pytest
> * Port the entire code base from Python 2.7 to Python 3.6
> * Update run_dtests.py to work with pytest
> * Add --dtest-print-tests-only option to run_dtests.py to get easily parsable 
> list of all available collected tests
> * Update README.md for executing the dtests with pytest
> * Add new debugging tips section to README.md to help with some basics of 
> debugging python3 and pytest
> * Migrate all existing Enviornment Variable usage as a means to control dtest 
> operation modes to argparse command line options with documented help on each 
> toggles intended usage
> * Migration of old unitTest and nose based test structure to modern pytest 
> fixture approach
> * Automatic detection of physical system resources to automatically determine 
> if @pytest.mark.resource_intensive annotated tests should be collected and 
> run on the system where they are being executed
> * new pytest fixture replacements for @since and @pytest.mark.upgrade_test 
> annotations
> * Migration to python logging framework
> * Upgrade thrift bindings to latest version with full python3 compatibility
> * Remove deprecated cql and pycassa dependencies and migrate any remaining 
> tests to fully remove those dependencies
> * Fixed dozens of tests that would hang the pytest framework forever when run 
> in CI enviornments
> * Ran code nearly 300 times in CircleCI during the migration and to find, 
> identify, and fix any tests capable of hanging CI
> * Upgrade Tests do not yet run in CI and still need additional migration work 
> (although all upgrade test classes compile successfully)
> I started with the *nose2pytest* [https://github.com/pytest-dev/nose2pytest] 
> migration tool. As this required python 3 language support I found myself 
> down the 2to3 python migration path. While painful to do this, the benefits 
> of python3 over python2.7 are numerous and moving to python3 for the 
> additional debugging tools now available to use when fixing dtests makes the 
> effort worth it for that reason alone!
> After the automated tools did their thing I began what was a much longer and 
> tedious manual process than I ever could have expected due to the custom many 
> ways we did things in dtests (frequently to work around nosetest limitations 
> of missing features that thankfully are now all included with the pytest 
> framework). I've done nearly 300 test runs of my migration branch with 
> circleci.
> The latest CircleCI runs can be found at:
> (dtests without vnodes) [https://circleci.com/gh/mkjellman/cassandra/277]
> (dtests with vnodes) [https://circleci.com/gh/mkjellman/cassandra/278]
> With vnodes, there are currently only 6 remaining dtest test failures.
> Without vnodes, there are 12 remaining dtest failures.
> It turns out that after the dtests were moved to ASF Jenkins from cassci, the 
> jobs were misconfigured and we actually haven't been running the dtests in 
> the non-vnodes configuration. The current most recent trunk dtest job to 
> complete on ASF Jenkins (with vnodes) was 
> [https://builds.apache.org/view/A-D/view/Cassandra/job/Cassandra-trunk-dtest/387/].
>  That test run had 36 test failures.
> There are great performance improvements so far with pytests. With a 
> parallism factor of