[jira] [Commented] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[ https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932992#comment-16932992 ] maxwellguo commented on CASSANDRA-15092: Thank you [~jjirsa] [~brandon.williams] :) > Add a new Snitch for Alibaba Cloud Platform > --- > > Key: CASSANDRA-15092 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15092 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core, Local/Config >Reporter: maxwellguo >Assignee: maxwellguo >Priority: Normal > Labels: pull-request-available > Fix For: 4.0-alpha > > Attachments: trunk-15092.txt > > Time Spent: 20m > Remaining Estimate: 0h > > Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch > for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service) > meta data from here : > https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-14737) Limit the dependencies used by UDFs/UDAs
[
https://issues.apache.org/jira/browse/CASSANDRA-14737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932719#comment-16932719
]
Olivier Michallat commented on CASSANDRA-14737:
---
(non-binding) +1 to remove the dependency to the driver.
One indirect consequence of the current situation is that driver types are
leaked into UDF implementations. For example, given the following declaration:
{code}
CREATE FUNCTION test.first (t tuple) RETURNS NULL ON NULL INPUT
RETURNS int LANGUAGE java AS 'return t.getInt(0);';
{code}
The variable {{t}} in the method body is a {{TupleValue}}, a driver type. Same
for UDTs.
This is fine as long as Cassandra remains on version 3.x of the driver, but if
it upgrades to the latest (4.x), there are a few breaking changes to the API.
For example, the getters for temporal types now return {{java.time}} types
instead of {{java.util.Date}}. So upgrading the driver would break existing UDF
implementations.
Copying the code gives Cassandra full control over the API.
> Limit the dependencies used by UDFs/UDAs
>
>
> Key: CASSANDRA-14737
> URL: https://issues.apache.org/jira/browse/CASSANDRA-14737
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/CQL
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Low
> Labels: UDF
> Fix For: 4.0
>
>
> In an effort to clean up our hygiene and limit the dependencies used by
> UDFs/UDAs, I think we should refactor the UDF code parts and remove the
> dependency to the Java Driver in that area without breaking existing
> UDFs/UDAs.
>
> The patch is in [this
> branch|https://github.com/snazy/cassandra/tree/feature/remove-udf-driver-dep-trunk].
> The changes are rather trivial and provide 100% backwards compatibility for
> existing UDFs.
>
> The prototype copies the necessary parts from the Java Driver into the C*
> source tree to {{org.apache.cassandra.cql3.functions.types}} and adopts its
> usages - i.e. UDF/UDA code plus {{CQLSSTableWriter}} +
> {{StressCQLSSTableWriter}}. The latter two classes have a reference to UDF's
> {{UDHelper}} and had to be changed as well.
>
> Some functionality, like type parsing & handling, is duplicated in the code
> base with this prototype - once in the "current" source tree and once for
> UDFs. However, unifying the code paths is not trivial, since the UDF sandbox
> prohibits the use of internal classes (direct and likely indirect
> dependencies).
>
> /cc [~jbellis]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[
https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931810#comment-16931810
]
Jordan West edited comment on CASSANDRA-15295 at 9/18/19 5:10 PM:
--
Happy to [~gzh1992n]. A few comments:
* I verified it does not affect 3.0 branch because
{{CommitLogSegmentManager#start}} exits immediately after starting
{{managerThread}} instead of calling {{advaceAllocatingFrom(null)}};
* The database doesn’t start, which causes many tests to fail as well, because
there is no default commit log segment manager factory set. Test runs:
https://circleci.com/gh/jrwest/cassandra/tree/bug-commitlog-deadlock
* CommitLogInitWithExpcetionTest#L63 - should check prior to this call that
initThread is not null
Minor naming nits (do with them what you please):
* Rename KillerHook => OnKillHook, and onKill => execute
* Drop the “I*” naming for the CommitLogSegmentMgrFactoryInterface. Consider
renaming it CommitLogSegmentManagerFactory
was (Author: jrwest):
Happy to [~gzh1992n]. A few comments:
* I verified it does not affect 3.0 branch because
CommitLogSegmentManager#start exits immediately after starting managerThread
instead of calling advaceAllocatingFrom(null);
* The database doesn’t start, which causes many tests to fail as well, because
there is no default commit log segment manager factory set. Test runs:
https://circleci.com/gh/jrwest/cassandra/tree/bug-commitlog-deadlock
* CommitLogInitWithExpcetionTest#L63 - should check prior to this call that
initThread is not null
Minor naming nits (do with them what you please):
* Rename KillerHook => OnKillHook, and onKill => execute
* Drop the “I*” naming for the CommitLogSegmentMgrFactoryInterface. Consider
renaming it CommitLogSegmentManagerFactory
> Running into deadlock when do CommitLog initialization
> --
>
> Key: CASSANDRA-15295
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15295
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Commit Log
>Reporter: Zephyr Guo
>Assignee: Zephyr Guo
>Priority: Normal
> Attachments: jstack.log, pstack.log, screenshot-1.png,
> screenshot-2.png, screenshot-3.png
>
>
> Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a
> long time.
> I used jstack to saw what happened. The main thread stuck in
> *AbstractCommitLogSegmentManager.awaitAvailableSegment*
> !screenshot-1.png!
> The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it
> was not actually running.
> !screenshot-2.png!
> And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on
> java class initialization.
> !screenshot-3.png!
> This is a deadlock obviously. CommitLog waits for a CommitLogSegment when
> initializing. In this moment, the CommitLog class is not initialized and the
> main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a
> CommitLogSegment with exception and call *CommitLog.handleCommitError*(static
> method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog
> class is still initializing.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[
https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Jirsa updated CASSANDRA-15092:
---
Fix Version/s: 4.0-alpha
Source Control Link:
https://github.com/apache/cassandra/commit/a7a0e7fe40cd375c647d2f5e56d04fd98963cff2
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
Committed. I fixed the log message, and some code style issues (tab instead of
spaces in a few places).
Noting for the record, that I asked in {{#cassandra-dev}} ASF slack on Sept 3
to see if anyone objected to landing this in the alpha since it's "new" and
received no objections (Benedict said "I don’t really consider a snitch for a
cloud provider to be a new feature, but more of a plugin", Aleksey said
"¯\_(ツ)_/¯ I don’t really mind having it in, and am fine with dropping it in
4.0 - it’s entirely self-contained after all", and Josh said "Maybe include but
mark experimental for a cycle?")
> Add a new Snitch for Alibaba Cloud Platform
> ---
>
> Key: CASSANDRA-15092
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15092
> Project: Cassandra
> Issue Type: New Feature
> Components: Legacy/Core, Local/Config
>Reporter: maxwellguo
>Assignee: maxwellguo
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.0-alpha
>
> Attachments: trunk-15092.txt
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch
> for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service)
> meta data from here :
> https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[ https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15092: --- Status: Ready to Commit (was: Review In Progress) > Add a new Snitch for Alibaba Cloud Platform > --- > > Key: CASSANDRA-15092 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15092 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core, Local/Config >Reporter: maxwellguo >Assignee: maxwellguo >Priority: Normal > Labels: pull-request-available > Attachments: trunk-15092.txt > > Time Spent: 20m > Remaining Estimate: 0h > > Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch > for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service) > meta data from here : > https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[ https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Jirsa updated CASSANDRA-15092: --- Reviewers: Brandon Williams, Jeff Jirsa Status: Review In Progress (was: Patch Available) > Add a new Snitch for Alibaba Cloud Platform > --- > > Key: CASSANDRA-15092 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15092 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core, Local/Config >Reporter: maxwellguo >Assignee: maxwellguo >Priority: Normal > Labels: pull-request-available > Attachments: trunk-15092.txt > > Time Spent: 20m > Remaining Estimate: 0h > > Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch > for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service) > meta data from here : > https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[cassandra] branch trunk updated: Add a new snitch for Alibaba cloud platform
This is an automated email from the ASF dual-hosted git repository.
jjirsa pushed a commit to branch trunk
in repository https://gitbox.apache.org/repos/asf/cassandra.git
The following commit(s) were added to refs/heads/trunk by this push:
new a7a0e7f Add a new snitch for Alibaba cloud platform
a7a0e7f is described below
commit a7a0e7fe40cd375c647d2f5e56d04fd98963cff2
Author: xuanling.gc
AuthorDate: Tue Sep 3 11:13:42 2019 +0800
Add a new snitch for Alibaba cloud platform
Closes #350
Patch by xuanling.gc, reviewed by Brandon Williams for CASSANDRA-15092
---
CHANGES.txt| 2 +
.../cassandra/locator/AlibabaCloudSnitch.java | 146 +
.../cassandra/locator/AlibabaCloudSnitchTest.java | 103 +++
3 files changed, 251 insertions(+)
diff --git a/CHANGES.txt b/CHANGES.txt
index 709e436..2929a98 100644
--- a/CHANGES.txt
+++ b/CHANGES.txt
@@ -1,5 +1,7 @@
4.0-alpha2
* Add `allocate_tokens_for_local_replication_factor` option for token
allocation (CASSANDRA-15260)
+ * Add Alibaba Cloud Platform snitch (CASSANDRA-15092)
+
4.0-alpha1
* Inaccurate exception message with nodetool snapshot (CASSANDRA-15287)
diff --git a/src/java/org/apache/cassandra/locator/AlibabaCloudSnitch.java
b/src/java/org/apache/cassandra/locator/AlibabaCloudSnitch.java
new file mode 100644
index 000..729e1b3
--- /dev/null
+++ b/src/java/org/apache/cassandra/locator/AlibabaCloudSnitch.java
@@ -0,0 +1,146 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.cassandra.locator;
+
+import java.io.DataInputStream;
+import java.io.FilterInputStream;
+import java.io.IOException;
+import java.net.HttpURLConnection;
+import java.net.MalformedURLException;
+import java.net.SocketTimeoutException;
+import java.net.URL;
+import java.nio.charset.StandardCharsets;
+import java.util.Map;
+import org.apache.cassandra.db.SystemKeyspace;
+import org.apache.cassandra.exceptions.ConfigurationException;
+import org.apache.cassandra.gms.ApplicationState;
+import org.apache.cassandra.gms.EndpointState;
+import org.apache.cassandra.gms.Gossiper;
+import org.apache.cassandra.io.util.FileUtils;
+import org.apache.cassandra.utils.FBUtilities;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A snitch that assumes an ECS region is a DC and an ECS availability_zone
+ * is a rack. This information is available in the config for the node. the
+ * format of the zone-id is like :cn-hangzhou-a where cn means china, hangzhou
+ * means the hangzhou region, a means the az id. We use cn-hangzhou as the dc,
+ * and f as the zone-id.
+ */
+public class AlibabaCloudSnitch extends AbstractNetworkTopologySnitch
+{
+protected static final Logger logger =
LoggerFactory.getLogger(AlibabaCloudSnitch.class);
+protected static final String ZONE_NAME_QUERY_URL =
"http://100.100.100.200/latest/meta-data/zone-id";;
+private static final String DEFAULT_DC = "UNKNOWN-DC";
+private static final String DEFAULT_RACK = "UNKNOWN-RACK";
+private Map> savedEndpoints;
+protected String ecsZone;
+protected String ecsRegion;
+
+private static final int HTTP_CONNECT_TIMEOUT = 3;
+
+
+public AlibabaCloudSnitch() throws MalformedURLException, IOException
+{
+String response = alibabaApiCall(ZONE_NAME_QUERY_URL);
+String[] splits = response.split("/");
+String az = splits[splits.length - 1];
+
+// Split "us-central1-a" or "asia-east1-a" into "us-central1"/"a" and
"asia-east1"/"a".
+splits = az.split("-");
+ecsZone = splits[splits.length - 1];
+
+int lastRegionIndex = az.lastIndexOf("-");
+ecsRegion = az.substring(0, lastRegionIndex);
+
+String datacenterSuffix = (new SnitchProperties()).get("dc_suffix",
"");
+ecsRegion = ecsRegion.concat(datacenterSuffix);
+logger.info("AlibabaSnitch using region: {}, zone: {}.", ecsRegion,
ecsZone);
+
+}
+
+String alibabaApiCall(String url) throws ConfigurationException,
IOException, SocketTimeoutException
+{
+// Populate the region and zone by introspection, fail if 404 on
me
[jira] [Commented] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[ https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932589#comment-16932589 ] Brandon Williams commented on CASSANDRA-15092: -- I reviewed, but my commit machinery is old and defunct at this point. Looks good, but I will bikeshed the error here: [https://github.com/apache/cassandra/pull/350/commits/cc8df9503acb2258c95278626dccadaddd5f712c#diff-8b7f8aed7f8ac62a3ccaa677bf9dd755R101] to something like "Timeout occurred reading a response from the Alibaba ECS metadata", otherwise +1. > Add a new Snitch for Alibaba Cloud Platform > --- > > Key: CASSANDRA-15092 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15092 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core, Local/Config >Reporter: maxwellguo >Assignee: maxwellguo >Priority: Normal > Labels: pull-request-available > Attachments: trunk-15092.txt > > Time Spent: 20m > Remaining Estimate: 0h > > Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch > for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service) > meta data from here : > https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15092) Add a new Snitch for Alibaba Cloud Platform
[ https://issues.apache.org/jira/browse/CASSANDRA-15092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932353#comment-16932353 ] maxwellguo commented on CASSANDRA-15092: [~jjirsa] thank you for you reply . [~brandon.williams] If you got some time ,can you take a look at this pacth agagin ? > Add a new Snitch for Alibaba Cloud Platform > --- > > Key: CASSANDRA-15092 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15092 > Project: Cassandra > Issue Type: New Feature > Components: Legacy/Core, Local/Config >Reporter: maxwellguo >Assignee: maxwellguo >Priority: Normal > Labels: pull-request-available > Attachments: trunk-15092.txt > > Time Spent: 20m > Remaining Estimate: 0h > > Add snitch for Alibaba cloud platform, for we have saw cloud platform snitch > for aws and google cloud ,for we can ge alibaba ecs (Elastic Compute Service) > meta data from here : > https://help.aliyun.com/document_detail/108460.html?spm=a2c4g.11186623.6.675.36684f8bLQrIMY -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15295) Running into deadlock when do CommitLog initialization
[ https://issues.apache.org/jira/browse/CASSANDRA-15295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16932337#comment-16932337 ] Zephyr Guo commented on CASSANDRA-15295: Thanks for the comments [~jrwest]. I will update the patch as soon as possible. > Running into deadlock when do CommitLog initialization > -- > > Key: CASSANDRA-15295 > URL: https://issues.apache.org/jira/browse/CASSANDRA-15295 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Zephyr Guo >Assignee: Zephyr Guo >Priority: Normal > Attachments: jstack.log, pstack.log, screenshot-1.png, > screenshot-2.png, screenshot-3.png > > > Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a > long time. > I used jstack to saw what happened. The main thread stuck in > *AbstractCommitLogSegmentManager.awaitAvailableSegment* > !screenshot-1.png! > The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it > was not actually running. > !screenshot-2.png! > And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on > java class initialization. > !screenshot-3.png! > This is a deadlock obviously. CommitLog waits for a CommitLogSegment when > initializing. In this moment, the CommitLog class is not initialized and the > main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a > CommitLogSegment with exception and call *CommitLog.handleCommitError*(static > method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog > class is still initializing. > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
