Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3758633939 [SF] Trillian test result (tid-15191) Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8 Total time taken: 55310 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12124-t15191-kvm-ol8.zip Smoke tests completed. 148 look OK, 2 have errors, 0 did not run Only failed and skipped tests results shown below: Test | Result | Time (s) | Test File --- | --- | --- | --- test_03_deploy_and_scale_kubernetes_cluster | `Failure` | 42.19 | test_kubernetes_clusters.py test_01_create_redundant_VPC_2tiers_4VMs_4IPs_4PF_ACL | `Failure` | 293.88 | test_vpc_redundant.py -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3755508384 @abh1sar a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
abh1sar commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3755505240 @blueorangutan test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3754787337 Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16381 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3754489362 @mhkadhum a [SL] Jenkins job has been kicked to build packages. It will be bundled withHello @DaanHoogland , Can u test, I believe this is ready to merge. SystemVM template(s). I'll keep you posted as I make progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3754484297 > @blueorangutan package Hello @DaanHoogland , Can u test, I believe this is ready to merge. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3754200819 Packaging result [SF]: ✖️ el8 ✖️ el9 ✔️ debian ✖️ suse15. SL-JID 16379 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3753969163 @blueorangutan package -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3753988844 @DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3753497119 Hello @DaanHoogland , Can u approve the pipeline ? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2689231133
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsConstants.java:
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cloudstack.storage.datastore.driver;
+
+public final class EcsConstants {
+private EcsConstants() {}
Review Comment:
I am keeping this just to follow the standard pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
abh1sar commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748034686 > > > @mhkadhum , it looks like you are ready to merge. If you wish to ignore my latest comments please let us know. (cc @abh1sar ). > > > > > > Lets go ahead with the merge > > Thank you > > I am going to do some regression testing. Should be finished by tomorrow. Tested common object storage workflows with Ceph. Add/Delete Object Store. Create/Update/Delete Bucket. Working as expected. @mhkadhum Please answer any pending comments and we can move forward with this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2689226429
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -137,68 +93,88 @@ public DataStoreTO getStoreTO(final DataStore store) {
@Override
public Bucket createBucket(final Bucket bucket, final boolean objectLock) {
final long storeId = bucket.getObjectStoreId();
-final String name = bucket.getName();
+final String name = bucket.getName();
if (objectLock) {
-throw new CloudRuntimeException("Dell ECS doesn't support this
feature: object locking");
+throw new InvalidParameterValueException("Dell ECS doesn't support
this feature: object locking");
}
final Map ds = storeDetailsDao.getDetails(storeId);
final EcsCfg cfg = ecsCfgFromDetails(ds, storeId);
-// Resolve owner username for this bucket
final BucketVO vo = bucketDao.findById(bucket.getId());
+if (vo == null) {
+throw new CloudRuntimeException("ECS createBucket: bucket record
not found: id=" + bucket.getId());
+}
+
final long accountId = vo.getAccountId();
final Account acct = accountDao.findById(accountId);
if (acct == null) {
throw new CloudRuntimeException("ECS createBucket: account not
found: id=" + accountId);
}
-final String ownerUser = "cs-" + acct.getUuid();
+
+final String ownerUser = getUserPrefix(ds) + acct.getUuid();
// Ensure per-account credentials exist (single-key policy with
adopt-if-exists)
ensureAccountUserAndSecret(accountId, ownerUser, cfg.mgmtUrl,
cfg.saUser, cfg.saPass, cfg.ns, cfg.insecure);
-// Quota from UI (INT GB)
+// Quota from UI (INT GB). Bucket.getQuota may be Integer;
Bucket.getSize may be Long.
Integer quotaGb = null;
try {
Review Comment:
https://github.com/mhkadhum/cloudstack/commit/6f3efbefc9ae8ba2ab62e00e1e0df472fd5beca0
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -137,68 +93,88 @@ public DataStoreTO getStoreTO(final DataStore store) {
@Override
public Bucket createBucket(final Bucket bucket, final boolean objectLock) {
final long storeId = bucket.getObjectStoreId();
-final String name = bucket.getName();
+final String name = bucket.getName();
if (objectLock) {
-throw new CloudRuntimeException("Dell ECS doesn't support this
feature: object locking");
+throw new InvalidParameterValueException("Dell ECS doesn't support
this feature: object locking");
}
final Map ds = storeDetailsDao.getDetails(storeId);
final EcsCfg cfg = ecsCfgFromDetails(ds, storeId);
-// Resolve owner username for this bucket
final BucketVO vo = bucketDao.findById(bucket.getId());
+if (vo == null) {
+throw new CloudRuntimeException("ECS createBucket: bucket record
not found: id=" + bucket.getId());
+}
+
final long accountId = vo.getAccountId();
final Account acct = accountDao.findById(accountId);
if (acct == null) {
throw new CloudRuntimeException("ECS createBucket: account not
found: id=" + accountId);
}
-final String ownerUser = "cs-" + acct.getUuid();
+
+final String ownerUser = getUserPrefix(ds) + acct.getUuid();
// Ensure per-account credentials exist (single-key policy with
adopt-if-exists)
ensureAccountUserAndSecret(accountId, ownerUser, cfg.mgmtUrl,
cfg.saUser, cfg.saPass, cfg.ns, cfg.insecure);
-// Quota from UI (INT GB)
+// Quota from UI (INT GB). Bucket.getQuota may be Integer;
Bucket.getSize may be Long.
Integer quotaGb = null;
try {
-quotaGb = safeIntFromGetter(bucket, "getQuota");
-if (quotaGb == null) quotaGb = safeIntFromGetter(bucket,
"getSize");
-} catch (Throwable ignored) { }
+quotaGb = bucket.getQuota();
+} catch (Throwable ignored) {
+}
-final int blockSizeGb = quotaGb != null && quotaGb > 0 ? quotaGb : 2;
-final int notifSizeGb = quotaGb != null && quotaGb > 0 ? quotaGb : 1;
+if (quotaGb == null) {
+try {
+final Long sz = bucket.getSize();
+if (sz != null) {
+quotaGb = sz.intValue();
+}
+} catch (Throwable ignored) {
+}
+}
+
+final int blockSizeGb;
+final int notifSizeGb;
+if (quotaGb != null && quotaGb > 0) {
+blockSizeGb = quotaGb;
+notifSizeGb = quotaGb;
+} else {
+blockSizeGb = 2;
+notifSizeGb = 1;
+
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748219868 @blueorangutan package -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748223226 @DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
abh1sar commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748144769 @blueorangutan package -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748561114 @abh1sar a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748070850 > > > > @mhkadhum , it looks like you are ready to merge. If you wish to ignore my latest comments please let us know. (cc @abh1sar ). > > > > > > > > > Lets go ahead with the merge > > > Thank you > > > > > > I am going to do some regression testing. Should be finished by tomorrow. > > Tested common object storage workflows with Ceph. Add/Delete Object Store. Create/Update/Delete Bucket. Working as expected. > > @mhkadhum Please answer any pending comments and we can move forward with this. Hi @abh1sar , I have addressed all queries and comments.I believe we are ready to proceed with this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
abh1sar commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2663735171
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsConstants.java:
##
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cloudstack.storage.datastore.driver;
+
+public final class EcsConstants {
+private EcsConstants() {}
Review Comment:
Don't think constructor is required here
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -137,68 +93,88 @@ public DataStoreTO getStoreTO(final DataStore store) {
@Override
public Bucket createBucket(final Bucket bucket, final boolean objectLock) {
final long storeId = bucket.getObjectStoreId();
-final String name = bucket.getName();
+final String name = bucket.getName();
if (objectLock) {
-throw new CloudRuntimeException("Dell ECS doesn't support this
feature: object locking");
+throw new InvalidParameterValueException("Dell ECS doesn't support
this feature: object locking");
}
final Map ds = storeDetailsDao.getDetails(storeId);
final EcsCfg cfg = ecsCfgFromDetails(ds, storeId);
-// Resolve owner username for this bucket
final BucketVO vo = bucketDao.findById(bucket.getId());
+if (vo == null) {
+throw new CloudRuntimeException("ECS createBucket: bucket record
not found: id=" + bucket.getId());
+}
+
final long accountId = vo.getAccountId();
final Account acct = accountDao.findById(accountId);
if (acct == null) {
throw new CloudRuntimeException("ECS createBucket: account not
found: id=" + accountId);
}
-final String ownerUser = "cs-" + acct.getUuid();
+
+final String ownerUser = getUserPrefix(ds) + acct.getUuid();
// Ensure per-account credentials exist (single-key policy with
adopt-if-exists)
ensureAccountUserAndSecret(accountId, ownerUser, cfg.mgmtUrl,
cfg.saUser, cfg.saPass, cfg.ns, cfg.insecure);
-// Quota from UI (INT GB)
+// Quota from UI (INT GB). Bucket.getQuota may be Integer;
Bucket.getSize may be Long.
Integer quotaGb = null;
try {
-quotaGb = safeIntFromGetter(bucket, "getQuota");
-if (quotaGb == null) quotaGb = safeIntFromGetter(bucket,
"getSize");
-} catch (Throwable ignored) { }
+quotaGb = bucket.getQuota();
+} catch (Throwable ignored) {
+}
-final int blockSizeGb = quotaGb != null && quotaGb > 0 ? quotaGb : 2;
-final int notifSizeGb = quotaGb != null && quotaGb > 0 ? quotaGb : 1;
+if (quotaGb == null) {
+try {
+final Long sz = bucket.getSize();
+if (sz != null) {
+quotaGb = sz.intValue();
+}
+} catch (Throwable ignored) {
+}
+}
+
+final int blockSizeGb;
+final int notifSizeGb;
+if (quotaGb != null && quotaGb > 0) {
+blockSizeGb = quotaGb;
+notifSizeGb = quotaGb;
+} else {
+blockSizeGb = 2;
+notifSizeGb = 1;
+}
-// Encryption flag from request/VO best-effort
-boolean encryptionEnabled =
-getBooleanFlagLoose(bucket, "getEncryption", "isEncryption",
false) ||
-getBooleanFlagLoose(bucket, "getEncryptionEnabled",
"isEncryptionEnabled", false);
+// Encryption flag from request (Bucket has isEncryption()).
+boolean encryptionEnabled = bucket.isEncryption();
-if (!encryptionEnabled && vo != null) {
-encryptionEnabled =
-getBooleanFlagLoose(vo, "getEncryption", "isEncryption",
false) ||
-getBooleanFlagLoose(vo, "getEncryptionEnabled",
"isEncryptionEnabled", false);
+// Fallback to persisted value if request did not explicitly enable it.
+if (!encryptionEnabled) {
+try {
Review Comment:
this try
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
abh1sar commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748554631 @blueorangutan package -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
abh1sar commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748317834 @mhkadhum please fix build errors `EcsObjectStoreDriverImpl.java:120: Line has trailing spaces. [RegexpSingleline]` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748422810 Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 16355 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3748407545 > @mhkadhum please fix build errors `EcsObjectStoreDriverImpl.java:120: Line has trailing spaces. [RegexpSingleline]` Missed the pre-commit hook. Fixed and pushed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
abh1sar commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3717467798 > > @mhkadhum , it looks like you are ready to merge. If you wish to ignore my latest comments please let us know. (cc @abh1sar ). > > Lets go ahead with the merge > Thank you I am going to do some regression testing. Should be finished by tomorrow. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3717459307 > @mhkadhum , it looks like you are ready to merge. If you wish to ignore my latest comments please let us know. (cc @abh1sar ). Lets go ahead with the merge Thank you -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2667188340
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsMgmtTokenManager.java:
##
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.util.EntityUtils;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsMgmtTokenManager {
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300;
+private static final long EXPIRY_SKEW_SEC = 30;
+
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+static final class EcsUnauthorizedException extends RuntimeException {
+EcsUnauthorizedException(final String msg) { super(msg); }
+}
+
+@FunctionalInterface
+public interface WithToken { T run(String token) throws Exception; }
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
+}
+
+private static final class TokenEntry {
+final String token;
+final long expiresAtMs;
+TokenEntry(final String token, final long expiresAtMs) {
+this.token = token;
+this.expiresAtMs = expiresAtMs;
+}
+boolean validNow() {
+return token != null && !token.isBlank() &&
System.currentTimeMillis() < expiresAtMs;
+}
+}
+
+public T callWithRetry401(final EcsObjectStoreDriverImpl.EcsCfg cfg,
+ final WithToken op,
+ final HttpClientFactory httpFactory) throws
Exception {
+try {
+return op.run(getAuthToken(cfg, httpFactory));
+} catch (EcsUnauthorizedException u) {
+invalidate(cfg);
+return op.run(getAuthToken(cfg, httpFactory));
+}
+}
+
+public void invalidate(final EcsObjectStoreDriverImpl.EcsCfg cfg) {
+TOKEN_CACHE.remove(new TokenKey(trimTail(cfg.mgmtUrl), cfg.saUser));
+}
+
+public String getAuthToken(final EcsObjectStoreDriverImpl.EcsCfg cfg,
+ final HttpClientFactory httpFactory) {
+final String mu = trimTail(cfg.mgmtUrl);
+final TokenKey key = new TokenKey(mu, cfg.saUser);
+
+final TokenEntry cached = TOKEN_CACHE.get(key);
+if (cached != null && cached.validNow()) return cached.token;
+
+final Object lock = TOKEN_LOCKS.computeIfAbsent(key, k -> new
Object());
+synchronized (lock) {
+final TokenEntry cached2 = TOKEN_CACHE.get(key);
+if (cached2 != null && cached2.validNow()) return cached2.token;
+
+final TokenEntry fresh = loginAndGetTokenFresh(mu, cfg.saUser,
cfg.saPass, cfg.insecure, httpFactory);
+TOKEN_CACHE.put(key, fresh);
+return fresh.token;
+}
+}
+
+private TokenEntry loginAndGetTokenFresh(final String mgmtUrl,
Review Comment:
the first two try-with-resources blocks which have a finally to close the
resources so they dont need a catch or finally. The third try is only for
pa
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3713695598 @mhkadhum , it looks like you are ready to merge. If you wish to ignore my latest comments please let us know. (cc @abh1sar ). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2664086469
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsMgmtTokenManager.java:
##
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.util.EntityUtils;
+
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsMgmtTokenManager {
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300;
+private static final long EXPIRY_SKEW_SEC = 30;
+
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+static final class EcsUnauthorizedException extends RuntimeException {
+EcsUnauthorizedException(final String msg) { super(msg); }
+}
+
+@FunctionalInterface
+public interface WithToken { T run(String token) throws Exception; }
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
+}
+
+private static final class TokenEntry {
+final String token;
+final long expiresAtMs;
+TokenEntry(final String token, final long expiresAtMs) {
+this.token = token;
+this.expiresAtMs = expiresAtMs;
+}
+boolean validNow() {
+return token != null && !token.isBlank() &&
System.currentTimeMillis() < expiresAtMs;
+}
+}
+
+public T callWithRetry401(final EcsObjectStoreDriverImpl.EcsCfg cfg,
+ final WithToken op,
+ final HttpClientFactory httpFactory) throws
Exception {
+try {
+return op.run(getAuthToken(cfg, httpFactory));
+} catch (EcsUnauthorizedException u) {
+invalidate(cfg);
+return op.run(getAuthToken(cfg, httpFactory));
+}
+}
+
+public void invalidate(final EcsObjectStoreDriverImpl.EcsCfg cfg) {
+TOKEN_CACHE.remove(new TokenKey(trimTail(cfg.mgmtUrl), cfg.saUser));
+}
+
+public String getAuthToken(final EcsObjectStoreDriverImpl.EcsCfg cfg,
+ final HttpClientFactory httpFactory) {
+final String mu = trimTail(cfg.mgmtUrl);
+final TokenKey key = new TokenKey(mu, cfg.saUser);
+
+final TokenEntry cached = TOKEN_CACHE.get(key);
+if (cached != null && cached.validNow()) return cached.token;
+
+final Object lock = TOKEN_LOCKS.computeIfAbsent(key, k -> new
Object());
+synchronized (lock) {
+final TokenEntry cached2 = TOKEN_CACHE.get(key);
+if (cached2 != null && cached2.validNow()) return cached2.token;
+
+final TokenEntry fresh = loginAndGetTokenFresh(mu, cfg.saUser,
cfg.saPass, cfg.insecure, httpFactory);
+TOKEN_CACHE.put(key, fresh);
+return fresh.token;
+}
+}
+
+private TokenEntry loginAndGetTokenFresh(final String mgmtUrl,
Review Comment:
this method contains three levels of try blocks. the second level has no
catch or final blocks. @mhkadhum , are you sure about this? I’d try to dis
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3713197161 [SF] Trillian test result (tid-15110) Environment: kvm-ol8 (x2), zone: Advanced Networking with Mgmt server ol8 Total time taken: 60102 seconds Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr12124-t15110-kvm-ol8.zip Smoke tests completed. 149 look OK, 1 have errors, 0 did not run Only failed and skipped tests results shown below: Test | Result | Time (s) | Test File --- | --- | --- | --- test_03_deploy_and_scale_kubernetes_cluster | `Failure` | 27.81 | test_kubernetes_clusters.py -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3709985045 @DaanHoogland a [SL] Trillian-Jenkins test job (ol8 mgmt + kvm-ol8) has been kicked to run smoke tests -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3709980744 @blueorangutan test -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3709809927 Packaging result [SF]: ✔️ el8 ✔️ el9 ✔️ el10 ✔️ debian ✔️ suse15. SL-JID 16248 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3709465396 @DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3709460841 @blueorangutan package -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2659513400
##
ui/src/views/storage/UpdateBucket.vue:
##
@@ -110,36 +132,31 @@ export default {
this.loading = true
Object.keys(this.apiParams).forEach(item => {
const field = this.apiParams[item]
-let fieldValue = null
-let fieldName = null
-
+let fieldName = field.name
if (field.type === 'list' || field.name === 'account') {
fieldName = field.name.replace('ids', 'name').replace('id', 'name')
-} else {
- fieldName = field.name
}
-fieldValue = this.resource[fieldName] ? this.resource[fieldName] : null
-if (fieldValue) {
+const fieldValue = this.resource[fieldName] ?? null
+if (fieldValue !== null && fieldValue !== undefined) {
form[field.name] = fieldValue
}
})
this.loading = false
},
-handleSubmit (e) {
+handleSubmit () {
if (this.loading) return
this.formRef.value.validate().then(() => {
-const formRaw = toRaw(this.form)
-const values = this.handleRemoveFields(formRaw)
-
-var data = {
+const values = toRaw(this.form)
Review Comment:
This was removed by mistake during the refactoring. I’ve restored the
original handleRemoveFields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2659500263
##
ui/src/views/storage/UpdateBucket.vue:
##
@@ -110,36 +132,31 @@ export default {
this.loading = true
Object.keys(this.apiParams).forEach(item => {
const field = this.apiParams[item]
-let fieldValue = null
-let fieldName = null
-
+let fieldName = field.name
if (field.type === 'list' || field.name === 'account') {
fieldName = field.name.replace('ids', 'name').replace('id', 'name')
-} else {
- fieldName = field.name
}
-fieldValue = this.resource[fieldName] ? this.resource[fieldName] : null
-if (fieldValue) {
+const fieldValue = this.resource[fieldName] ?? null
+if (fieldValue !== null && fieldValue !== undefined) {
Review Comment:
u r correct with
`
const fieldValue = this.resource[fieldName] ?? null
`
fieldValue can never be undefined.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2659493373
##
ui/src/views/storage/CreateBucket.vue:
##
@@ -110,18 +118,56 @@ export default {
inject: ['parentFetchData'],
data () {
return {
- loading: false
+ loading: false,
+ objectstores: [],
+ policyList: ['Public', 'Private']
+}
+ },
+ computed: {
+isEcsObjectStore () {
+ const selectedId = this.form?.objectstore
+ const stores = this.objectstores || []
+ const selected = stores.find(os => os.id === selectedId)
+ if (!selected) {
+return false
+ }
+
+ const provider = (
+selected.objectstoreprovider ||
+selected.objectStoreProvider ||
+selected.provider ||
+''
+ ).toString().toUpperCase()
+
+ const name = (selected.name || '').toString().toUpperCase()
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2659491716
##
ui/src/views/storage/CreateBucket.vue:
##
@@ -110,18 +118,56 @@ export default {
inject: ['parentFetchData'],
data () {
return {
- loading: false
+ loading: false,
+ objectstores: [],
+ policyList: ['Public', 'Private']
+}
+ },
+ computed: {
+isEcsObjectStore () {
+ const selectedId = this.form?.objectstore
+ const stores = this.objectstores || []
+ const selected = stores.find(os => os.id === selectedId)
+ if (!selected) {
+return false
+ }
+
+ const provider = (
+selected.objectstoreprovider ||
Review Comment:
my code was appearing to work before because it relied on fallbacks
objectstoreprovider, provider, and even the object store name containing ECS
However that behavior was not correct
it updated now and uses providername
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2659489345
##
ui/src/views/infra/AddObjectStorage.vue:
##
@@ -165,26 +240,66 @@ export default {
const formRaw = toRaw(this.form)
const values = this.handleRemoveFields(formRaw)
-var data = {
- name: values.name,
- size: values.size
+const data = {
+ name: values.name
Review Comment:
ok
##
ui/src/views/infra/AddObjectStorage.vue:
##
@@ -67,16 +68,82 @@
-
+
https://s3-hostname or http://s3-hostname"/>
-
+
https://iam-hostname:16443 or http://iam-hostname:16080"/>
+
+
+
+
+
+
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2659481421
##
ui/src/views/infra/AddObjectStorage.vue:
##
@@ -165,26 +240,66 @@ export default {
const formRaw = toRaw(this.form)
const values = this.handleRemoveFields(formRaw)
-var data = {
- name: values.name,
- size: values.size
+const data = {
+ name: values.name
}
-var provider = values.provider
+const provider = values.provider
data.provider = provider
data.url = values.url
-data['details[0].key'] = 'accesskey'
-data['details[0].value'] = values.accessKey
-data['details[1].key'] = 'secretkey'
-data['details[1].value'] = values.secretKey
if (provider === 'Cloudian HyperStore') {
+ // Cloudian HyperStore details
+ data['details[0].key'] = 'accesskey'
+ data['details[0].value'] = values.accessKey
+ data['details[1].key'] = 'secretkey'
+ data['details[1].value'] = values.secretKey
data['details[2].key'] = 'validateSSL'
data['details[2].value'] = values.validateSSL
data['details[3].key'] = 's3Url'
data['details[3].value'] = values.s3Url
data['details[4].key'] = 'iamUrl'
data['details[4].value'] = values.iamUrl
+} else if (provider === 'ECS') {
Review Comment:
Added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2655109984 ## ui/src/views/infra/AddObjectStorage.vue: ## @@ -67,16 +68,82 @@ - + https://s3-hostname or http://s3-hostname"/> - + https://iam-hostname:16443 or http://iam-hostname:16080"/> + + + + Review Comment: Yes it currently it is. https://github.com/mhkadhum/cloudstack/blob/7c86383f111898744245643f63d7831aa728e8af/ui/src/views/infra/AddObjectStorage.vue#L224-L228 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2655109984 ## ui/src/views/infra/AddObjectStorage.vue: ## @@ -67,16 +68,82 @@ - + https://s3-hostname or http://s3-hostname"/> - + https://iam-hostname:16443 or http://iam-hostname:16080"/> + + + + Review Comment: Yes it currently it is. https://github.com/mhkadhum/cloudstack/blob/7c86383f111898744245643f63d7831aa728e8af/ui/src/views/infra/AddObjectStorage.vue#L225 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2655078092
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1560 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.exception.InvalidParameterValueException;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+// Injected dependencies
+@Inject private AccountDao accountDao;
Review Comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
abh1sar commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2650849778
##
ui/src/views/infra/AddObjectStorage.vue:
##
@@ -165,26 +240,66 @@ export default {
const formRaw = toRaw(this.form)
const values = this.handleRemoveFields(formRaw)
-var data = {
- name: values.name,
- size: values.size
+const data = {
+ name: values.name
Review Comment:
size can apply to all providers (it's for framework level tracking)
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1560 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.commons.lang3.StringUtils;
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.cloudstack.storage.object.BucketObject;
+
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.exception.InvalidParameterValueException;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+
+// Injected dependencies
+@Inject private AccountDao accountDao;
Review Comment:
```suggestion
@Inject
private AccountDao accountDao;
```
Please break into two lines
##
ui/src/views/storage/CreateBucket.vue:
##
@@ -110,18 +118,56 @@ export default {
inject: ['parentFetchData'],
data () {
return {
- loading: false
+ loading: false,
+ objectstores: [],
+ policyList: ['Public', 'Private']
+}
+ },
+ computed: {
+isEcsObjectStore () {
+ const selectedId = this.form?.objectstore
+ const stores = this.objectstores || []
+ const selected = stores.find(os => os.id === selectedId)
+ if (!selected) {
+return false
+ }
+
+ const provider = (
+selected.objectstoreprovider ||
+selected.objectStoreProvider ||
+selected.provider ||
+''
+ ).toString().toUpperCase()
+
+ const name = (selected.name || '').toString().toUpperCase()
Review Comment:
We should only rely on the `providername` not `name`
##
ui/src/views/infra/AddObjectStorage.vue:
##
@@ -67,16 +68,82 @@
-
+
https://s3-hostname or http://s3-hostname"/>
-
+
https://iam-hostname:16443 or http://iam-hostname:16080"/>
+
+
+
+
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3696431693 Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 16193 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3696418259 sorry @mhkadhum , still compile errors (not lint anymore): https://github.com/apache/cloudstack/actions/runs/20570652663/job/59082786785?pr=12124#step:7:18860 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3696101494 Hello @blueorangutan Had some stuff going on my build now should work now from my server mvn -pl :cloud-plugin-storage-object-ecs -Dcheckstyle.consoleOutput=true checkstyle:check [INFO] Scanning for projects... [INFO] [INFO] ---< org.apache.cloudstack:cloud-plugin-storage-object-ecs > [INFO] Building Apache CloudStack Plugin - ECS object storage provider 4.23.0.0-SNAPSHOT [INFO] [ jar ]- [INFO] [INFO] --- maven-checkstyle-plugin:3.1.0:check (default-cli) @ cloud-plugin-storage-object-ecs --- [INFO] Starting audit... Audit done. [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 1.275 s [INFO] Finished at: 2025-12-29T13:25:30+03:00 [INFO] -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3695804667 Packaging result [SF]: ✖️ el8 ✖️ el9 ✖️ debian ✖️ suse15. SL-JID 16184 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
blueorangutan commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3695802000 @DaanHoogland a [SL] Jenkins job has been kicked to build packages. It will be bundled with no SystemVM templates. I'll keep you posted as I make progress. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3695799704 @blueorangutan package -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2649626049
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2649625816
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2649625597
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2649625102
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2649624927
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2649624533
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2649624413
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2646865649
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2646865355
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2646863096
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2646862841
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
Review Comment:
Deleted.
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+i
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2646862507
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
Review Comment:
I agree.
Token management and XML parsing have been extracted into separate classes
(EcsMgmtTokenManager and EcsXmlParser) to improve maintainability and reuse.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2646860595
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2646853882
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2646853882
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
+private static final ConcurrentHashMap TOKEN_CACHE =
new ConcurrentHashMap<>();
+private static final ConcurrentHashMap TOKEN_LOCKS = new
ConcurrentHashMap<>();
+
+private static final class TokenKey {
+final String mgmtUrl;
+final String user;
+TokenKey(final String mgmtUrl, final String user) {
+this.mgmtUrl = mgmtUrl;
+this.user = user;
+}
+@Override public boolean equals(final Object o) {
+if (this == o) return true;
+if (!(o instanceof TokenKey)) return false;
+final TokenKey k = (TokenKey) o;
+return Objects.equals(mgmtUrl, k.mgmtUrl) && Objects.equals(user,
k.user);
+}
+@Override public int hashCode() { return Objects.hash(mgmtUrl, user); }
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2639023627
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
@@ -0,0 +1,1556 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.driver;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Base64;
+import java.util.List;
+import java.util.Locale;
+import java.util.Map;
+import java.util.Objects;
+import java.util.concurrent.ConcurrentHashMap;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpDelete;
+import org.apache.http.client.methods.HttpGet;
+import org.apache.http.client.methods.HttpPost;
+import org.apache.http.client.methods.HttpPut;
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.entity.StringEntity;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+import org.apache.http.util.EntityUtils;
+
+import org.apache.cloudstack.context.CallContext;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreDetailsDao;
+import org.apache.cloudstack.storage.object.BaseObjectStoreDriverImpl;
+import org.apache.cloudstack.storage.object.Bucket;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+
+import com.amazonaws.services.s3.model.AccessControlList;
+import com.amazonaws.services.s3.model.BucketPolicy;
+import com.cloud.agent.api.to.BucketTO;
+import com.cloud.agent.api.to.DataStoreTO;
+import com.cloud.storage.BucketVO;
+import com.cloud.storage.dao.BucketDao;
+import com.cloud.user.Account;
+import com.cloud.user.AccountDetailsDao;
+import com.cloud.user.AccountDetailVO;
+import com.cloud.user.dao.AccountDao;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreDriverImpl extends BaseObjectStoreDriverImpl {
+private static final Logger logger =
LogManager.getLogger(EcsObjectStoreDriverImpl.class);
+
+// Object store details keys
+private static final String MGMT_URL = "mgmt_url"; // e.g.
https://ecs-api.example.com
+private static final String SA_USER= "sa_user"; // service
account user
+private static final String SA_PASS= "sa_password"; // service
account password
+private static final String NAMESPACE = "namespace"; // e.g.
cloudstack
+private static final String INSECURE = "insecure"; // "true" to
ignore TLS cert/host
+private static final String S3_HOST= "s3_host"; // S3 endpoint
host (or URL if UI provides it)
+
+// Per-account keys
+private static final String AD_KEY_ACCESS = "ecs.accesskey";
+private static final String AD_KEY_SECRET = "ecs.secretkey";
+
+// ECS token caching
+private static final long DEFAULT_TOKEN_MAX_AGE_SEC = 300; // fallback if
header missing
+private static final long EXPIRY_SKEW_SEC = 30; // refresh early
Review Comment:
ECS returns the token expiry via a response header at creation time. The
default value is only used as a fallback in cases where the header is missing
or unavailable. Since the expiry is determined by ECS and not intended to be
tuned at runtime, adding this to global configuration does not seem necessary
at this point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3671450355 > Reviewed EcsObjectStoreDriverImpl Overall functionality looks good but there are some comments on code maintainability and organisation. Please check. Overall, I think use of reflections can be avoided in most cases. Hello @abh1sar Thank you for ur feedback, am working on addressing all the issues u provided. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3641370080 > Do you have an ECS environment where you plan to conduct the testing? No, but we will regard this as 3rd party extension and trust you with specific feature testing, only doing regression testing. (sponsored donation in the sense that you make an instance available for testing is welcome, but you maintaining it as even more ;) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
abh1sar commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3640863011 Hi @mhkadhum, I am planning to finish my review by early next week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3640838864 > > Hello @abh1sar @DaanHoogland I just wanted to quickly follow up on my last comment. I understand u r probably busy but if u get a chance, could u take a look when possible ? Appreciate the help! > > sorry @mhkadhum , this [#12124 (comment)](https://github.com/apache/cloudstack/pull/12124#issuecomment-3606486217) you mean right? I didn’t read it as a question and though this is ready for testing. What issues are you having right now? No worries @DaanHoogland Yes this one. There are no issues at the moment. I am simply checking the status of the PR. Additionally, I am planning to develop another plugin for a different third-party object storage. Do you have an ECS environment where you plan to conduct the testing? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3640742450 > Hello @abh1sar @DaanHoogland I just wanted to quickly follow up on my last comment. I understand u r probably busy but if u get a chance, could u take a look when possible ? Appreciate the help! sorry @mhkadhum , this https://github.com/apache/cloudstack/pull/12124#issuecomment-3606486217 you mean right? I didn’t read it as a question and though this is ready for testing. What issues are you having right now? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3640279625 Hello @abh1sar @DaanHoogland I just wanted to quickly follow up on my last comment. I understand u r probably busy but if u get a chance, could u take a look when possible ? Appreciate the help! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3606486217 Hi, @DaanHoogland @abh1sar Just made the Update talked about in my previous comment with @DaanHoogland The screen displays the token creation request, which responds with the token and expiration details in the response headers. These headers are then utilized in the code to store the token for the specified duration. X-SDS-AUTH-MAX-AGE https://github.com/user-attachments/assets/ead88adb-0e40-4ea2-aa75-10f0c141d566"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2581093407 ## plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java: ## Review Comment: Hello @DaanHoogland , I've been working the comments you provided in order to optimize the code. While these changes may not reduce the lines in the driver file :) but they will enhance the readability of the methods. Additionally, I am working on implementing token caching for the communication between CloudStack and ECS. Currently, the token is being generated for each request, but due to rate limits on token generation, this approach is inefficient. I will update the code soon. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2571517861
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java:
##
Review Comment:
methods `createBucket`, listBuckets`, `listBucketObjects`, `deleteBucket`,
`setBucketPolicy`, `getBucketPolicy`, `setOrSuspendVersioning`,
`setBucketQuota`, `ensureAccountUserAndSecret` and `ecsBucketExists` are rather
great. Again not my code to maintain, but extracting smaller methods will make
it more readible and may help find some code deduplication possibilities.
##
plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/EcsObjectStoreLifeCycleImpl.java:
##
@@ -0,0 +1,255 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.cloudstack.storage.datastore.lifecycle;
+
+import java.util.HashMap;
+import java.util.Map;
+
+import javax.inject.Inject;
+import javax.net.ssl.SSLContext;
+
+import org.apache.cloudstack.engine.subsystem.api.storage.ClusterScope;
+import org.apache.cloudstack.engine.subsystem.api.storage.DataStore;
+import org.apache.cloudstack.engine.subsystem.api.storage.HostScope;
+import org.apache.cloudstack.engine.subsystem.api.storage.ZoneScope;
+import org.apache.cloudstack.storage.datastore.db.ObjectStoreVO;
+import org.apache.cloudstack.storage.object.datastore.ObjectStoreHelper;
+import
org.apache.cloudstack.storage.object.datastore.ObjectStoreProviderManager;
+import
org.apache.cloudstack.storage.object.store.lifecycle.ObjectStoreLifeCycle;
+import org.apache.logging.log4j.LogManager;
+import org.apache.logging.log4j.Logger;
+import org.apache.http.auth.UsernamePasswordCredentials;
+import org.apache.http.client.methods.CloseableHttpResponse;
+import org.apache.http.client.methods.HttpGet; // change to POST if ECS needs
it
+import org.apache.http.conn.ssl.NoopHostnameVerifier;
+import org.apache.http.impl.auth.BasicScheme;
+import org.apache.http.impl.client.CloseableHttpClient;
+import org.apache.http.impl.client.HttpClients;
+import org.apache.http.ssl.SSLContextBuilder;
+import org.apache.http.ssl.TrustStrategy;
+
+import com.cloud.agent.api.StoragePoolInfo;
+import com.cloud.hypervisor.Hypervisor.HypervisorType;
+import com.cloud.utils.exception.CloudRuntimeException;
+
+public class EcsObjectStoreLifeCycleImpl implements ObjectStoreLifeCycle {
+
+private static final Logger LOG =
LogManager.getLogger(EcsObjectStoreLifeCycleImpl.class);
+
+// detail keys coming from the API
+private static final String MGMT_URL = "mgmt_url";
+private static final String SA_USER= "sa_user";
+private static final String SA_PASS= "sa_password";
+private static final String INSECURE = "insecure";
+
+// optional details (currently not used in persistence logic but accepted)
+private static final String S3_HOST= "s3_host";
+private static final String NAMESPACE = "namespace";
+
+private static final String PROVIDER_NAME = "ECS";
+
+@Inject
+ObjectStoreHelper objectStoreHelper;
+
+@Inject
+ObjectStoreProviderManager objectStoreMgr;
+
+public EcsObjectStoreLifeCycleImpl() {
+}
+
+@SuppressWarnings("unchecked")
+@Override
+public DataStore initialize(Map dsInfos) {
Review Comment:
this method can be split into several verify methods. Your code to maintain
@mhkadhum , so I won’t -1 on this point but for readability it would be good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3588949629 > looks generally good @mhkadhum , thanks for the contribution. Will you be maintaining this? As it is a 3rd party component and it may require hardware or licenses not available to the project. Yes, I will be the maintainer for the plugin. I have made a new commit to fix the previous build. Could you kindly review it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3588806240 looks generally good @mhkadhum , thanks for the contribution. Will you be maintaining this? As it is a 3rd party component and it may require hardware or licenses not available to the project. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2570978130
##
tools/marvin/setup.py:
##
@@ -27,7 +27,7 @@
raise RuntimeError("python setuptools is required to build Marvin")
-VERSION = "4.23.0.0-SNAPSHOT"
+VERSION = "4.23.0.0"
Review Comment:
Ok,
I've just added it back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2570917106
##
tools/marvin/setup.py:
##
@@ -27,7 +27,7 @@
raise RuntimeError("python setuptools is required to build Marvin")
-VERSION = "4.23.0.0-SNAPSHOT"
+VERSION = "4.23.0.0"
Review Comment:
yes, it should only be updated by building (not by pre-commit) somebody
introduced that behaviour for testing I think and it forces us to be more
vigilent at checking in, so I don’t mind. We need to keep snapshot though
untill releasing .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2569681765
##
tools/marvin/setup.py:
##
@@ -27,7 +27,7 @@
raise RuntimeError("python setuptools is required to build Marvin")
-VERSION = "4.23.0.0-SNAPSHOT"
+VERSION = "4.23.0.0"
Review Comment:
Hi @DaanHoogland
This was updated after running the pre-commit, should i keep it as
`VERSION = "4.23.0.0-SNAPSHOT"
`
?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on code in PR #12124:
URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2568713743
##
tools/marvin/setup.py:
##
@@ -27,7 +27,7 @@
raise RuntimeError("python setuptools is required to build Marvin")
-VERSION = "4.23.0.0-SNAPSHOT"
+VERSION = "4.23.0.0"
Review Comment:
this should not be in the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
mhkadhum commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3583083458 Thanks for the feedback! ive added the missing license headers and ran `pre-commit` locally, and pushed an updated commit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on code in PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2560307058 ## plugins/storage/object/ECS/src/main/resources/META-INF/cloudstack/storage-object-ecs/spring-storage-object-ecs-context.xml: ## @@ -0,0 +1,14 @@ +http://www.springframework.org/schema/beans"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xmlns:context="http://www.springframework.org/schema/context"; + xsi:schemaLocation=" + http://www.springframework.org/schema/beans + http://www.springframework.org/schema/beans/spring-beans.xsd + http://www.springframework.org/schema/context + http://www.springframework.org/schema/context/spring-context.xsd";> + + + + Review Comment: i know, lint can be picky at times -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on code in PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2560305737 ## plugins/storage/object/ECS/src/main/resources/META-INF/cloudstack/storage-object-ecs/spring-storage-object-ecs-context.xml: ## @@ -0,0 +1,14 @@ +http://www.springframework.org/schema/beans"; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"; + xmlns:context="http://www.springframework.org/schema/context"; + xsi:schemaLocation=" + http://www.springframework.org/schema/beans + http://www.springframework.org/schema/beans/spring-beans.xsd + http://www.springframework.org/schema/context + http://www.springframework.org/schema/context/spring-context.xsd";> + + + + Review Comment: ```suggestion ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on code in PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2560304885 ## plugins/storage/object/ECS/src/main/resources/META-INF/cloudstack/storage-object-ecs/spring-storage-object-ecs-context.xml: ## @@ -0,0 +1,14 @@ +http://www.springframework.org/schema/beans"; Review Comment: missing license -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
DaanHoogland commented on code in PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#discussion_r2560303357 ## plugins/storage/object/ECS/src/main/java/org/apache/cloudstack/storage/datastore/driver/EcsObjectStoreDriverImpl.java: ## @@ -0,0 +1,1490 @@ +/* ... license header unchanged ... */ Review Comment: this is not a valid apache license -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
sureshanaparti commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3575577974 @mhkadhum can you check the build errors? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
codecov[bot] commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3571077135 ## [Codecov](https://app.codecov.io/gh/apache/cloudstack/pull/12124?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report :white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 3.58%. Comparing base ([`8171d95`](https://app.codecov.io/gh/apache/cloudstack/commit/8171d9568c941877e0290f77051ed45f31a73544?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`625fb4e`](https://app.codecov.io/gh/apache/cloudstack/commit/625fb4ebc4cfff2dbd169ea9683182fb71f7f34b?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)). > :exclamation: There is a different number of reports uploaded between BASE (8171d95) and HEAD (625fb4e). Click for more details. > > HEAD has 1 upload less than BASE > >| Flag | BASE (8171d95) | HEAD (625fb4e) | >|--|--|--| >|unittests|1|0| > Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #12124 +/- ## = - Coverage 17.56%3.58% -13.98% = Files 5912 445 -5467 Lines52938337571 -491812 Branches 64660 6915-57745 = - Hits 92984 1347-91637 + Misses 42594136060 -389881 + Partials 10458 164-10294 ``` | [Flag](https://app.codecov.io/gh/apache/cloudstack/pull/12124/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | | |---|---|---| | [uitests](https://app.codecov.io/gh/apache/cloudstack/pull/12124/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `3.58% <ø> (-0.01%)` | :arrow_down: | | [unittests](https://app.codecov.io/gh/apache/cloudstack/pull/12124/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more. [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/cloudstack/pull/12124?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache). :rocket: New features to boost your workflow: - :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems. - :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] Adding new Dell EMC ECS Object Storage Plugin for CloudStack [cloudstack]
boring-cyborg[bot] commented on PR #12124: URL: https://github.com/apache/cloudstack/pull/12124#issuecomment-3570935844 Congratulations on your first Pull Request and welcome to the Apache CloudStack community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/cloudstack/blob/main/CONTRIBUTING.md) Here are some useful points: - In case of a new feature add useful documentation (raise doc PR at https://github.com/apache/cloudstack-documentation) - Be patient and persistent. It might take some time to get a review or get the final approval from the committers. - Pay attention to the quality of your code, ensure tests are passing and your PR doesn't have conflicts. - Please follow [ASF Code of Conduct](https://github.com/apache/.github/blob/main/.github/CODE_OF_CONDUCT.md) for all communication including (but not limited to) comments on Pull Requests, Issues, Mailing list and Slack. - Be sure to read the [CloudStack Coding Conventions](https://cwiki.apache.org/confluence/display/CLOUDSTACK/Coding+conventions). Apache CloudStack is a community-driven project and together we are making it better 🚀. In case of doubts contact the developers at: Mailing List: [email protected] (https://cloudstack.apache.org/mailing-lists.html) Slack: https://apachecloudstack.slack.com/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
