Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-5596982950 @janhoy, any pointers on who in the Solr community might have access to an Azure environment and could help with the final review/testing of this PR? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
github-actions[bot] commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-5486656531 This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the [email protected] mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4812107094 To make the final review easier: testing doesn't actually require an Azure account — the integration tests run against the Azurite emulator (Docker, free)- https://hub.docker.com/r/microsoft/azure-storage-azurite. I've separately verified backup/restore against a real Azure Storage account, including under the Java Security Manager. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4812106077 Thanks @janhoy. I've addressed all your points (dependency trim, lockfile cleanup, README, solr.xml, and the verified security.policy example). If you are ok with the updates could you please clear the "changes requested" status so it's not blocking the final reviewer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4794359744 Thanks for the latest iteration, to me this starts to look quite mature. But as said earlier I'll defer to someone who have an actual AZ environment at hand to test it and do the last blessing. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4792604405 @janhoy thank you for the review. I have addressed all review feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r3469490063 ## solr/modules/azure-blob-repository/gradle.lockfile: ## @@ -0,0 +1,195 @@ +# This is a Gradle generated file for dependency locking. +# Manual edits can break the build and are not advised. +# This file is expected to be part of source control. +com.azure:azure-core-http-okhttp:1.13.3=jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.azure:azure-core:1.57.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.azure:azure-identity:1.18.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.azure:azure-json:1.5.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.azure:azure-sdk-bom:1.3.6=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.azure:azure-storage-blob-batch:12.29.3=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.azure:azure-storage-blob:12.33.3=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.azure:azure-storage-common:12.32.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.azure:azure-storage-internal-avro:12.18.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.azure:azure-xml:1.2.1=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.4=jarValidation,testCompileClasspath,testRuntimeClasspath +com.carrotsearch:hppc:0.10.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath +com.fasterxml.jackson.core:jackson-annotations:2.21=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath +com.fasterxml.jackson.core:jackson-core:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath +com.fasterxml.jackson.core:jackson-databind:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath +com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.21.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath +com.fasterxml.jackson.dataformat:jackson-dataformat-smile:2.21.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath +com.fasterxml.jackson.datatype:jackson-datatype-jsr310:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath +com.fasterxml.jackson.module:jackson-module-jakarta-xmlbind-annotations:2.21.2=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath +com.fasterxml.jackson:jackson-bom:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath +com.fasterxml.woodstox:woodstox-core:7.0.0=apiHelper,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testRuntimeClasspath +com.github.ben-manes.caffeine:caffeine:3.2.3=annotationProcessor,apiHelper,errorprone,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testAnnotationProcessor,testRuntimeClasspath +com.github.docker-java:docker-java-api:3.7.0=jarValidation,testCompileClasspath,testRuntimeClasspath +com.github.docker-java:docker-java-transport-zerodep:3.7.0=jarValidation,testCompileClasspath,testRuntimeClasspath +com.github.docker-java:docker-java-transport:3.7.0=jarValidation,testCompileClasspath,testRuntimeClasspath +com.github.kevinstern:software-and-algorithms:1.0=annotationProcessor,errorprone,testAnnotationProcessor +com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor +com.google.auto.value:auto-value-annotations:1.11.0=annotationProcessor,errorprone,testAnnotationProcessor +com.google.auto:auto-common:1.2.2=annotationProcessor,errorprone,testAnnotationProcessor +com.google.errorprone:error_prone_annotation:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor +com.google.errorprone:error_prone_annotations:2.41.0=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,solrPlatformLibs,testCompileClasspath,testRuntimeClasspath +com.google.errorprone:error_prone_annotations:2.43.0=annotationProcessor,errorprone,testAnnotationProcessor +com.google.errorprone:error_prone_check_api:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3469426912
##
solr/modules/azure-blob-repository/build.gradle:
##
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+apply plugin: 'java-library'
+
+description = 'Azure Blob Storage Repository'
+
+ext {
+ // Disable security manager for azure-blob-repository module tests
+ // Required because Testcontainers needs access to Docker socket and system
properties
+ useSecurityManager = false
+}
+
+dependencies {
+ implementation platform(project(':platform'))
+ api(project(':solr:core'))
+ implementation project(':solr:solrj')
+
+ implementation libs.apache.lucene.core
+
+ implementation platform(libs.azure.sdk.bom)
+
+ // Azure Storage SDK dependencies
+ implementation(libs.azure.storage.blob) {
+exclude group: 'com.azure', module: 'azure-core-http-netty'
+ }
+ implementation(libs.azure.storage.blob.batch) {
+exclude group: 'com.azure', module: 'azure-core-http-netty'
+ }
+ implementation(libs.azure.identity) {
+exclude group: 'com.azure', module: 'azure-core-http-netty'
+ }
Review Comment:
agreed. I've excluded msal4j-persistence-extension. Also excluded
jna-platform from libs.azure.identity. Dropping them removes three jars now. I
have deleted their license files as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3469346644
##
solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc:
##
@@ -826,3 +826,124 @@
https://docs.aws.amazon.com/sdkref/latest/guide/settings-global.html[These optio
* Retries
** RetryMode (`LEGACY`, `STANDARD`, `ADAPTIVE`)
** Max Attempts
+
+=== AzureBlobBackupRepository
+
+Stores and retrieves backup files in a Microsoft Azure Blob Storage container.
+
+This is provided via the `azure-blob-repository`
xref:configuration-guide:solr-modules.adoc[Solr Module] that needs to be
enabled before use.
+
+This plugin supports multiple authentication methods: connection strings,
account keys, SAS tokens, and Azure Identity (Managed Identity, Service
Principal, Azure CLI).
+For Azure Identity, ensure the identity has the "Storage Blob Data
Contributor" role on the storage account.
+
+[source,xml]
+
+
+
+solr-backup
+DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net
+
+
+
+
+AzureBlobBackupRepository accepts the following options for configuration:
+
+`azure.blob.container.name`::
++
+[%autowidth,frame=none]
+|===
+|Required |Default: none
+|===
++
+The name of the Azure Blob Storage container. The container must exist before
performing backup operations.
+
+`azure.blob.connection.string`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Complete Azure Storage connection string. Mutually exclusive with other
authentication methods.
+
+`azure.blob.account.name`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure Storage account name. Used with account key or SAS token authentication.
+
+`azure.blob.account.key`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure Storage account access key. Mutually exclusive with SAS token and Azure
Identity.
+
+`azure.blob.sas.token`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+SAS token for time-limited access. Must include `srt=sco` and `sp=rwdlac`
permissions.
+The `&` characters must be XML-escaped as `&`.
+
+`azure.blob.endpoint`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure Blob Storage endpoint URL (e.g.,
`https://myaccount.blob.core.windows.net`).
+Required for Azure Identity authentication.
+
+`azure.blob.tenant.id`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure AD tenant ID for Service Principal authentication.
+
+`azure.blob.client.id`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure AD application (client) ID for Service Principal authentication.
+
+`azure.blob.client.secret`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Azure AD application secret for Service Principal authentication.
+
+`location`::
++
+[%autowidth,frame=none]
+|===
+|Optional |Default: none
+|===
++
+Default path prefix within the container for backup storage.
+
+The target container must already exist; it is not created automatically.
+
+ Known Limitation: Azure Identity and the Security Manager
+
+Azure Identity authentication (Managed Identity, Service Principal, and
`DefaultAzureCredential`) does not work when Solr is started with the Java
Security Manager enabled, which is the default
(`SOLR_SECURITY_MANAGER_ENABLED=true`).
+To use Azure Identity, set `SOLR_SECURITY_MANAGER_ENABLED=false` before
starting Solr.
+The Connection String, Account Key, and SAS Token authentication methods are
unaffected and work with the Security Manager enabled.
Review Comment:
Looked into it, there is.
The only SM denials come from the developer credentials that spawn a
subprocess — Azure CLI and Azure PowerShell — e.g. access denied
("java.io.FilePermission" "/bin/sh" "execute").
For the Azure CLI credential specifically, I iterated to the minimal working
grant and confirmed the backup then completed and the files landed in the
container:
permission java.io.FilePermission "/bin/sh", "execute";
permission java.io.FilePermission "/dev/null", "read,write";
I've rewritten the ref-guide section accordingly: Managed Identity / Service
Principal work with the default policy (no change needed); the CLI/PowerShell
credentials need the grant above (paths are platform-specific) and are really
only for local dev; otherwise prefer Managed Identity / Service Principal or
set SOLR_SECURITY_MANAGER_ENABLED=false.
I deliberately did not add this grant to the shipped security.policy, to
keep the default minimal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r3469305946 ## solr/modules/azure-blob-repository/README.md: ## @@ -0,0 +1,118 @@ + + +# Apache Solr Azure Blob Storage Backup Repository + +A backup repository implementation for storing Solr backups in Azure Blob Storage. + +## Prerequisites + +- Azure Storage Account with a blob container (must already exist) +- Network access to Azure Blob Storage (HTTPS port 443) + +Enable the module: +```bash +export SOLR_MODULES=azure-blob-repository +``` + +## Configuration + +Add to `solr.xml`: + +```xml + + +YOUR_CONTAINER_NAME + + + +``` + +## Authentication Methods + +### Connection String (Development) + +```xml +DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net +``` + +### Account Name + Account Key + +```xml +https://YOUR_ACCOUNT.blob.core.windows.net +YOUR_ACCOUNT +YOUR_ACCOUNT_KEY +``` + +### SAS Token (Production) + +Generate a SAS token with permissions: Read, Write, Delete, List, Add, Create (`sp=rwdlac`) and resource types: Service, Container, Object (`srt=sco`). + +```xml +https://YOUR_ACCOUNT.blob.core.windows.net +sv=2024-11-04&ss=b&srt=sco&sp=rwdlac&... +``` + +Note: Escape `&` as `&` in XML. + +### Azure Identity (Production - Recommended) + +Uses Azure AD authentication. Requires "Storage Blob Data Contributor" role on the storage account. + +```xml +https://YOUR_ACCOUNT.blob.core.windows.net + +``` + +For Service Principal, add: +```xml +YOUR_TENANT_ID +YOUR_CLIENT_ID +YOUR_CLIENT_SECRET +``` + +Or set environment variables: `AZURE_TENANT_ID`, `AZURE_CLIENT_ID`, `AZURE_CLIENT_SECRET`. + +## Known Limitations + +Azure Identity authentication (Service Principal, Managed Identity, `DefaultAzureCredential`) does not work when Solr is started with the Java `SecurityManager` enabled. The Azure Identity SDK relies on `doPrivileged` patterns that fail under Solr's default security policy; see the [upstream issue](https://github.com/Azure/azure-sdk-for-java/issues/37464) for details. Note that the `SecurityManager` is deprecated for removal in modern JDKs, but Solr still enables it by default via `SOLR_SECURITY_MANAGER_ENABLED=true`. + +Workaround: set `SOLR_SECURITY_MANAGER_ENABLED=false` (in `solr.in.sh` / `solr.in.cmd`, or as an environment variable) before starting Solr. The Connection String, Account Key, and SAS Token authentication methods are unaffected and work with the `SecurityManager` enabled. + +## Usage + +```bash +# Backup +curl "http://localhost:8983/solr/admin/collections?action=BACKUP&name=my-backup&collection=my-collection&repository=azure_blob&location=/"; Review Comment: Agreed, I have trimmed README to a short developer note (how to enable the module). The curl examples were originally there to help a future developer debug the module locally; I've dropped them from the README. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r3469267027 ## solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc: ## @@ -826,3 +826,124 @@ https://docs.aws.amazon.com/sdkref/latest/guide/settings-global.html[These optio * Retries ** RetryMode (`LEGACY`, `STANDARD`, `ADAPTIVE`) ** Max Attempts + +=== AzureBlobBackupRepository + +Stores and retrieves backup files in a Microsoft Azure Blob Storage container. + +This is provided via the `azure-blob-repository` xref:configuration-guide:solr-modules.adoc[Solr Module] that needs to be enabled before use. + +This plugin supports multiple authentication methods: connection strings, account keys, SAS tokens, and Azure Identity (Managed Identity, Service Principal, Azure CLI). +For Azure Identity, ensure the identity has the "Storage Blob Data Contributor" role on the storage account. + +[source,xml] + + Review Comment: Updated it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r3469250665 ## solr/solrj/gradle.lockfile: ## @@ -17,13 +17,19 @@ com.github.stephenc.jcip:jcip-annotations:1.0-1=compileClasspath,compileOnlyHelp com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor com.google.auto.value:auto-value-annotations:1.11.0=annotationProcessor,errorprone,testAnnotationProcessor com.google.auto:auto-common:1.2.2=annotationProcessor,errorprone,testAnnotationProcessor +com.google.code.findbugs:jsr305:3.0.2=spotless865458226 com.google.errorprone:error_prone_annotation:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor +com.google.errorprone:error_prone_annotations:2.18.0=spotless865458226 Review Comment: Same as above, they came from running resolveAndLockAll --write-locks repo-wide while validating the CI build locally, it added *Copy/spotless config tags to existing modules. I've reverted those to match main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r3469248764 ## solr/core/gradle.lockfile: ## @@ -1,191 +1,191 @@ # This is a Gradle generated file for dependency locking. # Manual edits can break the build and are not advised. # This file is expected to be part of source control. -biz.aQute.bnd:biz.aQute.bnd.annotation:7.1.0=compileClasspath,testCompileClasspath -com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.4=jarValidation,testCompileClasspath,testRuntimeClasspath -com.carrotsearch:hppc:0.10.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath -com.fasterxml.jackson.core:jackson-annotations:2.21=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath -com.fasterxml.jackson.core:jackson-core:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath -com.fasterxml.jackson.core:jackson-databind:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath -com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath -com.fasterxml.jackson.dataformat:jackson-dataformat-smile:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath -com.fasterxml.jackson.module:jackson-module-jakarta-xmlbind-annotations:2.21.2=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath -com.fasterxml.jackson:jackson-bom:2.21.2=apiHelper,compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath -com.fasterxml.woodstox:woodstox-core:7.0.0=compileClasspath,jarValidation,runtimeClasspath,runtimeLibs,testCompileClasspath,testRuntimeClasspath -com.github.ben-manes.caffeine:caffeine:3.2.3=annotationProcessor,compileClasspath,errorprone,jarValidation,runtimeClasspath,runtimeLibs,testAnnotationProcessor,testCompileClasspath,testRuntimeClasspath +biz.aQute.bnd:biz.aQute.bnd.annotation:7.1.0=compileClasspath,compileClasspathCopy,testCompileClasspath,testCompileClasspathCopy +com.carrotsearch.randomizedtesting:randomizedtesting-runner:2.8.4=jarValidation,testCompileClasspath,testCompileClasspathCopy,testRuntimeClasspath,testRuntimeClasspathCopy +com.carrotsearch:hppc:0.10.0=compileClasspath,compileClasspathCopy,jarValidation,runtimeClasspath,runtimeClasspathCopy,runtimeLibs,testCompileClasspath,testCompileClasspathCopy,testRuntimeClasspath,testRuntimeClasspathCopy Review Comment: They came from running resolveAndLockAll --write-locks repo-wide while validating the CI build locally, it added *Copy/spotless config tags to existing modules. I've reverted those to match main. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r3458518356 ## solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc: ## @@ -826,3 +826,124 @@ https://docs.aws.amazon.com/sdkref/latest/guide/settings-global.html[These optio * Retries ** RetryMode (`LEGACY`, `STANDARD`, `ADAPTIVE`) ** Max Attempts + +=== AzureBlobBackupRepository + +Stores and retrieves backup files in a Microsoft Azure Blob Storage container. + +This is provided via the `azure-blob-repository` xref:configuration-guide:solr-modules.adoc[Solr Module] that needs to be enabled before use. + +This plugin supports multiple authentication methods: connection strings, account keys, SAS tokens, and Azure Identity (Managed Identity, Service Principal, Azure CLI). +For Azure Identity, ensure the identity has the "Storage Blob Data Contributor" role on the storage account. + +[source,xml] + + + +solr-backup +DefaultEndpointsProtocol=https;AccountName=myaccount;AccountKey=mykey;EndpointSuffix=core.windows.net + + + + +AzureBlobBackupRepository accepts the following options for configuration: + +`azure.blob.container.name`:: ++ +[%autowidth,frame=none] +|=== +|Required |Default: none +|=== ++ +The name of the Azure Blob Storage container. The container must exist before performing backup operations. + +`azure.blob.connection.string`:: ++ +[%autowidth,frame=none] +|=== +|Optional |Default: none +|=== ++ +Complete Azure Storage connection string. Mutually exclusive with other authentication methods. + +`azure.blob.account.name`:: ++ +[%autowidth,frame=none] +|=== +|Optional |Default: none +|=== ++ +Azure Storage account name. Used with account key or SAS token authentication. + +`azure.blob.account.key`:: ++ +[%autowidth,frame=none] +|=== +|Optional |Default: none +|=== ++ +Azure Storage account access key. Mutually exclusive with SAS token and Azure Identity. + +`azure.blob.sas.token`:: ++ +[%autowidth,frame=none] +|=== +|Optional |Default: none +|=== ++ +SAS token for time-limited access. Must include `srt=sco` and `sp=rwdlac` permissions. +The `&` characters must be XML-escaped as `&`. + +`azure.blob.endpoint`:: ++ +[%autowidth,frame=none] +|=== +|Optional |Default: none +|=== ++ +Azure Blob Storage endpoint URL (e.g., `https://myaccount.blob.core.windows.net`). +Required for Azure Identity authentication. + +`azure.blob.tenant.id`:: ++ +[%autowidth,frame=none] +|=== +|Optional |Default: none +|=== ++ +Azure AD tenant ID for Service Principal authentication. + +`azure.blob.client.id`:: ++ +[%autowidth,frame=none] +|=== +|Optional |Default: none +|=== ++ +Azure AD application (client) ID for Service Principal authentication. + +`azure.blob.client.secret`:: ++ +[%autowidth,frame=none] +|=== +|Optional |Default: none +|=== ++ +Azure AD application secret for Service Principal authentication. + +`location`:: ++ +[%autowidth,frame=none] +|=== +|Optional |Default: none +|=== ++ +Default path prefix within the container for backup storage. + +The target container must already exist; it is not created automatically. + + Known Limitation: Azure Identity and the Security Manager + +Azure Identity authentication (Managed Identity, Service Principal, and `DefaultAzureCredential`) does not work when Solr is started with the Java Security Manager enabled, which is the default (`SOLR_SECURITY_MANAGER_ENABLED=true`). +To use Azure Identity, set `SOLR_SECURITY_MANAGER_ENABLED=false` before starting Solr. +The Connection String, Account Key, and SAS Token authentication methods are unaffected and work with the Security Manager enabled. Review Comment: Is it an option here to give an example of what to add to `security.policy` as an alternative to disabling? ## solr/solrj/gradle.lockfile: ## @@ -17,13 +17,19 @@ com.github.stephenc.jcip:jcip-annotations:1.0-1=compileClasspath,compileOnlyHelp com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor com.google.auto.value:auto-value-annotations:1.11.0=annotationProcessor,errorprone,testAnnotationProcessor com.google.auto:auto-common:1.2.2=annotationProcessor,errorprone,testAnnotationProcessor +com.google.code.findbugs:jsr305:3.0.2=spotless865458226 com.google.errorprone:error_prone_annotation:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor +com.google.errorprone:error_prone_annotations:2.18.0=spotless865458226 Review Comment: Why do we suddenly have all these spotless dependencies added to lockfiles? I would not expect any chagnes to lockfiles outside of the new module itself? ## solr/core/gradle.lockfile: ## @@ -1,191 +1,191 @@ # This is a Gradle generated file for dependency locking. # Manual edits can break the build and are not advised. # This file is expected to be part of source control.
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r3458591665 ## solr/solrj/gradle.lockfile: ## @@ -17,13 +17,19 @@ com.github.stephenc.jcip:jcip-annotations:1.0-1=compileClasspath,compileOnlyHelp com.google.auto.service:auto-service-annotations:1.0.1=annotationProcessor,errorprone,testAnnotationProcessor com.google.auto.value:auto-value-annotations:1.11.0=annotationProcessor,errorprone,testAnnotationProcessor com.google.auto:auto-common:1.2.2=annotationProcessor,errorprone,testAnnotationProcessor +com.google.code.findbugs:jsr305:3.0.2=spotless865458226 com.google.errorprone:error_prone_annotation:2.41.0=annotationProcessor,errorprone,testAnnotationProcessor +com.google.errorprone:error_prone_annotations:2.18.0=spotless865458226 Review Comment: Why do we suddenly have all these spotless dependencies added to lockfiles? I would not expect any changes to lockfiles outside of the new module itself? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4775666316 @janhoy could you please trigger a CI re-run, seems like a unrelated flaky test in the CI run. I ran CI locally, it did run successfully. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4773590864 @janhoy could you please approve the workflow runs on this PR again? The previous CI failure was because my branch hadn't merged the latest main -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4772835763 @janhoy could you please approve workflow runs on this fork PR again. I had to make updates to the pr. Also I did run the ci jobs locally with all the flags to make sure . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4627101726 Status update: CI is now fully green (Gradle Precommit, Crave tests, and changelog validation all passing on the latest commit). All of @psalagnac's feedback is addressed. @psalagnac, when you have a chance, could you please review the pr . @HoustonPutman, your feedback is welcome too. Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359089044
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+this(
+createInternalClient(
+connectionString,
+endpoint,
+accountName,
+accountKey,
+sasToken,
+tenantId,
+clientId,
+clientSecret),
+containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String
containerName) {
+this.containerClient =
blobServiceClient.getBlobContainerClient(containerName);
+try {
+ containerClient.create();
+} catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+throw e;
+ }
+}
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+builder.httpClient(SHARED_HTTP_CLIENT);
+
+if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+} else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) &&
StrUtils.isNotNullOrEmpty(accountKey)) {
+builder.credential(
+new
com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+builder.sasToken(sasToken);
+ } else {
+TokenCredential credential = new
DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+builder.credential(credential);
+ }
+} else {
+ throw new IllegalArgumentException("Either connectionString or endpoint
must be provided");
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359089044
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+this(
+createInternalClient(
+connectionString,
+endpoint,
+accountName,
+accountKey,
+sasToken,
+tenantId,
+clientId,
+clientSecret),
+containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String
containerName) {
+this.containerClient =
blobServiceClient.getBlobContainerClient(containerName);
+try {
+ containerClient.create();
+} catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+throw e;
+ }
+}
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+builder.httpClient(SHARED_HTTP_CLIENT);
+
+if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+} else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) &&
StrUtils.isNotNullOrEmpty(accountKey)) {
+builder.credential(
+new
com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+builder.sasToken(sasToken);
+ } else {
+TokenCredential credential = new
DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+builder.credential(credential);
+ }
+} else {
+ throw new IllegalArgumentException("Either connectionString or endpoint
must be provided");
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359089044
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+this(
+createInternalClient(
+connectionString,
+endpoint,
+accountName,
+accountKey,
+sasToken,
+tenantId,
+clientId,
+clientSecret),
+containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String
containerName) {
+this.containerClient =
blobServiceClient.getBlobContainerClient(containerName);
+try {
+ containerClient.create();
+} catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+throw e;
+ }
+}
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+builder.httpClient(SHARED_HTTP_CLIENT);
+
+if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+} else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) &&
StrUtils.isNotNullOrEmpty(accountKey)) {
+builder.credential(
+new
com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+builder.sasToken(sasToken);
+ } else {
+TokenCredential credential = new
DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+builder.credential(credential);
+ }
+} else {
+ throw new IllegalArgumentException("Either connectionString or endpoint
must be provided");
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359080483
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs.
Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+this.blobClient = blobClient;
+this.blobPath = blobPath;
+this.closed = false;
+this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+this.blockUpload = null;
+this.committed = false;
+
+if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+}
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+buffer.put((byte) b);
+
+if (!buffer.hasRemaining()) {
+ uploadBlock();
+}
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len,
b.length)) {
+ throw new IndexOutOfBoundsException();
+} else if (len == 0) {
+ return;
+}
+
+int currentOffset = off;
+int lenRemaining = len;
+while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+}
+if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+}
+ }
+
+ private static boolean outOfRange(int off, int len) {
+return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+int size = buffer.position() - buffer.arrayOffset();
+
+if (size == 0) {
+ return;
+}
+
+if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+}
+
+try (ByteArrayInputStream inputStream =
+new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+} catch (BlobStorageException e) {
+ if (blockUpload != null) {
+blockUpload.abort();
+if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+}
+ }
+
+ throw new IOException(
+ "Failed to upload block",
AzureBlobStorageClient.handleBlobException(e));
+}
+
+buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (buffer.position() - buffer.arrayOffset() > 0) {
Review Comment:
made the change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3359082665
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs.
Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+this.blobClient = blobClient;
+this.blobPath = blobPath;
+this.closed = false;
+this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+this.blockUpload = null;
+this.committed = false;
+
+if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+}
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+buffer.put((byte) b);
+
+if (!buffer.hasRemaining()) {
+ uploadBlock();
+}
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len,
b.length)) {
+ throw new IndexOutOfBoundsException();
+} else if (len == 0) {
+ return;
+}
+
+int currentOffset = off;
+int lenRemaining = len;
+while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+}
+if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+}
+ }
+
+ private static boolean outOfRange(int off, int len) {
+return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+int size = buffer.position() - buffer.arrayOffset();
+
+if (size == 0) {
+ return;
+}
+
+if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+}
+
+try (ByteArrayInputStream inputStream =
+new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+} catch (BlobStorageException e) {
+ if (blockUpload != null) {
+blockUpload.abort();
+if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+}
+ }
+
+ throw new IOException(
+ "Failed to upload block",
AzureBlobStorageClient.handleBlobException(e));
+}
+
+buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+}
+
+if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+}
+ }
+
+ @Override
+ public void close() throws
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4580360021 @janhoy could you please approve workflow runs on this fork PR again. I did run the ci jobs locally with all the flags used in CI to make sure . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4567142525 Could a committer please approve workflow runs on this fork PR? The checks are currently waiting for maintainer approval. cc @janhoy -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232299832
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs.
Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+this.blobClient = blobClient;
+this.blobPath = blobPath;
+this.closed = false;
+this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+this.blockUpload = null;
+this.committed = false;
+
+if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+}
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+buffer.put((byte) b);
+
+if (!buffer.hasRemaining()) {
+ uploadBlock();
+}
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len,
b.length)) {
+ throw new IndexOutOfBoundsException();
+} else if (len == 0) {
+ return;
+}
+
+int currentOffset = off;
+int lenRemaining = len;
+while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+}
+if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+}
+ }
+
+ private static boolean outOfRange(int off, int len) {
+return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+int size = buffer.position() - buffer.arrayOffset();
+
+if (size == 0) {
+ return;
+}
+
+if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+}
+
+try (ByteArrayInputStream inputStream =
+new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+} catch (BlobStorageException e) {
+ if (blockUpload != null) {
+blockUpload.abort();
+if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+}
+ }
+
+ throw new IOException(
+ "Failed to upload block",
AzureBlobStorageClient.handleBlobException(e));
+}
+
+buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+}
+
+if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+}
+ }
+
+ @Override
+ public void close() throws
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232147739
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+this(
+createInternalClient(
+connectionString,
+endpoint,
+accountName,
+accountKey,
+sasToken,
+tenantId,
+clientId,
+clientSecret),
+containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String
containerName) {
+this.containerClient =
blobServiceClient.getBlobContainerClient(containerName);
+try {
+ containerClient.create();
+} catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+throw e;
+ }
+}
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+builder.httpClient(SHARED_HTTP_CLIENT);
+
+if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+} else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) &&
StrUtils.isNotNullOrEmpty(accountKey)) {
+builder.credential(
+new
com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+builder.sasToken(sasToken);
+ } else {
+TokenCredential credential = new
DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+builder.credential(credential);
+ }
+} else {
+ throw new IllegalArgumentException("Either connectionString or endpoint
must be provided");
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232296536
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs.
Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+this.blobClient = blobClient;
+this.blobPath = blobPath;
+this.closed = false;
+this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+this.blockUpload = null;
+this.committed = false;
+
+if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+}
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+buffer.put((byte) b);
+
+if (!buffer.hasRemaining()) {
+ uploadBlock();
+}
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len,
b.length)) {
+ throw new IndexOutOfBoundsException();
+} else if (len == 0) {
+ return;
+}
+
+int currentOffset = off;
+int lenRemaining = len;
+while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+}
+if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+}
+ }
+
+ private static boolean outOfRange(int off, int len) {
+return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+int size = buffer.position() - buffer.arrayOffset();
+
+if (size == 0) {
+ return;
+}
+
+if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+}
+
+try (ByteArrayInputStream inputStream =
+new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+} catch (BlobStorageException e) {
+ if (blockUpload != null) {
+blockUpload.abort();
+if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+}
+ }
+
+ throw new IOException(
+ "Failed to upload block",
AzureBlobStorageClient.handleBlobException(e));
+}
+
+buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+}
+
+if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+}
+ }
+
+ @Override
+ public void close() throws
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232277896
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs.
Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+this.blobClient = blobClient;
+this.blobPath = blobPath;
+this.closed = false;
+this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+this.blockUpload = null;
+this.committed = false;
+
+if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+}
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+buffer.put((byte) b);
+
+if (!buffer.hasRemaining()) {
+ uploadBlock();
+}
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len,
b.length)) {
+ throw new IndexOutOfBoundsException();
+} else if (len == 0) {
+ return;
+}
+
+int currentOffset = off;
+int lenRemaining = len;
+while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+}
+if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+}
+ }
+
+ private static boolean outOfRange(int off, int len) {
+return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+int size = buffer.position() - buffer.arrayOffset();
+
+if (size == 0) {
+ return;
+}
+
+if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+}
+
+try (ByteArrayInputStream inputStream =
Review Comment:
fixed. Now passing BinaryData.fromByteBuffer(ByteBuffer.wrap(buffer.array(),
0, size)) to the new stageBlock(String, BinaryData) overload, so no
ByteArrayInputStream is involved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232260800
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs.
Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+this.blobClient = blobClient;
+this.blobPath = blobPath;
+this.closed = false;
+this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+this.blockUpload = null;
+this.committed = false;
+
+if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+}
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+buffer.put((byte) b);
+
+if (!buffer.hasRemaining()) {
+ uploadBlock();
+}
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len,
b.length)) {
+ throw new IndexOutOfBoundsException();
+} else if (len == 0) {
+ return;
+}
+
+int currentOffset = off;
+int lenRemaining = len;
+while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+}
+if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+}
+ }
+
+ private static boolean outOfRange(int off, int len) {
+return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+int size = buffer.position() - buffer.arrayOffset();
+
+if (size == 0) {
+ return;
+}
+
+if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+}
+
+try (ByteArrayInputStream inputStream =
+new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+} catch (BlobStorageException e) {
+ if (blockUpload != null) {
+blockUpload.abort();
+if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+}
+ }
+
+ throw new IOException(
+ "Failed to upload block",
AzureBlobStorageClient.handleBlobException(e));
+}
+
+buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+}
+
+if (blockUpload != null) {
+ blockUpload.complete();
Review Comment:
fixed. flush() now only calls uploadBlock()
--
This is an automated message from the Apache
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232226265
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+this(
+createInternalClient(
+connectionString,
+endpoint,
+accountName,
+accountKey,
+sasToken,
+tenantId,
+clientId,
+clientSecret),
+containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String
containerName) {
+this.containerClient =
blobServiceClient.getBlobContainerClient(containerName);
+try {
+ containerClient.create();
+} catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+throw e;
+ }
+}
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+builder.httpClient(SHARED_HTTP_CLIENT);
+
+if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+} else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) &&
StrUtils.isNotNullOrEmpty(accountKey)) {
+builder.credential(
+new
com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+builder.sasToken(sasToken);
+ } else {
+TokenCredential credential = new
DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+builder.credential(credential);
+ }
+} else {
+ throw new IllegalArgumentException("Either connectionString or endpoint
must be provided");
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232220445
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length)
{
+this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize,
int maxCachedPages) {
+super(path);
+this.path = path;
+this.client = client;
+this.length = length;
+this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+closed = true;
+cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ensureOpen();
+if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
Review Comment:
made the change to seekInternal function as extending BufferedIndexInput
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232194020
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length)
{
+this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize,
int maxCachedPages) {
+super(path);
+this.path = path;
+this.client = client;
+this.length = length;
+this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+closed = true;
+cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ensureOpen();
+if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+}
+
+position = pos;
+ }
+
+ @Override
+ public long length() {
+return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length)
throws IOException {
+ensureOpen();
+if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ",
length=" + length);
+}
+
+AzureBlobIndexInput slice =
+new AzureBlobIndexInput(
+getFullSliceDescription(sliceDescription), client, length,
pageSize, MAX_CACHED_PAGES);
+
+slice.position = 0L;
+
+// Wrap client in a view that remaps range requests by adding base offset
+slice.clientViewBaseOffset = this.clientViewBaseOffset + offset;
+return slice;
+ }
+
+ @Override
+ public byte readByte() throws IOException {
+ensureOpen();
+if (position >= length) {
+ throw new EOFException("End of stream reached");
+}
+
+byte[] page = getPage(pageIndex(position));
+int inPageOffset = (int) (position % pageSize);
+byte value = page[inPageOffset];
+position += 1L;
+return value;
+ }
+
+ @Override
+ public void readBytes(byte[] b, int offset, int len) throws IOException {
+ensureOpen();
+if (len < 0) {
+ throw new IOException("Length must be non-negative");
+}
+
+if (position + len > length) {
+ throw new EOFException("End of stream reached");
+}
+
+int remaining = len;
+while (remaining > 0) {
+ long pageIdx = pageIndex(position);
+ byte[] page = getPage(pageIdx);
+ int inPageOffset = (int) (position % pageSize);
+ int toCopy = Math.min(remaining, pageSize - inPageOffset);
+ System.arraycopy(page, inPageOffset, b, offset + (len - remaining),
toCopy);
+ position += toCopy;
+ remaining -= toCopy;
+}
+ }
+
+ // Internal state for slices: base offset to add to all range requests
+ private long clientViewBaseOffset = 0L;
Review Comment:
no longer being used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232186613
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length)
{
+this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize,
int maxCachedPages) {
+super(path);
+this.path = path;
+this.client = client;
+this.length = length;
+this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+closed = true;
+cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ensureOpen();
+if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+}
+
+position = pos;
+ }
+
+ @Override
+ public long length() {
+return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length)
throws IOException {
+ensureOpen();
+if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ",
length=" + length);
+}
+
+AzureBlobIndexInput slice =
+new AzureBlobIndexInput(
+getFullSliceDescription(sliceDescription), client, length,
pageSize, MAX_CACHED_PAGES);
+
+slice.position = 0L;
Review Comment:
updated the logic for this as AzureBlobIndexInput uses BufferedIndexInput
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232166756
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs.
Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+this.blobClient = blobClient;
+this.blobPath = blobPath;
+this.closed = false;
+this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+this.blockUpload = null;
+this.committed = false;
+
+if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+}
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+buffer.put((byte) b);
+
+if (!buffer.hasRemaining()) {
+ uploadBlock();
+}
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len,
b.length)) {
+ throw new IndexOutOfBoundsException();
+} else if (len == 0) {
+ return;
+}
+
+int currentOffset = off;
+int lenRemaining = len;
+while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+}
+if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+}
+ }
+
+ private static boolean outOfRange(int off, int len) {
+return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+int size = buffer.position() - buffer.arrayOffset();
+
+if (size == 0) {
+ return;
+}
+
+if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+}
+
+try (ByteArrayInputStream inputStream =
+new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+} catch (BlobStorageException e) {
+ if (blockUpload != null) {
+blockUpload.abort();
+if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+}
+ }
+
+ throw new IOException(
+ "Failed to upload block",
AzureBlobStorageClient.handleBlobException(e));
+}
+
+buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+}
+
+if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+}
+ }
+
+ @Override
+ public void close() throws
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232173682
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs.
Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+this.blobClient = blobClient;
+this.blobPath = blobPath;
+this.closed = false;
+this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+this.blockUpload = null;
+this.committed = false;
+
+if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+}
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+buffer.put((byte) b);
+
+if (!buffer.hasRemaining()) {
+ uploadBlock();
+}
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len,
b.length)) {
+ throw new IndexOutOfBoundsException();
+} else if (len == 0) {
+ return;
+}
+
+int currentOffset = off;
+int lenRemaining = len;
+while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+}
+if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+}
+ }
+
+ private static boolean outOfRange(int off, int len) {
+return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+int size = buffer.position() - buffer.arrayOffset();
Review Comment:
updated it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232168396
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobOutputStream.java:
##
@@ -0,0 +1,269 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.specialized.BlockBlobClient;
+import java.io.ByteArrayInputStream;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.Base64;
+import java.util.List;
+import java.util.UUID;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * OutputStream implementation for Azure Blob Storage using block blobs.
Supports chunked uploads
+ * for large files.
+ */
+public class AzureBlobOutputStream extends OutputStream {
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final int BLOCK_SIZE = 4 * 1024 * 1024;
+
+ private final BlobClient blobClient;
+ private final String blobPath;
+ private volatile boolean closed;
+ private final ByteBuffer buffer;
+ private BlockUpload blockUpload;
+ private boolean committed;
+
+ public AzureBlobOutputStream(BlobClient blobClient, String blobPath) {
+this.blobClient = blobClient;
+this.blobPath = blobPath;
+this.closed = false;
+this.buffer = ByteBuffer.allocate(BLOCK_SIZE);
+this.blockUpload = null;
+this.committed = false;
+
+if (log.isDebugEnabled()) {
+ log.debug("Created BlobOutputStream for blobPath '{}'", blobPath);
+}
+ }
+
+ @Override
+ public void write(int b) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+buffer.put((byte) b);
+
+if (!buffer.hasRemaining()) {
+ uploadBlock();
+}
+ }
+
+ @Override
+ public void write(byte[] b, int off, int len) throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (outOfRange(off, b.length) || len < 0 || outOfRange(off + len,
b.length)) {
+ throw new IndexOutOfBoundsException();
+} else if (len == 0) {
+ return;
+}
+
+int currentOffset = off;
+int lenRemaining = len;
+while (buffer.remaining() < lenRemaining) {
+ int firstPart = buffer.remaining();
+ buffer.put(b, currentOffset, firstPart);
+ uploadBlock();
+
+ currentOffset += firstPart;
+ lenRemaining -= firstPart;
+}
+if (lenRemaining > 0) {
+ buffer.put(b, currentOffset, lenRemaining);
+}
+ }
+
+ private static boolean outOfRange(int off, int len) {
+return off < 0 || off > len;
+ }
+
+ private void uploadBlock() throws IOException {
+int size = buffer.position() - buffer.arrayOffset();
+
+if (size == 0) {
+ return;
+}
+
+if (blockUpload == null) {
+ if (log.isDebugEnabled()) {
+log.debug("New block upload for blobPath '{}'", blobPath);
+ }
+
+ blockUpload = newBlockUpload();
+}
+
+try (ByteArrayInputStream inputStream =
+new ByteArrayInputStream(buffer.array(), buffer.arrayOffset(), size)) {
+ blockUpload.uploadBlock(inputStream, size);
+} catch (BlobStorageException e) {
+ if (blockUpload != null) {
+blockUpload.abort();
+if (log.isDebugEnabled()) {
+ log.debug("Block upload aborted for blobPath '{}'.", blobPath);
+}
+ }
+
+ throw new IOException(
+ "Failed to upload block",
AzureBlobStorageClient.handleBlobException(e));
+}
+
+buffer.clear();
+ }
+
+ @Override
+ public void flush() throws IOException {
+if (closed) {
+ throw new IOException("Stream closed");
+}
+
+if (buffer.position() - buffer.arrayOffset() > 0) {
+ uploadBlock();
+}
+
+if (blockUpload != null) {
+ blockUpload.complete();
+ blockUpload = null;
+ committed = true;
+}
+ }
+
+ @Override
+ public void close() throws
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232162138
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
Review Comment:
removed the static http client
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232147739
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+this(
+createInternalClient(
+connectionString,
+endpoint,
+accountName,
+accountKey,
+sasToken,
+tenantId,
+clientId,
+clientSecret),
+containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String
containerName) {
+this.containerClient =
blobServiceClient.getBlobContainerClient(containerName);
+try {
+ containerClient.create();
+} catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+throw e;
+ }
+}
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+builder.httpClient(SHARED_HTTP_CLIENT);
+
+if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+} else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) &&
StrUtils.isNotNullOrEmpty(accountKey)) {
+builder.credential(
+new
com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+builder.sasToken(sasToken);
+ } else {
+TokenCredential credential = new
DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+builder.credential(credential);
+ }
+} else {
+ throw new IllegalArgumentException("Either connectionString or endpoint
must be provided");
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232137250
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+this(
+createInternalClient(
+connectionString,
+endpoint,
+accountName,
+accountKey,
+sasToken,
+tenantId,
+clientId,
+clientSecret),
+containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String
containerName) {
+this.containerClient =
blobServiceClient.getBlobContainerClient(containerName);
+try {
+ containerClient.create();
+} catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+throw e;
+ }
+}
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+builder.httpClient(SHARED_HTTP_CLIENT);
+
+if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+} else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) &&
StrUtils.isNotNullOrEmpty(accountKey)) {
+builder.credential(
+new
com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+builder.sasToken(sasToken);
+ } else {
+TokenCredential credential = new
DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+builder.credential(credential);
+ }
+} else {
+ throw new IllegalArgumentException("Either connectionString or endpoint
must be provided");
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232112168
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length)
{
+this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize,
int maxCachedPages) {
+super(path);
+this.path = path;
+this.client = client;
+this.length = length;
+this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+closed = true;
+cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ensureOpen();
+if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+}
+
+position = pos;
+ }
+
+ @Override
+ public long length() {
+return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length)
throws IOException {
+ensureOpen();
+if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ",
length=" + length);
+}
+
+AzureBlobIndexInput slice =
+new AzureBlobIndexInput(
Review Comment:
removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232111208
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobIndexInput.java:
##
@@ -0,0 +1,200 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import java.io.EOFException;
+import java.io.IOException;
+import java.io.InputStream;
+import java.util.LinkedHashMap;
+import java.util.Map;
+import org.apache.lucene.store.IndexInput;
+
+class AzureBlobIndexInput extends IndexInput {
+
+ private static final int MIN_PAGE_SIZE = 4 * 1024;
+ private static final int DEFAULT_PAGE_SIZE = 512 * 1024;
+ private static final int MAX_CACHED_PAGES = 128;
+
+ private final String path;
+ private final AzureBlobStorageClient client;
+ private final long length;
+ private final int pageSize;
+ private final LruPageCache cache;
+
+ private long position = 0L;
+ private boolean closed = false;
+
+ AzureBlobIndexInput(String path, AzureBlobStorageClient client, long length)
{
+this(path, client, length, DEFAULT_PAGE_SIZE, MAX_CACHED_PAGES);
+ }
+
+ AzureBlobIndexInput(
+ String path, AzureBlobStorageClient client, long length, int pageSize,
int maxCachedPages) {
+super(path);
+this.path = path;
+this.client = client;
+this.length = length;
+this.pageSize = Math.max(MIN_PAGE_SIZE, pageSize);
+this.cache = new LruPageCache(maxCachedPages);
+ }
+
+ @Override
+ public void close() throws IOException {
+closed = true;
+cache.clear();
+ }
+
+ @Override
+ public long getFilePointer() {
+return position;
+ }
+
+ @Override
+ public void seek(long pos) throws IOException {
+ensureOpen();
+if (pos < 0 || pos > length) {
+ throw new IOException("Seek position out of bounds: " + pos);
+}
+
+position = pos;
+ }
+
+ @Override
+ public long length() {
+return length;
+ }
+
+ @Override
+ public IndexInput slice(String sliceDescription, long offset, long length)
throws IOException {
+ensureOpen();
+if (offset < 0 || length < 0 || offset + length > this.length) {
+ throw new IOException("Slice out of bounds: offset=" + offset + ",
length=" + length);
+}
+
+AzureBlobIndexInput slice =
+new AzureBlobIndexInput(
+getFullSliceDescription(sliceDescription), client, length,
pageSize, MAX_CACHED_PAGES);
+
+slice.position = 0L;
+
+// Wrap client in a view that remaps range requests by adding base offset
+slice.clientViewBaseOffset = this.clientViewBaseOffset + offset;
+return slice;
+ }
+
+ @Override
+ public byte readByte() throws IOException {
+ensureOpen();
+if (position >= length) {
+ throw new EOFException("End of stream reached");
+}
+
+byte[] page = getPage(pageIndex(position));
+int inPageOffset = (int) (position % pageSize);
+byte value = page[inPageOffset];
+position += 1L;
+return value;
+ }
+
+ @Override
+ public void readBytes(byte[] b, int offset, int len) throws IOException {
+ensureOpen();
+if (len < 0) {
+ throw new IOException("Length must be non-negative");
+}
+
+if (position + len > length) {
+ throw new EOFException("End of stream reached");
+}
+
+int remaining = len;
+while (remaining > 0) {
+ long pageIdx = pageIndex(position);
+ byte[] page = getPage(pageIdx);
+ int inPageOffset = (int) (position % pageSize);
+ int toCopy = Math.min(remaining, pageSize - inPageOffset);
+ System.arraycopy(page, inPageOffset, b, offset + (len - remaining),
toCopy);
+ position += toCopy;
+ remaining -= toCopy;
+}
+ }
+
+ // Internal state for slices: base offset to add to all range requests
+ private long clientViewBaseOffset = 0L;
+
+ private byte[] getPage(long pageIdx) throws IOException {
+byte[] page = cache.get(pageIdx);
+if (page != null) {
+ return page;
+}
+
+long absoluteOffset = clientViewBaseOffset + pageIdx * (long) pageSize;
+int bytesToRead = (int) Math.min(pageSize, length - pageIdx * (long)
pageSize);
+if (bytesToRead <= 0) {
+ throw new EOFException("End of stream
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3232090766
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+this(
+createInternalClient(
+connectionString,
+endpoint,
+accountName,
+accountKey,
+sasToken,
+tenantId,
+clientId,
+clientSecret),
+containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String
containerName) {
+this.containerClient =
blobServiceClient.getBlobContainerClient(containerName);
+try {
+ containerClient.create();
+} catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+throw e;
+ }
+}
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+
+BlobServiceClientBuilder builder = new BlobServiceClientBuilder();
+builder.httpClient(SHARED_HTTP_CLIENT);
+
+if (StrUtils.isNotNullOrEmpty(connectionString)) {
+ builder.connectionString(connectionString);
+} else if (StrUtils.isNotNullOrEmpty(endpoint)) {
+ builder.endpoint(endpoint);
+ if (StrUtils.isNotNullOrEmpty(accountName) &&
StrUtils.isNotNullOrEmpty(accountKey)) {
+builder.credential(
+new
com.azure.storage.common.StorageSharedKeyCredential(accountName, accountKey));
+ } else if (StrUtils.isNotNullOrEmpty(sasToken)) {
+builder.sasToken(sasToken);
+ } else {
+TokenCredential credential = new
DefaultAzureCredentialBuilder().tenantId(tenantId).build();
+builder.credential(credential);
+ }
+} else {
+ throw new IllegalArgumentException("Either connectionString or endpoint
must be provided");
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3228749058
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobStorageClient.java:
##
@@ -0,0 +1,529 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.credential.TokenCredential;
+import com.azure.identity.DefaultAzureCredentialBuilder;
+import com.azure.storage.blob.BlobClient;
+import com.azure.storage.blob.BlobContainerClient;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.azure.storage.blob.models.BlobItem;
+import com.azure.storage.blob.models.BlobStorageException;
+import com.azure.storage.blob.models.ListBlobsOptions;
+import com.google.common.annotations.VisibleForTesting;
+import java.io.ByteArrayInputStream;
+import java.io.FilterInputStream;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.Locale;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.solr.common.util.ResumableInputStream;
+import org.apache.solr.common.util.StrUtils;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * Creates a {@link BlobServiceClient} for communicating with Azure Blob
Storage. Utilizes the
+ * default Azure credential provider chain.
+ */
+public class AzureBlobStorageClient {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_FILE_PATH_DELIMITER = "/";
+ private static final int HTTP_NOT_FOUND = 404;
+ private static final int HTTP_CONFLICT = 409;
+ private static final int SKIP_BUFFER_SIZE = 8192;
+ private static final int DELETE_BATCH_SIZE = 1000;
+
+ private static final com.azure.core.http.HttpClient SHARED_HTTP_CLIENT =
+ new com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder().build();
+
+ private final BlobContainerClient containerClient;
+
+ AzureBlobStorageClient(
+ String containerName,
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
+ String clientSecret) {
+this(
+createInternalClient(
+connectionString,
+endpoint,
+accountName,
+accountKey,
+sasToken,
+tenantId,
+clientId,
+clientSecret),
+containerName);
+ }
+
+ @VisibleForTesting
+ AzureBlobStorageClient(BlobServiceClient blobServiceClient, String
containerName) {
+this.containerClient =
blobServiceClient.getBlobContainerClient(containerName);
+try {
+ containerClient.create();
+} catch (BlobStorageException e) {
+ if (e.getStatusCode() != HTTP_CONFLICT) {
+throw e;
+ }
+}
+ }
+
+ private static BlobServiceClient createInternalClient(
+ String connectionString,
+ String endpoint,
+ String accountName,
+ String accountKey,
+ String sasToken,
+ String tenantId,
+ String clientId,
Review Comment:
using them now to build, ClientSecretCredentialBuilder. Helps with users
using service principal authentication method to do solr backup and restore to
azure blob storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
-
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r3228702224
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobBackupRepository.java:
##
@@ -0,0 +1,396 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.google.common.annotations.VisibleForTesting;
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.IOContext;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.backup.repository.AbstractBackupRepository;
+import org.apache.solr.core.backup.repository.BackupRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A concrete implementation of {@link BackupRepository} interface supporting
backup/restore of Solr
+ * indexes to Azure Blob Storage.
+ */
+public class AzureBlobBackupRepository extends AbstractBackupRepository {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_SCHEME = "blob";
+ private static final int CHUNK_SIZE = 16 * 1024 * 1024;
+ private static final int COPY_BUFFER_SIZE = 8192;
+
+ private AzureBlobStorageClient client;
+
+ @Override
+ public void init(NamedList args) {
+super.init(args);
+AzureBlobBackupRepositoryConfig backupConfig = new
AzureBlobBackupRepositoryConfig(this.config);
+
+if (client != null) {
+ client.close();
+}
+
+this.client = backupConfig.buildClient();
+ }
+
+ @VisibleForTesting
+ public void setClient(AzureBlobStorageClient client) {
+this.client = client;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T getConfigProperty(String name) {
+return (T) this.config.get(name);
+ }
+
+ @Override
+ public URI createURI(String location) {
+if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty
location");
+}
+
+URI result;
+try {
+ if (location.startsWith(BLOB_SCHEME + ":")) {
+result = new URI(location);
+ } else if (location.startsWith("/")) {
+result = new URI(BLOB_SCHEME, "", location, null);
+ } else {
+result = new URI(BLOB_SCHEME, "", "/" + location, null);
+ }
+ return result;
+} catch (URISyntaxException ex) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ex);
+}
+ }
+
+ @Override
+ public URI createDirectoryURI(String location) {
+if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty
location");
+}
+
+if (!location.endsWith("/")) {
+ location += "/";
+}
+
+return createURI(location);
+ }
+
+ @Override
+ public URI resolve(URI baseUri, String... pathComponents) {
+if (!BLOB_SCHEME.equalsIgnoreCase(baseUri.getScheme())) {
+ throw new IllegalArgumentException("URI must begin with 'blob:' scheme");
+}
+
+String path = baseUri + "/" + String.join("/", pathComponents);
+return URI.create(path).normalize();
+ }
+
+ @Override
+ public URI resolveDirectory(URI baseUri, String... pathComponents) {
+if (pathComponents.length > 0) {
+ if (!pathComponents[pathComponents.length - 1].endsWith("/")) {
+pathComponents[pathComponents.length - 1] =
pathComponents[pathComponents.length - 1] + "/";
+ }
+} else {
+ if (!baseUri.toString().endsWith("/")) {
+baseUri = URI.create(baseUri + "/");
+ }
+}
+return resolve(baseUri, pathComponents);
+ }
+
+ @Override
+ public void createDirectory(URI path) throws
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4428454450 > Only reviewed the production code (skiped tests and other stuff). > > Some general comments on the review: > > * can you replace all unnecessary qualified class names by imports? (replacing `java.lang.String` by `String`) This makes the code painful to read > * there is a lot of shared logic between Azure backups and S3 backups. Lot of code is duplicated. I wonder if abstract classes should be introduced to factor out some logic > * we always upload files with a _block list_. At least for small files, can we keep things simples with a simple pull? It think we may skip the commit call. > * overall, I think it's pretty inefficient that `AzureBlobIndexInput` always reads files by small pages. Mostly for backup/restore, we will only fetch full files. You should not do a request by page, or use much bigger pages (without keeping them in memory). > > Please note I did a code review, but I have no way to test it with Azure. @psalagnac , thank you for taking the time to review. 1) made the change by replacing all unnecessary qualified class names by imports 2) s3-repository is in active production use; a refactor would force shared abstractions and re-testing on a stable module to land a brand-new one. That's significant scope creep with no end-user benefit. Also the duplication is mostly in boilerplate (resolveDirectory shape, sanitization helpers); both repositories have different SDK contracts underneath. Let me know what you think. 3) Given the comments on AzureBlobIndexInput, I have rewritten AzureBlobIndexInput to extend Lucene's standard BufferedIndexInput instead of IndexInput directly. Now only override readInternal(ByteBuffer) and seekInternal(long), and everything else (buffer management, slice, clone, EOF handling) comes from the parent class. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4428441333 @psalagnac , thank you for taking the time to review. 1. can you replace all unnecessary qualified class names by imports? made the change by replacing all unnecessary qualified class names by imports 2. there is a lot of shared logic between Azure backups and S3 backups. Lot of code is duplicated. I wonder if abstract classes should be introduced to factor out some logic? - s3-repository is in active production use; a refactor would force shared abstractions and re-testing on a stable module to land a brand-new one. That's significant scope creep with no end-user benefit. Also the duplication is mostly in boilerplate (resolveDirectory shape, sanitization helpers); both repositories have different SDK contracts underneath. Let me know what you think. 3. overall, I think it's pretty inefficient that AzureBlobIndexInput always reads files by small pages. Mostly for backup/restore, we will only fetch full files. You should not do a request by page, or use much bigger pages (without keeping them in memory) ? Given the comments on AzureBlobIndexInput, I have rewritten AzureBlobIndexInput to extend Lucene's standard BufferedIndexInput instead of IndexInput directly. Now only override readInternal(ByteBuffer) and seekInternal(long), and everything else (buffer management, slice, clone, EOF handling) comes from the parent class. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
psalagnac commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4326968039 Thanks for adding me as a reviewer. I've been pretty busy recently and I haven't got the chance to do the review yet. I'll try to do a complete review before end of next week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4290990310 Thanks @janhoy for adding more reviewers. @HoustonPutman and @psalagnac ,let me know if you have any questions related to the pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
github-actions[bot] commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-4064286390 This PR has had no activity for 60 days and is now labeled as stale. Any new activity will remove the stale label. To attract more reviewers, please tag people who might be familiar with the code area and/or notify the [email protected] mailing list. To exempt this PR from being marked as stale, make it a draft PR or add the label "exempt-stale". If left unattended, this PR will be closed after another 60 days of inactivity. Thank you for your contribution! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-3600575623 Hi All, I’ve fixed the recent CI failures: 1. Added proper thread leak filters for Testcontainers / JNA so tests are cleanly skipped when Docker isn’t available in CI. 2. Fixed two error-prone findings (IntLongMath and MissingOverride) that only surfaced in CI because error-prone is disabled by default in local builds unless -Pvalidation.errorprone=true is used. Please let me know if anything else should be adjusted to help move the review forward. I’m happy to iterate further as needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
pratsgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-3594909885 Hi all, I’ve pushed a set of updates based on the feedback so far: 1. Switched the integration tests to use Testcontainers with Azurite (no hard-coded ports / external prereqs) and disabled the Security Manager for this module’s tests, following the pattern used in the extraction module. 2. Addressed HTTP client concerns by using azure-core-http-okhttp and keeping those deps in test scope only, plus explicit test deps. 3. Cleaned up and significantly shortened the README and docs 4. Did additional cleanup to align naming, structure, and configuration patterns with those of the existing repository plugins. Please let me know if anything else should be adjusted that would help move the review forward. Happy to make further changes. Thanks again for the thoughtful review and guidance so far! — Prateek -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2563739297 ## solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc: ## @@ -383,7 +383,7 @@ If the status is anything other than "success", an error message will explain wh Solr provides a repository abstraction to allow users to backup and restore their data to a variety of different storage systems. For example, a Solr cluster running on a local filesystem (e.g., EXT3) can store backup data on the same disk, on a remote network-mounted drive, or in some popular "cloud storage" providers, depending on the 'repository' implementation chosen. -Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository` and `S3BackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it. +Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository`, `S3BackupRepository`, and `BlobBackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it. Review Comment: @janhoy Thank you for taking the time and effort to review the PR and providing valuable feedback. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563707779
##
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * Uses ThreadLeakFilters to ignore OkHttp's global singleton threads
(TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a
standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+})
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+// Create a single shared OkHttpClient for all tests
+// This reuses OkHttp's global thread pools efficiently
+sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+// Check if Azurite is running before attempting to connect
+assumeAzuriteIsRunning();
+
+setAzureTestCredentials();
+
+// Use Azurite connection string for local testing
+connectionString =
+
"DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:1/devstoreaccount1;";;
+
+// Put a proxy in front of Azurite to simulate connection loss like S3
tests
+proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+// Reuse the shared OkHttpClient
+HttpClient httpClient = new
OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+// Route Blob endpoint through the proxy by adjusting the connection string
+String proxiedConn = connectionString.replace(":1", ":" +
proxy.getListenPort());
+BlobServiceClient blobServiceClient =
+new BlobServiceClientBuilder()
+.connectionString(proxiedConn)
+.httpClient(httpClient)
+.buildClient();
+
+containerName = "test-" + java.util.UUID.randomUUID();
+client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials
during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test
environment.
+ */
+ public static void setAzureTestCredentials() {
+// Set test Azure credentials to avoid using real credentials
+System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+// Set Azurite-specific environment variables
+System.setProperty(
+
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563445786
##
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * Uses ThreadLeakFilters to ignore OkHttp's global singleton threads
(TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a
standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+})
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+// Create a single shared OkHttpClient for all tests
+// This reuses OkHttp's global thread pools efficiently
+sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+// Check if Azurite is running before attempting to connect
+assumeAzuriteIsRunning();
+
+setAzureTestCredentials();
+
+// Use Azurite connection string for local testing
+connectionString =
+
"DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:1/devstoreaccount1;";;
+
+// Put a proxy in front of Azurite to simulate connection loss like S3
tests
+proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+// Reuse the shared OkHttpClient
+HttpClient httpClient = new
OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+// Route Blob endpoint through the proxy by adjusting the connection string
+String proxiedConn = connectionString.replace(":1", ":" +
proxy.getListenPort());
+BlobServiceClient blobServiceClient =
+new BlobServiceClientBuilder()
+.connectionString(proxiedConn)
+.httpClient(httpClient)
+.buildClient();
+
+containerName = "test-" + java.util.UUID.randomUUID();
+client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials
during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test
environment.
+ */
+ public static void setAzureTestCredentials() {
+// Set test Azure credentials to avoid using real credentials
+System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+// Set Azurite-specific environment variables
+System.setProperty(
+
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563445786
##
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * Uses ThreadLeakFilters to ignore OkHttp's global singleton threads
(TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a
standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+})
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+// Create a single shared OkHttpClient for all tests
+// This reuses OkHttp's global thread pools efficiently
+sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+// Check if Azurite is running before attempting to connect
+assumeAzuriteIsRunning();
+
+setAzureTestCredentials();
+
+// Use Azurite connection string for local testing
+connectionString =
+
"DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:1/devstoreaccount1;";;
+
+// Put a proxy in front of Azurite to simulate connection loss like S3
tests
+proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+// Reuse the shared OkHttpClient
+HttpClient httpClient = new
OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+// Route Blob endpoint through the proxy by adjusting the connection string
+String proxiedConn = connectionString.replace(":1", ":" +
proxy.getListenPort());
+BlobServiceClient blobServiceClient =
+new BlobServiceClientBuilder()
+.connectionString(proxiedConn)
+.httpClient(httpClient)
+.buildClient();
+
+containerName = "test-" + java.util.UUID.randomUUID();
+client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials
during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test
environment.
+ */
+ public static void setAzureTestCredentials() {
+// Set test Azure credentials to avoid using real credentials
+System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+// Set Azurite-specific environment variables
+System.setProperty(
+
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563445786
##
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * Uses ThreadLeakFilters to ignore OkHttp's global singleton threads
(TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a
standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+})
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+// Create a single shared OkHttpClient for all tests
+// This reuses OkHttp's global thread pools efficiently
+sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+// Check if Azurite is running before attempting to connect
+assumeAzuriteIsRunning();
+
+setAzureTestCredentials();
+
+// Use Azurite connection string for local testing
+connectionString =
+
"DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:1/devstoreaccount1;";;
+
+// Put a proxy in front of Azurite to simulate connection loss like S3
tests
+proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+// Reuse the shared OkHttpClient
+HttpClient httpClient = new
OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+// Route Blob endpoint through the proxy by adjusting the connection string
+String proxiedConn = connectionString.replace(":1", ":" +
proxy.getListenPort());
+BlobServiceClient blobServiceClient =
+new BlobServiceClientBuilder()
+.connectionString(proxiedConn)
+.httpClient(httpClient)
+.buildClient();
+
+containerName = "test-" + java.util.UUID.randomUUID();
+client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials
during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test
environment.
+ */
+ public static void setAzureTestCredentials() {
+// Set test Azure credentials to avoid using real credentials
+System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+// Set Azurite-specific environment variables
+System.setProperty(
+
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2563445786
##
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * Uses ThreadLeakFilters to ignore OkHttp's global singleton threads
(TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a
standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+})
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+// Create a single shared OkHttpClient for all tests
+// This reuses OkHttp's global thread pools efficiently
+sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+// Check if Azurite is running before attempting to connect
+assumeAzuriteIsRunning();
+
+setAzureTestCredentials();
+
+// Use Azurite connection string for local testing
+connectionString =
+
"DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:1/devstoreaccount1;";;
+
+// Put a proxy in front of Azurite to simulate connection loss like S3
tests
+proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+// Reuse the shared OkHttpClient
+HttpClient httpClient = new
OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+// Route Blob endpoint through the proxy by adjusting the connection string
+String proxiedConn = connectionString.replace(":1", ":" +
proxy.getListenPort());
+BlobServiceClient blobServiceClient =
+new BlobServiceClientBuilder()
+.connectionString(proxiedConn)
+.httpClient(httpClient)
+.buildClient();
+
+containerName = "test-" + java.util.UUID.randomUUID();
+client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials
during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test
environment.
+ */
+ public static void setAzureTestCredentials() {
+// Set test Azure credentials to avoid using real credentials
+System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+// Set Azurite-specific environment variables
+System.setProperty(
+
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-3574263302 I'll leave the rest of the review to others more proficient in Azure Blob than me (have never used it). I'd love to test it with a read AzBlob but I'll leave that to someone who already have an active account. I have a concern that the PR *feels* largely AI generated(?), lacking the care to details that we require for contributions. Have me excused @prateeksinghalgit if this is not correct, it was just a hunch I got while reviewing. I'd rather have a short well thought out README with useful advice for *users* than three pages of detailed step by step instructions for testing and developing the feature. For other reviewers to pick up where I left, consider in partifular the HTTP client choice, correct licensing and avoiding hard coded ports in tests. I have *not* done a complete review, not read the ref-guide part at all, just commented on things me and Copilot saw immediately. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2558931280 ## solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc: ## @@ -383,7 +383,7 @@ If the status is anything other than "success", an error message will explain wh Solr provides a repository abstraction to allow users to backup and restore their data to a variety of different storage systems. For example, a Solr cluster running on a local filesystem (e.g., EXT3) can store backup data on the same disk, on a remote network-mounted drive, or in some popular "cloud storage" providers, depending on the 'repository' implementation chosen. -Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository` and `S3BackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it. +Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository`, `S3BackupRepository`, and `BlobBackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it. Review Comment: Thanks for quick and efficient implementation of review comments @prateeksinghalgit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2558918492
##
solr/modules/azure-blob-repository/src/test/org/apache/solr/azureblob/AbstractAzureBlobClientTest.java:
##
@@ -0,0 +1,249 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.azureblob;
+
+import com.azure.core.http.HttpClient;
+import com.azure.core.http.okhttp.OkHttpAsyncHttpClientBuilder;
+import com.azure.storage.blob.BlobServiceClient;
+import com.azure.storage.blob.BlobServiceClientBuilder;
+import com.carrotsearch.randomizedtesting.ThreadFilter;
+import com.carrotsearch.randomizedtesting.annotations.ThreadLeakFilters;
+import java.io.IOException;
+import java.io.OutputStream;
+import java.net.Socket;
+import java.nio.charset.StandardCharsets;
+import okhttp3.OkHttpClient;
+import org.apache.solr.SolrTestCaseJ4;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import static org.junit.Assume.assumeTrue;
+
+/**
+ * Abstract class for tests with Azure Blob Storage emulator.
+ *
+ * Uses ThreadLeakFilters to ignore OkHttp's global singleton threads
(TaskRunner and Okio
+ * Watchdog) which are JVM-wide and cannot be shut down per-client. This is a
standard pattern in
+ * Solr tests for HTTP clients with global thread pools.
+ */
+@ThreadLeakFilters(
+filters = {
+ AbstractAzureBlobClientTest.OkHttpThreadLeakFilter.class,
+})
+public class AbstractAzureBlobClientTest extends SolrTestCaseJ4 {
+
+ protected String containerName;
+
+ @Rule public TemporaryFolder tempFolder = new TemporaryFolder();
+
+ AzureBlobStorageClient client;
+ private static String connectionString;
+ // Shared OkHttpClient across all tests to reuse the global thread pools
+ private static OkHttpClient sharedOkHttpClient;
+ protected org.apache.solr.client.solrj.cloud.SocketProxy proxy;
+
+ @BeforeClass
+ public static void setUpClass() {
+// Create a single shared OkHttpClient for all tests
+// This reuses OkHttp's global thread pools efficiently
+sharedOkHttpClient = new OkHttpClient.Builder().build();
+ }
+
+ @Before
+ public void setUpClient() throws Exception {
+// Check if Azurite is running before attempting to connect
+assumeAzuriteIsRunning();
+
+setAzureTestCredentials();
+
+// Use Azurite connection string for local testing
+connectionString =
+
"DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:1/devstoreaccount1;";;
+
+// Put a proxy in front of Azurite to simulate connection loss like S3
tests
+proxy = new org.apache.solr.client.solrj.cloud.SocketProxy();
+proxy.open(new java.net.URI(getBlobServiceUrl()));
+
+// Reuse the shared OkHttpClient
+HttpClient httpClient = new
OkHttpAsyncHttpClientBuilder(sharedOkHttpClient).build();
+
+// Route Blob endpoint through the proxy by adjusting the connection string
+String proxiedConn = connectionString.replace(":1", ":" +
proxy.getListenPort());
+BlobServiceClient blobServiceClient =
+new BlobServiceClientBuilder()
+.connectionString(proxiedConn)
+.httpClient(httpClient)
+.buildClient();
+
+containerName = "test-" + java.util.UUID.randomUUID();
+client = new AzureBlobStorageClient(blobServiceClient, containerName);
+ }
+
+ /**
+ * Set up Azure test credentials to avoid using real Azure credentials
during testing. Similar to
+ * how S3 tests use ProfileFileSystemSetting to avoid polluting the test
environment.
+ */
+ public static void setAzureTestCredentials() {
+// Set test Azure credentials to avoid using real credentials
+System.setProperty("AZURE_CLIENT_ID", "test-client-id");
+System.setProperty("AZURE_TENANT_ID", "test-tenant-id");
+System.setProperty("AZURE_CLIENT_SECRET", "test-client-secret");
+
+// Set Azurite-specific environment variables
+System.setProperty(
+"AZURE_STORAGE_CONNECTION_STRING",
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2558060657 ## solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc: ## @@ -383,7 +383,7 @@ If the status is anything other than "success", an error message will explain wh Solr provides a repository abstraction to allow users to backup and restore their data to a variety of different storage systems. For example, a Solr cluster running on a local filesystem (e.g., EXT3) can store backup data on the same disk, on a remote network-mounted drive, or in some popular "cloud storage" providers, depending on the 'repository' implementation chosen. -Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository` and `S3BackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it. +Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository`, `S3BackupRepository`, and `BlobBackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it. Review Comment: made the change -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2556609430 ## solr/solr-ref-guide/modules/deployment-guide/pages/backup-restore.adoc: ## @@ -383,7 +383,7 @@ If the status is anything other than "success", an error message will explain wh Solr provides a repository abstraction to allow users to backup and restore their data to a variety of different storage systems. For example, a Solr cluster running on a local filesystem (e.g., EXT3) can store backup data on the same disk, on a remote network-mounted drive, or in some popular "cloud storage" providers, depending on the 'repository' implementation chosen. -Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository` and `S3BackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it. +Solr offers multiple different repository implementations out of the box (`LocalFileSystemRepository`, `GCSBackupRepository`, `S3BackupRepository`, and `BlobBackupRepository`), and allows users to create plugins for their own storage systems as needed. It is also possible to create a `DelegatingBackupRepository` that delegates to another `BackupRepository` and adds or modifies some behavior on top of it. Review Comment: Update class name in docs -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2555407396 ## gradle/libs.versions.toml: ## @@ -51,6 +51,10 @@ aqute-bnd = "6.4.1" asciidoctor-mathjax = "0.0.9" # @keep Asciidoctor tabs version used in ref-guide asciidoctor-tabs = "1.0.0-beta.6" +azure-storage = "12.25.0" +azure-identity = "1.12.0" +azure-core = "1.52.0" +azure-core-http-netty = "1.15.4" Review Comment: Yes, made the change to use azure-core-http-okhttp. I was having issues with using azure-core-http-jdk-httpclient. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2555407396 ## gradle/libs.versions.toml: ## @@ -51,6 +51,10 @@ aqute-bnd = "6.4.1" asciidoctor-mathjax = "0.0.9" # @keep Asciidoctor tabs version used in ref-guide asciidoctor-tabs = "1.0.0-beta.6" +azure-storage = "12.25.0" +azure-identity = "1.12.0" +azure-core = "1.52.0" +azure-core-http-netty = "1.15.4" Review Comment: Yes, made the change to use azure-core-http-okhttp. I was having issue with using azure-core-http-jdk-httpclient. Let me know if using azure-core-http-okhttp would be fine? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2554975363 ## gradle/libs.versions.toml: ## @@ -51,6 +51,10 @@ aqute-bnd = "6.4.1" asciidoctor-mathjax = "0.0.9" # @keep Asciidoctor tabs version used in ref-guide asciidoctor-tabs = "1.0.0-beta.6" +azure-storage = "12.25.0" +azure-identity = "1.12.0" +azure-core = "1.52.0" +azure-core-http-netty = "1.15.4" Review Comment: Did you have a chance to look at the http client? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2554734264 ## solr/licenses/msal4j-NOTICE.txt: ## @@ -0,0 +1,12 @@ +Microsoft Authentication Library for Java (MSAL4J) +Copyright (c) Microsoft Corporation. All rights reserved. + +This product includes software developed by +Microsoft Corporation (https://github.com/AzureAD/microsoft-authentication-library-for-java). + +Licensed under the MIT License. Review Comment: made the change. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2553391024 ## solr/licenses/msal4j-NOTICE.txt: ## @@ -0,0 +1,12 @@ +Microsoft Authentication Library for Java (MSAL4J) +Copyright (c) Microsoft Corporation. All rights reserved. + +This product includes software developed by +Microsoft Corporation (https://github.com/AzureAD/microsoft-authentication-library-for-java). + +Licensed under the MIT License. Review Comment: So the msal4j lib is licensed under MIT license. Then you need to change the LICENSE file to ba `-MIT` and the package probably does not require a NOTICE file at all. Please check that correct license (and if ASL, also NOTICE file) is in place for each new dependency. Copy exactly the license and notice they provide with the lib. There may be issues with more than this one. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750:
URL: https://github.com/apache/solr/pull/3750#discussion_r2553346877
##
solr/modules/azure-blob-repository/src/java/org/apache/solr/azureblob/AzureBlobBackupRepository.java:
##
@@ -0,0 +1,407 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.solr.blob;
+
+import java.io.IOException;
+import java.io.InputStream;
+import java.io.OutputStream;
+import java.lang.invoke.MethodHandles;
+import java.net.URI;
+import java.net.URISyntaxException;
+import java.time.Duration;
+import java.time.Instant;
+import java.util.Collection;
+import java.util.Objects;
+import java.util.Set;
+import java.util.stream.Collectors;
+import org.apache.lucene.store.Directory;
+import org.apache.lucene.store.IOContext;
+import org.apache.lucene.store.IndexInput;
+import org.apache.lucene.store.IndexOutput;
+import org.apache.solr.common.SolrException;
+import org.apache.solr.common.util.NamedList;
+import org.apache.solr.common.util.StrUtils;
+import org.apache.solr.core.backup.repository.AbstractBackupRepository;
+import org.apache.solr.core.backup.repository.BackupRepository;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+/**
+ * A concrete implementation of {@link BackupRepository} interface supporting
backup/restore of Solr
+ * indexes to Azure Blob Storage.
+ */
+public class BlobBackupRepository extends AbstractBackupRepository {
+
+ private static final Logger log =
LoggerFactory.getLogger(MethodHandles.lookup().lookupClass());
+
+ static final String BLOB_SCHEME = "blob";
+ private static final int CHUNK_SIZE = 16 * 1024 * 1024;
+
+ private BlobStorageClient client;
+
+ @Override
+ public void init(NamedList args) {
+super.init(args);
+BlobBackupRepositoryConfig backupConfig = new
BlobBackupRepositoryConfig(this.config);
+
+// If a client was already created, close it to avoid any resource leak
+if (client != null) {
+ client.close();
+}
+
+this.client = backupConfig.buildClient();
+ }
+
+ // Method to inject a mock client for testing
+ public void setClient(BlobStorageClient client) {
+this.client = client;
+ }
+
+ @Override
+ @SuppressWarnings("unchecked")
+ public T getConfigProperty(String name) {
+return (T) this.config.get(name);
+ }
+
+ @Override
+ public URI createURI(String location) {
+if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty
location");
+}
+
+URI result;
+try {
+ if (location.startsWith(BLOB_SCHEME + ":")) {
+result = new URI(location);
+ } else if (location.startsWith("/")) {
+result = new URI(BLOB_SCHEME, "", location, null);
+ } else {
+result = new URI(BLOB_SCHEME, "", "/" + location, null);
+ }
+ return result;
+} catch (URISyntaxException ex) {
+ throw new SolrException(SolrException.ErrorCode.BAD_REQUEST, ex);
+}
+ }
+
+ @Override
+ public URI createDirectoryURI(String location) {
+if (StrUtils.isNullOrEmpty(location)) {
+ throw new IllegalArgumentException("cannot create URI with an empty
location");
+}
+
+if (!location.endsWith("/")) {
+ location += "/";
+}
+
+return createURI(location);
+ }
+
+ @Override
+ public URI resolve(URI baseUri, String... pathComponents) {
+if (!BLOB_SCHEME.equalsIgnoreCase(baseUri.getScheme())) {
+ throw new IllegalArgumentException("URI must begin with 'blob:' scheme");
+}
+
+String path = baseUri + "/" + String.join("/", pathComponents);
+return URI.create(path).normalize();
+ }
+
+ @Override
+ public URI resolveDirectory(URI baseUri, String... pathComponents) {
+if (pathComponents.length > 0) {
+ if (!pathComponents[pathComponents.length - 1].endsWith("/")) {
+pathComponents[pathComponents.length - 1] =
pathComponents[pathComponents.length - 1] + "/";
+ }
+} else {
+ if (!baseUri.toString().endsWith("/")) {
+baseUri = URI.create(baseUri + "/");
+ }
+}
+return resolve(baseUri, pathComponents);
+ }
+
+ @Override
+ public void createDirectory(URI path) throws IOException {
+
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2553345594 ## solr/licenses/azure-NOTICE.txt: ## @@ -0,0 +1,25 @@ +AWS SDK for Java 2.0 +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +This product includes software developed by +Amazon Technologies, Inc (http://www.amazon.com/). + +** +THIRD PARTY COMPONENTS +** +This software includes third party software subject to the following copyrights: +- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty. +- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. Review Comment: updated it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2553345413 ## solr/licenses/reactor-NOTICE.txt: ## @@ -0,0 +1,25 @@ +AWS SDK for Java 2.0 +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +This product includes software developed by +Amazon Technologies, Inc (http://www.amazon.com/). Review Comment: updated it. ## solr/licenses/msal4j-NOTICE.txt: ## @@ -0,0 +1,25 @@ +AWS SDK for Java 2.0 +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +This product includes software developed by +Amazon Technologies, Inc (http://www.amazon.com/). + +** +THIRD PARTY COMPONENTS +** +This software includes third party software subject to the following copyrights: +- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty. +- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. +- Apache Commons Lang - https://github.com/apache/commons-lang +- Netty Reactive Streams - https://github.com/playframework/netty-reactive-streams +- Jackson-core - https://github.com/FasterXML/jackson-core +- Jackson-dataformat-cbor - https://github.com/FasterXML/jackson-dataformats-binary + +The licenses for these third party components are included in LICENSE.txt + Review Comment: updated it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
prateeksinghalgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-3567034117 > Only immediate comment is naming. «Blob-repository» is too generic. Should contain word «azure»? made the change to azure-blob-repository -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2544753855 ## gradle/libs.versions.toml: ## @@ -51,6 +51,10 @@ aqute-bnd = "6.4.1" asciidoctor-mathjax = "0.0.9" # @keep Asciidoctor tabs version used in ref-guide asciidoctor-tabs = "1.0.0-beta.6" +azure-storage = "12.25.0" +azure-identity = "1.12.0" +azure-core = "1.52.0" +azure-core-http-netty = "1.15.4" Review Comment: I see this PR adds a bunch of netty httpclient dependencies. We already have Netty as a dependency with version 4.2.6 but azure brings in v4.1.110 in addition. Either force the use of 4.2.6 for azure-blob if possible, or select another http-client, switch to the `azure-core-http-jdk-httpclient` client or make a wrapper around the jetty client (probably a tall order). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
Copilot commented on code in PR #3750: URL: https://github.com/apache/solr/pull/3750#discussion_r2544701213 ## solr/licenses/msal4j-NOTICE.txt: ## @@ -0,0 +1,25 @@ +AWS SDK for Java 2.0 +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +This product includes software developed by +Amazon Technologies, Inc (http://www.amazon.com/). + +** +THIRD PARTY COMPONENTS +** +This software includes third party software subject to the following copyrights: +- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty. +- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. +- Apache Commons Lang - https://github.com/apache/commons-lang +- Netty Reactive Streams - https://github.com/playframework/netty-reactive-streams +- Jackson-core - https://github.com/FasterXML/jackson-core +- Jackson-dataformat-cbor - https://github.com/FasterXML/jackson-dataformats-binary + +The licenses for these third party components are included in LICENSE.txt + Review Comment: Incorrect NOTICE file content: This file contains AWS/Amazon copyright information but should contain Microsoft copyright information for the MSAL4J library. ```suggestion Microsoft Authentication Library for Java (MSAL4J) Copyright (c) Microsoft Corporation. All rights reserved. This product includes software developed by Microsoft Corporation (https://github.com/AzureAD/microsoft-authentication-library-for-java). Licensed under the MIT License. ** THIRD PARTY COMPONENTS ** This software may include third party software subject to the following copyrights: ``` ## solr/licenses/azure-NOTICE.txt: ## @@ -0,0 +1,25 @@ +AWS SDK for Java 2.0 +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +This product includes software developed by +Amazon Technologies, Inc (http://www.amazon.com/). + +** +THIRD PARTY COMPONENTS +** +This software includes third party software subject to the following copyrights: +- XML parsing and utility functions from JetS3t - Copyright 2006-2009 James Murty. +- PKCS#1 PEM encoded private key parsing and utility functions from oauth.googlecode.com - Copyright 1998-2010 AOL Inc. Review Comment: Incorrect NOTICE file content: This file contains AWS/Amazon copyright information but should contain Azure/Microsoft copyright information for the Azure SDK libraries. ```suggestion Azure SDK for Java Copyright (c) Microsoft Corporation. All rights reserved. This product includes software developed by Microsoft Corporation (https://www.microsoft.com/). ** THIRD PARTY COMPONENTS ** This software includes third party software subject to the following copyrights: ``` ## solr/licenses/reactor-NOTICE.txt: ## @@ -0,0 +1,25 @@ +AWS SDK for Java 2.0 +Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. + +This product includes software developed by +Amazon Technologies, Inc (http://www.amazon.com/). Review Comment: Incorrect NOTICE file content: This file contains AWS/Amazon copyright information but should contain Project Reactor copyright information. ## solr/modules/blob-repository/src/java/org/apache/solr/blob/BlobBackupRepository.java: ## @@ -0,0 +1,407 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.solr.blob; + +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.lang.invoke.MethodHandles; +import java.net.URI; +import java.net.URISyntaxException; +import java.time.Duration; +import java.time.Instant; +import java.util.Collection; +import java.util.Objects; +import java.util.Set; +import java.util.stream.Collectors; +import org.apache.lucene.store.Directory; +import org.apache.lucene.store.IOContext; +import org.apache.lucene.store.IndexInput; +import org.apache.lucene.store.IndexOutput; +import org.apache.solr.common.SolrException; +import
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
janhoy commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-3556348621 Don’t have time to review but asked copilot for an opinion 😉 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
Re: [PR] SOLR-17949: Add Azure Blob Storage backup repository module [solr]
pratsgit commented on PR #3750: URL: https://github.com/apache/solr/pull/3750#issuecomment-3555962712 Quick update: following up on the dev@ thread. To clarify scope — although the diff is large, the actual implementation is centered in 8 main files and 8 test files; the rest are license header updates. Happy to split the PR into smaller logical chunks (core module / tests / docs) if that helps with review. Thanks everyone! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] - To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
