[
https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17083074#comment-17083074
]
ASF GitHub Bot commented on NUTCH-2757:
---------------------------------------
balashashanka commented on pull request #508: NUTCH-2757 : Indexer-elastic: add
authentication options
URL: https://github.com/apache/nutch/pull/508#discussion_r408027978
##########
File path:
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
##########
@@ -114,57 +136,78 @@ public void open(IndexWriterParams parameters) throws
IOException {
DEFAULT_BULK_CLOSE_TIMEOUT);
defaultIndex = parameters.get(ElasticConstants.INDEX, DEFAULT_INDEX);
- maxBulkDocs = parameters
- .getInt(ElasticConstants.MAX_BULK_DOCS, DEFAULT_MAX_BULK_DOCS);
- maxBulkLength = parameters
- .getInt(ElasticConstants.MAX_BULK_LENGTH, DEFAULT_MAX_BULK_LENGTH);
- expBackoffMillis = parameters
- .getInt(ElasticConstants.EXPONENTIAL_BACKOFF_MILLIS,
- DEFAULT_EXP_BACKOFF_MILLIS);
- expBackoffRetries = parameters
- .getInt(ElasticConstants.EXPONENTIAL_BACKOFF_RETRIES,
- DEFAULT_EXP_BACKOFF_RETRIES);
+ maxBulkDocs = parameters.getInt(ElasticConstants.MAX_BULK_DOCS,
+ DEFAULT_MAX_BULK_DOCS);
+ maxBulkLength = parameters.getInt(ElasticConstants.MAX_BULK_LENGTH,
+ DEFAULT_MAX_BULK_LENGTH);
+ expBackoffMillis = parameters.getInt(
+ ElasticConstants.EXPONENTIAL_BACKOFF_MILLIS,
+ DEFAULT_EXP_BACKOFF_MILLIS);
+ expBackoffRetries = parameters.getInt(
+ ElasticConstants.EXPONENTIAL_BACKOFF_RETRIES,
+ DEFAULT_EXP_BACKOFF_RETRIES);
client = makeClient(parameters);
LOG.debug("Creating BulkProcessor with maxBulkDocs={}, maxBulkLength={}",
maxBulkDocs, maxBulkLength);
- bulkProcessor = BulkProcessor.builder(
- (request, bulkListener) ->
- client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener),
- bulkProcessorListener())
+ bulkProcessor = BulkProcessor
+ .builder((request, bulkListener) -> client.bulkAsync(request,
+ RequestOptions.DEFAULT, bulkListener), bulkProcessorListener())
.setBulkActions(maxBulkDocs)
.setBulkSize(new ByteSizeValue(maxBulkLength, ByteSizeUnit.BYTES))
- .setConcurrentRequests(1).setBackoffPolicy(BackoffPolicy
- .exponentialBackoff(TimeValue.timeValueMillis(expBackoffMillis),
- expBackoffRetries)).build();
+ .setConcurrentRequests(1)
+ .setBackoffPolicy(BackoffPolicy.exponentialBackoff(
+ TimeValue.timeValueMillis(expBackoffMillis), expBackoffRetries))
+ .build();
}
/**
* Generates a RestHighLevelClient with the hosts given
*/
- protected RestHighLevelClient makeClient(IndexWriterParams parameters)
throws IOException {
+ protected RestHighLevelClient makeClient(IndexWriterParams parameters)
+ throws IOException {
hosts = parameters.getStrings(ElasticConstants.HOSTS);
port = parameters.getInt(ElasticConstants.PORT, DEFAULT_PORT);
+ user = parameters.get(ElasticConstants.USER, DEFAULT_USER);
+ password = parameters.get(ElasticConstants.PASSWORD, "");
+
+ final CredentialsProvider credentialsProvider = new
BasicCredentialsProvider();
+ credentialsProvider.setCredentials(AuthScope.ANY,
+ new UsernamePasswordCredentials(user, password));
+
RestHighLevelClient client = null;
if (hosts != null && port > 1) {
HttpHost[] hostsList = new HttpHost[hosts.length];
int i = 0;
- for(String host: hosts) {
+ for (String host : hosts) {
hostsList[i++] = new HttpHost(host, port);
}
RestClientBuilder restClientBuilder = RestClient.builder(hostsList);
if (StringUtils.isNotBlank(cluster)) {
- Header[] defaultHeaders = new Header[]{new BasicHeader("cluster.name",
cluster)};
+ Header[] defaultHeaders = new Header[] {
+ new BasicHeader("cluster.name", cluster) };
restClientBuilder.setDefaultHeaders(defaultHeaders);
- } else {
+ } else {
LOG.debug("No cluster name provided so using default");
}
+ // Only add username and password if password is configured
+ if (StringUtils.isNotBlank(password)) {
Review comment:
Hi @jorgelbg, thanks for pointing this out. I can change the code to follow
the same.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> indexer-elastic: add authentication options
> -------------------------------------------
>
> Key: NUTCH-2757
> URL: https://issues.apache.org/jira/browse/NUTCH-2757
> Project: Nutch
> Issue Type: Improvement
> Components: indexer, plugin
> Affects Versions: 1.16
> Reporter: Sebastian Nagel
> Assignee: Shashanka Balakuntala Srinivasa
> Priority: Major
> Fix For: 1.17
>
>
> The plugin
> [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md)
> allows to configure authentication properties (mainly user and password).
> This should be ported to the indexer-elastic plugin which is now REST-based
> (NUTCH-2739).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
