[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[ https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086905#comment-17086905 ] Hudson commented on NUTCH-2757: --- SUCCESS: Integrated in Jenkins build Nutch-trunk #3670 (See [https://builds.apache.org/job/Nutch-trunk/3670/]) NUTCH-2757 : Indexer-elastic: add authentication options (shbalakuntala: [https://github.com/apache/nutch/commit/f999ca5af9ce92a0cc44ced1caa91fab0445e568]) * (edit) src/plugin/indexer-elastic/README.md * (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java * (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticConstants.java * (edit) conf/index-writers.xml.template NUTCH-2757 - Indexer-elastic: add authentication options (shbalakuntala: [https://github.com/apache/nutch/commit/e5b61de2c33421d92a5cd51acc972513ba2c2f34]) * (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java NUTCH-2757 - Indexer-elastic: add authentication options (shbalakuntala: [https://github.com/apache/nutch/commit/e471397ddb358402d3b0bf16429a50c189e4d965]) * (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java * (edit) src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticConstants.java * (edit) conf/index-writers.xml.template * (edit) src/plugin/indexer-elastic/README.md > indexer-elastic: add authentication options > --- > > Key: NUTCH-2757 > URL: https://issues.apache.org/jira/browse/NUTCH-2757 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.16 >Reporter: Sebastian Nagel >Assignee: Shashanka Balakuntala Srinivasa >Priority: Major > Fix For: 1.17 > > > The plugin > [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md) > allows to configure authentication properties (mainly user and password). > This should be ported to the indexer-elastic plugin which is now REST-based > (NUTCH-2739). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[ https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086880#comment-17086880 ] ASF GitHub Bot commented on NUTCH-2757: --- sebastian-nagel commented on pull request #508: NUTCH-2757 : Indexer-elastic: add authentication options URL: https://github.com/apache/nutch/pull/508 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > indexer-elastic: add authentication options > --- > > Key: NUTCH-2757 > URL: https://issues.apache.org/jira/browse/NUTCH-2757 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.16 >Reporter: Sebastian Nagel >Assignee: Shashanka Balakuntala Srinivasa >Priority: Major > Fix For: 1.17 > > > The plugin > [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md) > allows to configure authentication properties (mainly user and password). > This should be ported to the indexer-elastic plugin which is now REST-based > (NUTCH-2739). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[ https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084794#comment-17084794 ] ASF GitHub Bot commented on NUTCH-2757: --- balashashanka commented on pull request #508: NUTCH-2757 : Indexer-elastic: add authentication options URL: https://github.com/apache/nutch/pull/508#discussion_r409492935 ## File path: src/plugin/indexer-elastic/README.md ## @@ -32,10 +32,12 @@ Parameter Name | Description | Default value --|--|-- host | Comma-separated list of hostnames to send documents to using [TransportClient](https://static.javadoc.io/org.elasticsearch/elasticsearch/5.3.0/org/elasticsearch/client/transport/TransportClient.html). Either host and port must be defined or cluster. | Review comment: Hi, kind of missed this one. I removed this in the latest commit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > indexer-elastic: add authentication options > --- > > Key: NUTCH-2757 > URL: https://issues.apache.org/jira/browse/NUTCH-2757 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.16 >Reporter: Sebastian Nagel >Assignee: Shashanka Balakuntala Srinivasa >Priority: Major > Fix For: 1.17 > > > The plugin > [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md) > allows to configure authentication properties (mainly user and password). > This should be ported to the indexer-elastic plugin which is now REST-based > (NUTCH-2739). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[ https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084775#comment-17084775 ] ASF GitHub Bot commented on NUTCH-2757: --- jorgelbg commented on pull request #508: NUTCH-2757 : Indexer-elastic: add authentication options URL: https://github.com/apache/nutch/pull/508#discussion_r409474506 ## File path: src/plugin/indexer-elastic/README.md ## @@ -32,10 +32,12 @@ Parameter Name | Description | Default value --|--|-- host | Comma-separated list of hostnames to send documents to using [TransportClient](https://static.javadoc.io/org.elasticsearch/elasticsearch/5.3.0/org/elasticsearch/client/transport/TransportClient.html). Either host and port must be defined or cluster. | Review comment: I missed that the `or cluster` part of the sentence is now obsolete (due to the removal) but that is not a blocker from my side. If you fix it before merging it would be great. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > indexer-elastic: add authentication options > --- > > Key: NUTCH-2757 > URL: https://issues.apache.org/jira/browse/NUTCH-2757 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.16 >Reporter: Sebastian Nagel >Assignee: Shashanka Balakuntala Srinivasa >Priority: Major > Fix For: 1.17 > > > The plugin > [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md) > allows to configure authentication properties (mainly user and password). > This should be ported to the indexer-elastic plugin which is now REST-based > (NUTCH-2739). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[ https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084414#comment-17084414 ] ASF GitHub Bot commented on NUTCH-2757: --- jorgelbg commented on pull request #508: NUTCH-2757 : Indexer-elastic: add authentication options URL: https://github.com/apache/nutch/pull/508#discussion_r409165970 ## File path: conf/index-writers.xml.template ## @@ -108,8 +108,10 @@ - + > Key: NUTCH-2757 > URL: https://issues.apache.org/jira/browse/NUTCH-2757 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.16 >Reporter: Sebastian Nagel >Assignee: Shashanka Balakuntala Srinivasa >Priority: Major > Fix For: 1.17 > > > The plugin > [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md) > allows to configure authentication properties (mainly user and password). > This should be ported to the indexer-elastic plugin which is now REST-based > (NUTCH-2739). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[ https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084415#comment-17084415 ] ASF GitHub Bot commented on NUTCH-2757: --- jorgelbg commented on pull request #508: NUTCH-2757 : Indexer-elastic: add authentication options URL: https://github.com/apache/nutch/pull/508#discussion_r409165970 ## File path: conf/index-writers.xml.template ## @@ -108,8 +108,10 @@ - + > Key: NUTCH-2757 > URL: https://issues.apache.org/jira/browse/NUTCH-2757 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.16 >Reporter: Sebastian Nagel >Assignee: Shashanka Balakuntala Srinivasa >Priority: Major > Fix For: 1.17 > > > The plugin > [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md) > allows to configure authentication properties (mainly user and password). > This should be ported to the indexer-elastic plugin which is now REST-based > (NUTCH-2739). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[
https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084411#comment-17084411
]
ASF GitHub Bot commented on NUTCH-2757:
---
jorgelbg commented on pull request #508: NUTCH-2757 : Indexer-elastic: add
authentication options
URL: https://github.com/apache/nutch/pull/508#discussion_r409165441
##
File path:
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
##
@@ -114,57 +136,78 @@ public void open(IndexWriterParams parameters) throws
IOException {
DEFAULT_BULK_CLOSE_TIMEOUT);
defaultIndex = parameters.get(ElasticConstants.INDEX, DEFAULT_INDEX);
-maxBulkDocs = parameters
-.getInt(ElasticConstants.MAX_BULK_DOCS, DEFAULT_MAX_BULK_DOCS);
-maxBulkLength = parameters
-.getInt(ElasticConstants.MAX_BULK_LENGTH, DEFAULT_MAX_BULK_LENGTH);
-expBackoffMillis = parameters
-.getInt(ElasticConstants.EXPONENTIAL_BACKOFF_MILLIS,
-DEFAULT_EXP_BACKOFF_MILLIS);
-expBackoffRetries = parameters
-.getInt(ElasticConstants.EXPONENTIAL_BACKOFF_RETRIES,
-DEFAULT_EXP_BACKOFF_RETRIES);
+maxBulkDocs = parameters.getInt(ElasticConstants.MAX_BULK_DOCS,
+DEFAULT_MAX_BULK_DOCS);
+maxBulkLength = parameters.getInt(ElasticConstants.MAX_BULK_LENGTH,
+DEFAULT_MAX_BULK_LENGTH);
+expBackoffMillis = parameters.getInt(
+ElasticConstants.EXPONENTIAL_BACKOFF_MILLIS,
+DEFAULT_EXP_BACKOFF_MILLIS);
+expBackoffRetries = parameters.getInt(
+ElasticConstants.EXPONENTIAL_BACKOFF_RETRIES,
+DEFAULT_EXP_BACKOFF_RETRIES);
client = makeClient(parameters);
LOG.debug("Creating BulkProcessor with maxBulkDocs={}, maxBulkLength={}",
maxBulkDocs, maxBulkLength);
-bulkProcessor = BulkProcessor.builder(
-(request, bulkListener) ->
-client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener),
-bulkProcessorListener())
+bulkProcessor = BulkProcessor
+.builder((request, bulkListener) -> client.bulkAsync(request,
+RequestOptions.DEFAULT, bulkListener), bulkProcessorListener())
.setBulkActions(maxBulkDocs)
.setBulkSize(new ByteSizeValue(maxBulkLength, ByteSizeUnit.BYTES))
-.setConcurrentRequests(1).setBackoffPolicy(BackoffPolicy
-.exponentialBackoff(TimeValue.timeValueMillis(expBackoffMillis),
-expBackoffRetries)).build();
+.setConcurrentRequests(1)
+.setBackoffPolicy(BackoffPolicy.exponentialBackoff(
+TimeValue.timeValueMillis(expBackoffMillis), expBackoffRetries))
+.build();
}
/**
* Generates a RestHighLevelClient with the hosts given
*/
- protected RestHighLevelClient makeClient(IndexWriterParams parameters)
throws IOException {
+ protected RestHighLevelClient makeClient(IndexWriterParams parameters)
+ throws IOException {
hosts = parameters.getStrings(ElasticConstants.HOSTS);
port = parameters.getInt(ElasticConstants.PORT, DEFAULT_PORT);
+user = parameters.get(ElasticConstants.USER, DEFAULT_USER);
+password = parameters.get(ElasticConstants.PASSWORD, "");
+
+final CredentialsProvider credentialsProvider = new
BasicCredentialsProvider();
+credentialsProvider.setCredentials(AuthScope.ANY,
+new UsernamePasswordCredentials(user, password));
+
RestHighLevelClient client = null;
if (hosts != null && port > 1) {
HttpHost[] hostsList = new HttpHost[hosts.length];
int i = 0;
- for(String host: hosts) {
+ for (String host : hosts) {
hostsList[i++] = new HttpHost(host, port);
}
RestClientBuilder restClientBuilder = RestClient.builder(hostsList);
if (StringUtils.isNotBlank(cluster)) {
-Header[] defaultHeaders = new Header[]{new BasicHeader("cluster.name",
cluster)};
+Header[] defaultHeaders = new Header[] {
+new BasicHeader("cluster.name", cluster) };
restClientBuilder.setDefaultHeaders(defaultHeaders);
- } else {
+ } else {
LOG.debug("No cluster name provided so using default");
}
+ // Only add username and password if password is configured
+ if (StringUtils.isNotBlank(password)) {
Review comment:
@balashashanka sure, let's make this part the same for both indexers
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> indexer-elastic: add authentication options
> ---
>
> Key: NUTCH-2757
> URL:
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[
https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083074#comment-17083074
]
ASF GitHub Bot commented on NUTCH-2757:
---
balashashanka commented on pull request #508: NUTCH-2757 : Indexer-elastic: add
authentication options
URL: https://github.com/apache/nutch/pull/508#discussion_r408027978
##
File path:
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
##
@@ -114,57 +136,78 @@ public void open(IndexWriterParams parameters) throws
IOException {
DEFAULT_BULK_CLOSE_TIMEOUT);
defaultIndex = parameters.get(ElasticConstants.INDEX, DEFAULT_INDEX);
-maxBulkDocs = parameters
-.getInt(ElasticConstants.MAX_BULK_DOCS, DEFAULT_MAX_BULK_DOCS);
-maxBulkLength = parameters
-.getInt(ElasticConstants.MAX_BULK_LENGTH, DEFAULT_MAX_BULK_LENGTH);
-expBackoffMillis = parameters
-.getInt(ElasticConstants.EXPONENTIAL_BACKOFF_MILLIS,
-DEFAULT_EXP_BACKOFF_MILLIS);
-expBackoffRetries = parameters
-.getInt(ElasticConstants.EXPONENTIAL_BACKOFF_RETRIES,
-DEFAULT_EXP_BACKOFF_RETRIES);
+maxBulkDocs = parameters.getInt(ElasticConstants.MAX_BULK_DOCS,
+DEFAULT_MAX_BULK_DOCS);
+maxBulkLength = parameters.getInt(ElasticConstants.MAX_BULK_LENGTH,
+DEFAULT_MAX_BULK_LENGTH);
+expBackoffMillis = parameters.getInt(
+ElasticConstants.EXPONENTIAL_BACKOFF_MILLIS,
+DEFAULT_EXP_BACKOFF_MILLIS);
+expBackoffRetries = parameters.getInt(
+ElasticConstants.EXPONENTIAL_BACKOFF_RETRIES,
+DEFAULT_EXP_BACKOFF_RETRIES);
client = makeClient(parameters);
LOG.debug("Creating BulkProcessor with maxBulkDocs={}, maxBulkLength={}",
maxBulkDocs, maxBulkLength);
-bulkProcessor = BulkProcessor.builder(
-(request, bulkListener) ->
-client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener),
-bulkProcessorListener())
+bulkProcessor = BulkProcessor
+.builder((request, bulkListener) -> client.bulkAsync(request,
+RequestOptions.DEFAULT, bulkListener), bulkProcessorListener())
.setBulkActions(maxBulkDocs)
.setBulkSize(new ByteSizeValue(maxBulkLength, ByteSizeUnit.BYTES))
-.setConcurrentRequests(1).setBackoffPolicy(BackoffPolicy
-.exponentialBackoff(TimeValue.timeValueMillis(expBackoffMillis),
-expBackoffRetries)).build();
+.setConcurrentRequests(1)
+.setBackoffPolicy(BackoffPolicy.exponentialBackoff(
+TimeValue.timeValueMillis(expBackoffMillis), expBackoffRetries))
+.build();
}
/**
* Generates a RestHighLevelClient with the hosts given
*/
- protected RestHighLevelClient makeClient(IndexWriterParams parameters)
throws IOException {
+ protected RestHighLevelClient makeClient(IndexWriterParams parameters)
+ throws IOException {
hosts = parameters.getStrings(ElasticConstants.HOSTS);
port = parameters.getInt(ElasticConstants.PORT, DEFAULT_PORT);
+user = parameters.get(ElasticConstants.USER, DEFAULT_USER);
+password = parameters.get(ElasticConstants.PASSWORD, "");
+
+final CredentialsProvider credentialsProvider = new
BasicCredentialsProvider();
+credentialsProvider.setCredentials(AuthScope.ANY,
+new UsernamePasswordCredentials(user, password));
+
RestHighLevelClient client = null;
if (hosts != null && port > 1) {
HttpHost[] hostsList = new HttpHost[hosts.length];
int i = 0;
- for(String host: hosts) {
+ for (String host : hosts) {
hostsList[i++] = new HttpHost(host, port);
}
RestClientBuilder restClientBuilder = RestClient.builder(hostsList);
if (StringUtils.isNotBlank(cluster)) {
-Header[] defaultHeaders = new Header[]{new BasicHeader("cluster.name",
cluster)};
+Header[] defaultHeaders = new Header[] {
+new BasicHeader("cluster.name", cluster) };
restClientBuilder.setDefaultHeaders(defaultHeaders);
- } else {
+ } else {
LOG.debug("No cluster name provided so using default");
}
+ // Only add username and password if password is configured
+ if (StringUtils.isNotBlank(password)) {
Review comment:
Hi @jorgelbg, thanks for pointing this out. I can change the code to follow
the same.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> indexer-elastic: add authentication options
> ---
>
> Key: NUTCH-2757
> URL:
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[ https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083059#comment-17083059 ] ASF GitHub Bot commented on NUTCH-2757: --- jorgelbg commented on pull request #508: NUTCH-2757 : Indexer-elastic: add authentication options URL: https://github.com/apache/nutch/pull/508#discussion_r408016020 ## File path: conf/index-writers.xml.template ## @@ -108,8 +108,10 @@ - + > Key: NUTCH-2757 > URL: https://issues.apache.org/jira/browse/NUTCH-2757 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.16 >Reporter: Sebastian Nagel >Assignee: Shashanka Balakuntala Srinivasa >Priority: Major > Fix For: 1.17 > > > The plugin > [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md) > allows to configure authentication properties (mainly user and password). > This should be ported to the indexer-elastic plugin which is now REST-based > (NUTCH-2739). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[ https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083057#comment-17083057 ] ASF GitHub Bot commented on NUTCH-2757: --- jorgelbg commented on pull request #508: NUTCH-2757 : Indexer-elastic: add authentication options URL: https://github.com/apache/nutch/pull/508#discussion_r408008904 ## File path: conf/index-writers.xml.template ## @@ -108,8 +108,10 @@ - Review comment: Although the `cluster` property was removed here (and from the README) I still see it used in the `ElasticConstants.java` and the `ElasticIndexWriter.java` files. Do we want to get rid of this property? If so, then I would advise removing all references from the plugin. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > indexer-elastic: add authentication options > --- > > Key: NUTCH-2757 > URL: https://issues.apache.org/jira/browse/NUTCH-2757 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.16 >Reporter: Sebastian Nagel >Assignee: Shashanka Balakuntala Srinivasa >Priority: Major > Fix For: 1.17 > > > The plugin > [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md) > allows to configure authentication properties (mainly user and password). > This should be ported to the indexer-elastic plugin which is now REST-based > (NUTCH-2739). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[
https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083058#comment-17083058
]
ASF GitHub Bot commented on NUTCH-2757:
---
jorgelbg commented on pull request #508: NUTCH-2757 : Indexer-elastic: add
authentication options
URL: https://github.com/apache/nutch/pull/508#discussion_r408018772
##
File path:
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
##
@@ -114,57 +136,78 @@ public void open(IndexWriterParams parameters) throws
IOException {
DEFAULT_BULK_CLOSE_TIMEOUT);
defaultIndex = parameters.get(ElasticConstants.INDEX, DEFAULT_INDEX);
-maxBulkDocs = parameters
-.getInt(ElasticConstants.MAX_BULK_DOCS, DEFAULT_MAX_BULK_DOCS);
-maxBulkLength = parameters
-.getInt(ElasticConstants.MAX_BULK_LENGTH, DEFAULT_MAX_BULK_LENGTH);
-expBackoffMillis = parameters
-.getInt(ElasticConstants.EXPONENTIAL_BACKOFF_MILLIS,
-DEFAULT_EXP_BACKOFF_MILLIS);
-expBackoffRetries = parameters
-.getInt(ElasticConstants.EXPONENTIAL_BACKOFF_RETRIES,
-DEFAULT_EXP_BACKOFF_RETRIES);
+maxBulkDocs = parameters.getInt(ElasticConstants.MAX_BULK_DOCS,
+DEFAULT_MAX_BULK_DOCS);
+maxBulkLength = parameters.getInt(ElasticConstants.MAX_BULK_LENGTH,
+DEFAULT_MAX_BULK_LENGTH);
+expBackoffMillis = parameters.getInt(
+ElasticConstants.EXPONENTIAL_BACKOFF_MILLIS,
+DEFAULT_EXP_BACKOFF_MILLIS);
+expBackoffRetries = parameters.getInt(
+ElasticConstants.EXPONENTIAL_BACKOFF_RETRIES,
+DEFAULT_EXP_BACKOFF_RETRIES);
client = makeClient(parameters);
LOG.debug("Creating BulkProcessor with maxBulkDocs={}, maxBulkLength={}",
maxBulkDocs, maxBulkLength);
-bulkProcessor = BulkProcessor.builder(
-(request, bulkListener) ->
-client.bulkAsync(request, RequestOptions.DEFAULT, bulkListener),
-bulkProcessorListener())
+bulkProcessor = BulkProcessor
+.builder((request, bulkListener) -> client.bulkAsync(request,
+RequestOptions.DEFAULT, bulkListener), bulkProcessorListener())
.setBulkActions(maxBulkDocs)
.setBulkSize(new ByteSizeValue(maxBulkLength, ByteSizeUnit.BYTES))
-.setConcurrentRequests(1).setBackoffPolicy(BackoffPolicy
-.exponentialBackoff(TimeValue.timeValueMillis(expBackoffMillis),
-expBackoffRetries)).build();
+.setConcurrentRequests(1)
+.setBackoffPolicy(BackoffPolicy.exponentialBackoff(
+TimeValue.timeValueMillis(expBackoffMillis), expBackoffRetries))
+.build();
}
/**
* Generates a RestHighLevelClient with the hosts given
*/
- protected RestHighLevelClient makeClient(IndexWriterParams parameters)
throws IOException {
+ protected RestHighLevelClient makeClient(IndexWriterParams parameters)
+ throws IOException {
hosts = parameters.getStrings(ElasticConstants.HOSTS);
port = parameters.getInt(ElasticConstants.PORT, DEFAULT_PORT);
+user = parameters.get(ElasticConstants.USER, DEFAULT_USER);
+password = parameters.get(ElasticConstants.PASSWORD, "");
+
+final CredentialsProvider credentialsProvider = new
BasicCredentialsProvider();
+credentialsProvider.setCredentials(AuthScope.ANY,
+new UsernamePasswordCredentials(user, password));
+
RestHighLevelClient client = null;
if (hosts != null && port > 1) {
HttpHost[] hostsList = new HttpHost[hosts.length];
int i = 0;
- for(String host: hosts) {
+ for (String host : hosts) {
hostsList[i++] = new HttpHost(host, port);
}
RestClientBuilder restClientBuilder = RestClient.builder(hostsList);
if (StringUtils.isNotBlank(cluster)) {
-Header[] defaultHeaders = new Header[]{new BasicHeader("cluster.name",
cluster)};
+Header[] defaultHeaders = new Header[] {
+new BasicHeader("cluster.name", cluster) };
restClientBuilder.setDefaultHeaders(defaultHeaders);
- } else {
+ } else {
LOG.debug("No cluster name provided so using default");
}
+ // Only add username and password if password is configured
+ if (StringUtils.isNotBlank(password)) {
Review comment:
On the Solr indexer, we have a dedicated `auth` `boolean` key to verify if
the authentication must be set (see
https://github.com/apache/nutch/blob/master/src/plugin/indexer-solr/src/java/org/apache/nutch/indexwriter/solr/SolrIndexWriter.java#L97).
I'm not sure I have a *strong* preference for either option (the boolean
`auth` key vs checking for the empty `password` key) but I would err in the
side of consistency since both Indexers work in a similar way.
This is an automated message from the Apache Git
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[
https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082902#comment-17082902
]
ASF GitHub Bot commented on NUTCH-2757:
---
balashashanka commented on pull request #508: NUTCH-2757 : Indexer-elastic: add
authentication options
URL: https://github.com/apache/nutch/pull/508#discussion_r407893877
##
File path:
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
##
@@ -162,6 +191,17 @@ protected RestHighLevelClient
makeClient(IndexWriterParams parameters) throws IO
} else {
LOG.debug("No cluster name provided so using default");
}
+ //Only add username and password if password is configured
+ if(StringUtils.isNotBlank(password)) {
Review comment:
Hi @sebastian-nagel, did the formatting.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> indexer-elastic: add authentication options
> ---
>
> Key: NUTCH-2757
> URL: https://issues.apache.org/jira/browse/NUTCH-2757
> Project: Nutch
> Issue Type: Improvement
> Components: indexer, plugin
>Affects Versions: 1.16
>Reporter: Sebastian Nagel
>Assignee: Shashanka Balakuntala Srinivasa
>Priority: Major
> Fix For: 1.17
>
>
> The plugin
> [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md)
> allows to configure authentication properties (mainly user and password).
> This should be ported to the indexer-elastic plugin which is now REST-based
> (NUTCH-2739).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[
https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17082615#comment-17082615
]
ASF GitHub Bot commented on NUTCH-2757:
---
sebastian-nagel commented on pull request #508: NUTCH-2757 : Indexer-elastic:
add authentication options
URL: https://github.com/apache/nutch/pull/508#discussion_r407676074
##
File path:
src/plugin/indexer-elastic/src/java/org/apache/nutch/indexwriter/elastic/ElasticIndexWriter.java
##
@@ -162,6 +191,17 @@ protected RestHighLevelClient
makeClient(IndexWriterParams parameters) throws IO
} else {
LOG.debug("No cluster name provided so using default");
}
+ //Only add username and password if password is configured
+ if(StringUtils.isNotBlank(password)) {
Review comment:
Great, @balashashanka! Looks good and should work.
Minor point: If possible try to apply the [Nutch Eclipse Code Formatting
rules](https://github.com/apache/nutch/blob/master/eclipse-codeformat.xml) and
also remove trailing white space. Are you using Eclipse? If not let me know or
whether you need help to set it up.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> indexer-elastic: add authentication options
> ---
>
> Key: NUTCH-2757
> URL: https://issues.apache.org/jira/browse/NUTCH-2757
> Project: Nutch
> Issue Type: Improvement
> Components: indexer, plugin
>Affects Versions: 1.16
>Reporter: Sebastian Nagel
>Assignee: Shashanka Balakuntala Srinivasa
>Priority: Major
> Fix For: 1.17
>
>
> The plugin
> [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md)
> allows to configure authentication properties (mainly user and password).
> This should be ported to the indexer-elastic plugin which is now REST-based
> (NUTCH-2739).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (NUTCH-2757) indexer-elastic: add authentication options
[ https://issues.apache.org/jira/browse/NUTCH-2757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17081296#comment-17081296 ] ASF GitHub Bot commented on NUTCH-2757: --- balashashanka commented on pull request #508: NUTCH-2757 : Indexer-elastic: add authentication options URL: https://github.com/apache/nutch/pull/508 Added username and password as default credentials to the HTTP Client This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > indexer-elastic: add authentication options > --- > > Key: NUTCH-2757 > URL: https://issues.apache.org/jira/browse/NUTCH-2757 > Project: Nutch > Issue Type: Improvement > Components: indexer, plugin >Affects Versions: 1.16 >Reporter: Sebastian Nagel >Assignee: Shashanka Balakuntala Srinivasa >Priority: Major > Fix For: 1.17 > > > The plugin > [indexer-elastic-rest](https://github.com/apache/nutch/blob/ac9c435db2c9b1317fc195a762fa84d4e79fd97c/src/plugin/indexer-elastic-rest/README.md) > allows to configure authentication properties (mainly user and password). > This should be ported to the indexer-elastic plugin which is now REST-based > (NUTCH-2739). -- This message was sent by Atlassian Jira (v8.3.4#803005)
