[jira] [Updated] (KNOX-903) KnoxShell allows self signed certs to be used without any checks
[
https://issues.apache.org/jira/browse/KNOX-903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Devillers updated KNOX-903:
---
Description:
A TrustStrategy of TrustSelfSignedStrategy is being used while setting up http
clients to communicate with Knox over SSL.
In the Hadoop class it should be:
{code:java}
HostnameVerifier hostnameVerifier = NoopHostnameVerifier.INSTANCE;
TrustStrategy trustStrategy = TrustSelfSignedStrategy.INSTANCE;
if (clientContext.connection().secure()) {
hostnameVerifier = SSLConnectionSocketFactory.getDefaultHostnameVerifier();
trustStrategy = null;
} else {
{code}
instead of:
{code:java}
HostnameVerifier hostnameVerifier = NoopHostnameVerifier.INSTANCE;
TrustStrategy trustStrategy = TrustSelfSignedStrategy.INSTANCE;
if (clientContext.connection().secure()) {
hostnameVerifier = SSLConnectionSocketFactory.getDefaultHostnameVerifier();
} else {
{code}
The trustStrategy must be null in order to keep the default X509TrustManager
defined for the default ssl algorithm.
was:A TrustStrategy of TrustSelfSignedStrategy is being used while setting up
http clients to communicate with Knox over SSL.
> KnoxShell allows self signed certs to be used without any checks
>
>
> Key: KNOX-903
> URL: https://issues.apache.org/jira/browse/KNOX-903
> Project: Apache Knox
> Issue Type: Bug
> Components: KnoxShell
>Affects Versions: 0.12.0
>Reporter: Sumit Gupta
>Priority: Critical
> Fix For: 0.12.0
>
>
> A TrustStrategy of TrustSelfSignedStrategy is being used while setting up
> http clients to communicate with Knox over SSL.
> In the Hadoop class it should be:
> {code:java}
> HostnameVerifier hostnameVerifier = NoopHostnameVerifier.INSTANCE;
> TrustStrategy trustStrategy = TrustSelfSignedStrategy.INSTANCE;
> if (clientContext.connection().secure()) {
> hostnameVerifier = SSLConnectionSocketFactory.getDefaultHostnameVerifier();
> trustStrategy = null;
> } else {
> {code}
> instead of:
> {code:java}
> HostnameVerifier hostnameVerifier = NoopHostnameVerifier.INSTANCE;
> TrustStrategy trustStrategy = TrustSelfSignedStrategy.INSTANCE;
> if (clientContext.connection().secure()) {
> hostnameVerifier = SSLConnectionSocketFactory.getDefaultHostnameVerifier();
> } else {
> {code}
> The trustStrategy must be null in order to keep the default X509TrustManager
> defined for the default ssl algorithm.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (KNOX-808) Multi-put for HBase
[ https://issues.apache.org/jira/browse/KNOX-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-808: --- Attachment: KNOX-808.patch > Multi-put for HBase > --- > > Key: KNOX-808 > URL: https://issues.apache.org/jira/browse/KNOX-808 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > Labels: KIP-4 > Fix For: 0.13.0 > > Attachments: KNOX-808.patch > > > If you want to insert some data in HBase the only available method is > store(), which allow you to put only one row in a table. That means if you > want to put 10k rows, you have to do 10k HTTP calls… The HBase REST API > provides a multi-put -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KNOX-808) Multi-put for HBase
[ https://issues.apache.org/jira/browse/KNOX-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-808: --- Attachment: (was: KNOX-808.patch) > Multi-put for HBase > --- > > Key: KNOX-808 > URL: https://issues.apache.org/jira/browse/KNOX-808 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > Labels: KIP-4 > Fix For: 0.13.0 > > > If you want to insert some data in HBase the only available method is > store(), which allow you to put only one row in a table. That means if you > want to put 10k rows, you have to do 10k HTTP calls… The HBase REST API > provides a multi-put -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KNOX-808) Multi-put for HBase
[ https://issues.apache.org/jira/browse/KNOX-808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862814#comment-15862814 ] Vincent Devillers commented on KNOX-808: [~sumit.gupta] Yes, it makes sense, I've updated the patch > Multi-put for HBase > --- > > Key: KNOX-808 > URL: https://issues.apache.org/jira/browse/KNOX-808 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > Labels: KIP-4 > Fix For: 0.13.0 > > Attachments: KNOX-808.patch > > > If you want to insert some data in HBase the only available method is > store(), which allow you to put only one row in a table. That means if you > want to put 10k rows, you have to do 10k HTTP calls… The HBase REST API > provides a multi-put -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15862807#comment-15862807 ] Vincent Devillers commented on KNOX-835: Using now "gateway-test-utils" > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.13.0 > > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-835: --- Attachment: KNOX-835.patch > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.13.0 > > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-835: --- Attachment: (was: KNOX-835.patch) > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.13.0 > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KNOX-829) HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate
[ https://issues.apache.org/jira/browse/KNOX-829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15836375#comment-15836375 ] Vincent Devillers commented on KNOX-829: Indeed, it was a GET and not a POST... Symlinks are not production-ready on HDFS (https://issues.apache.org/jira/browse/HADOOP-10019), they have been disabled by default (https://issues.apache.org/jira/browse/HADOOP-10052) > HDFS commands for client: > append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate > --- > > Key: KNOX-829 > URL: https://issues.apache.org/jira/browse/KNOX-829 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-829.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15833639#comment-15833639 ] Vincent Devillers commented on KNOX-835: Indeed, updated! > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-835: --- Attachment: (was: KNOX-835.patch) > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-835: --- Attachment: KNOX-835.patch > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-805) Allow defining a custom HttpClient when creating a new session
[ https://issues.apache.org/jira/browse/KNOX-805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-805: --- Status: Patch Available (was: In Progress) > Allow defining a custom HttpClient when creating a new session > -- > > Key: KNOX-805 > URL: https://issues.apache.org/jira/browse/KNOX-805 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-805.patch > > > When creating a new session, I think it should be possible to specify a > custom CloseableHttpClient in parameter of the static method > Hadoop.login(url, user,password). This will allow to prepare a well defined > CloaseableHttpClient if you want to add headers, limit the length of the > message send, define specific strategy on dns resolution or hostname > verification, size the connection pool… -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KNOX-806) Implement Closeable for deallocable resources
[
https://issues.apache.org/jira/browse/KNOX-806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on KNOX-806 started by Vincent Devillers.
--
> Implement Closeable for deallocable resources
> -
>
> Key: KNOX-806
> URL: https://issues.apache.org/jira/browse/KNOX-806
> Project: Apache Knox
> Issue Type: Improvement
> Components: ClientDSL
>Reporter: Vincent Devillers
>Assignee: Vincent Devillers
>Priority: Minor
> Labels: KIP-4
> Fix For: 0.12.0
>
> Attachments: KNOX-806.patch
>
>
> org.apache.hadoop.gateway.shell.BasicResponse and
> org.apache.hadoop.gateway.shell.Hadoop are not java.io.Closeable, which means
> that we have to do this:
> {code:java}
> BasicResponse response = null;
> String jobId;
> try {
> response = Workflow.submit(getHadoop()).text(xmlConfiguration).now();
> jobId = JsonPath.read(response.getString(), "$.id");
> } finally {
> if(response != null) {
> response.close();
> }
> }
> {code}
> instead of this:
> {code:java}
> String jobId;
> try (BasicResponse response =
> Workflow.submit(getHadoop()).text(xmlConfiguration).now()){
> jobId = JsonPath.read(response.getString(), "$.id");
> }
> {code}
> [Source|https://blog.layer4.fr/2016/12/06/knox-production-pitfalls-and-common-mistakes/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Work started] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KNOX-835 started by Vincent Devillers. -- > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-835: --- Fix Version/s: 0.12.0 Status: Patch Available (was: In Progress) > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work stopped] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KNOX-835 stopped by Vincent Devillers. -- > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-829) HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate
[ https://issues.apache.org/jira/browse/KNOX-829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-829: --- Fix Version/s: 0.12.0 Status: Patch Available (was: In Progress) > HDFS commands for client: > append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate > --- > > Key: KNOX-829 > URL: https://issues.apache.org/jira/browse/KNOX-829 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-829.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KNOX-828) Add truncate feature for HBase table in client
[ https://issues.apache.org/jira/browse/KNOX-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KNOX-828 started by Vincent Devillers. -- > Add truncate feature for HBase table in client > --- > > Key: KNOX-828 > URL: https://issues.apache.org/jira/browse/KNOX-828 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-828.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-807) Missing resource deallocation
[
https://issues.apache.org/jira/browse/KNOX-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Devillers updated KNOX-807:
---
Status: Patch Available (was: In Progress)
> Missing resource deallocation
> -
>
> Key: KNOX-807
> URL: https://issues.apache.org/jira/browse/KNOX-807
> Project: Apache Knox
> Issue Type: Bug
> Components: ClientDSL
>Reporter: Vincent Devillers
>Assignee: Vincent Devillers
> Labels: KIP-4
> Fix For: 0.12.0
>
> Attachments: KNOX-807.patch
>
>
> h2. Release connection to pool after HTTP call
> The close() method in BasicResponse does only
> EntityUtils.consumeQuietly(response.getEntity()):
> {code}
> public void close() {
> this.consume();
> }
> public void consume() {
> if(!this.consumed) {
> EntityUtils.consumeQuietly(this.response.getEntity());
> this.consumed = true;
> }
> }
> {code}
> The underlying HTTP connection is still held. In order to ensure correct
> deallocation of OS resources we need to call CloseableHttpResponse#close()
> like this:
> {code}
> public void close() {
> try {
> this.consume();
> } finally {
> if (response instanceof CloseableHttpResponse) {
> try {
> ((CloseableHttpResponse) response).close();
> } catch (IOException e) {
> throw Throwables.propagate(e);
> }
> }
> }
> }
> {code}
> h2. Shutdown connection pool
> Also when the Hadoop session is no more used, the only method present by to
> be call is shutdown(), but it close only the ExecutorService. When a
> CloseableHttpClient instance is no longer needed, we have to shut down the
> connection manager to ensure immediate deallocation of all OS resources.
> {code}
> public void close() {
> try {
> executor.shutdownNow();
> } catch(Exception e) {
> // log something here
> }
> try {
> client.close();
> // client.close() should call getConnectionManager().shutdown();
> } catch(Exception e) {
> // log something here
> }
> }
> {code}
> [Source|https://blog.layer4.fr/2016/12/06/knox-production-pitfalls-and-common-mistakes/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KNOX-828) Add truncate feature for HBase table in client
[ https://issues.apache.org/jira/browse/KNOX-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-828: --- Fix Version/s: 0.12.0 Status: Patch Available (was: In Progress) > Add truncate feature for HBase table in client > --- > > Key: KNOX-828 > URL: https://issues.apache.org/jira/browse/KNOX-828 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-828.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KNOX-829) HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate
[ https://issues.apache.org/jira/browse/KNOX-829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KNOX-829 started by Vincent Devillers. -- > HDFS commands for client: > append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate > --- > > Key: KNOX-829 > URL: https://issues.apache.org/jira/browse/KNOX-829 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-829.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KNOX-835 started by Vincent Devillers. -- > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KNOX-807) Missing resource deallocation
[
https://issues.apache.org/jira/browse/KNOX-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Work on KNOX-807 started by Vincent Devillers.
--
> Missing resource deallocation
> -
>
> Key: KNOX-807
> URL: https://issues.apache.org/jira/browse/KNOX-807
> Project: Apache Knox
> Issue Type: Bug
> Components: ClientDSL
>Reporter: Vincent Devillers
>Assignee: Vincent Devillers
> Labels: KIP-4
> Fix For: 0.12.0
>
> Attachments: KNOX-807.patch
>
>
> h2. Release connection to pool after HTTP call
> The close() method in BasicResponse does only
> EntityUtils.consumeQuietly(response.getEntity()):
> {code}
> public void close() {
> this.consume();
> }
> public void consume() {
> if(!this.consumed) {
> EntityUtils.consumeQuietly(this.response.getEntity());
> this.consumed = true;
> }
> }
> {code}
> The underlying HTTP connection is still held. In order to ensure correct
> deallocation of OS resources we need to call CloseableHttpResponse#close()
> like this:
> {code}
> public void close() {
> try {
> this.consume();
> } finally {
> if (response instanceof CloseableHttpResponse) {
> try {
> ((CloseableHttpResponse) response).close();
> } catch (IOException e) {
> throw Throwables.propagate(e);
> }
> }
> }
> }
> {code}
> h2. Shutdown connection pool
> Also when the Hadoop session is no more used, the only method present by to
> be call is shutdown(), but it close only the ExecutorService. When a
> CloseableHttpClient instance is no longer needed, we have to shut down the
> connection manager to ensure immediate deallocation of all OS resources.
> {code}
> public void close() {
> try {
> executor.shutdownNow();
> } catch(Exception e) {
> // log something here
> }
> try {
> client.close();
> // client.close() should call getConnectionManager().shutdown();
> } catch(Exception e) {
> // log something here
> }
> }
> {code}
> [Source|https://blog.layer4.fr/2016/12/06/knox-production-pitfalls-and-common-mistakes/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KNOX-808) Multi-put for HBase
[ https://issues.apache.org/jira/browse/KNOX-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-808: --- Status: Patch Available (was: In Progress) > Multi-put for HBase > --- > > Key: KNOX-808 > URL: https://issues.apache.org/jira/browse/KNOX-808 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-808.patch > > > If you want to insert some data in HBase the only available method is > store(), which allow you to put only one row in a table. That means if you > want to put 10k rows, you have to do 10k HTTP calls… The HBase REST API > provides a multi-put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KNOX-810) Add exists in HDFS for ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KNOX-810 started by Vincent Devillers. -- > Add exists in HDFS for ClientDSL > > > Key: KNOX-810 > URL: https://issues.apache.org/jira/browse/KNOX-810 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-810.patch > > > The ClientDSL for HDFS provides various methods like "list" or "get", but the > "exists" may be very useful too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KNOX-809) Add copyFromLocal along with the put in HDFS for ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KNOX-809 started by Vincent Devillers. -- > Add copyFromLocal along with the put in HDFS for ClientDSL > -- > > Key: KNOX-809 > URL: https://issues.apache.org/jira/browse/KNOX-809 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-809.patch > > > By default, Knox has the “put” endpoint for HDFS. It copies one or multiple > sources from local file system to the destination file system. The > copyFromLocal is very similar to the put command, except that you have an > option to overwrite the destination if it already exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-809) Add copyFromLocal along with the put in HDFS for ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-809: --- Status: Patch Available (was: In Progress) > Add copyFromLocal along with the put in HDFS for ClientDSL > -- > > Key: KNOX-809 > URL: https://issues.apache.org/jira/browse/KNOX-809 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-809.patch > > > By default, Knox has the “put” endpoint for HDFS. It copies one or multiple > sources from local file system to the destination file system. The > copyFromLocal is very similar to the put command, except that you have an > option to overwrite the destination if it already exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KNOX-805) Allow defining a custom HttpClient when creating a new session
[ https://issues.apache.org/jira/browse/KNOX-805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KNOX-805 started by Vincent Devillers. -- > Allow defining a custom HttpClient when creating a new session > -- > > Key: KNOX-805 > URL: https://issues.apache.org/jira/browse/KNOX-805 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-805.patch > > > When creating a new session, I think it should be possible to specify a > custom CloseableHttpClient in parameter of the static method > Hadoop.login(url, user,password). This will allow to prepare a well defined > CloaseableHttpClient if you want to add headers, limit the length of the > message send, define specific strategy on dns resolution or hostname > verification, size the connection pool… -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-810) Add exists in HDFS for ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-810: --- Status: Patch Available (was: In Progress) > Add exists in HDFS for ClientDSL > > > Key: KNOX-810 > URL: https://issues.apache.org/jira/browse/KNOX-810 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-810.patch > > > The ClientDSL for HDFS provides various methods like "list" or "get", but the > "exists" may be very useful too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-806) Implement Closeable for deallocable resources
[
https://issues.apache.org/jira/browse/KNOX-806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Devillers updated KNOX-806:
---
Status: Patch Available (was: In Progress)
> Implement Closeable for deallocable resources
> -
>
> Key: KNOX-806
> URL: https://issues.apache.org/jira/browse/KNOX-806
> Project: Apache Knox
> Issue Type: Improvement
> Components: ClientDSL
>Reporter: Vincent Devillers
>Assignee: Vincent Devillers
>Priority: Minor
> Labels: KIP-4
> Fix For: 0.12.0
>
> Attachments: KNOX-806.patch
>
>
> org.apache.hadoop.gateway.shell.BasicResponse and
> org.apache.hadoop.gateway.shell.Hadoop are not java.io.Closeable, which means
> that we have to do this:
> {code:java}
> BasicResponse response = null;
> String jobId;
> try {
> response = Workflow.submit(getHadoop()).text(xmlConfiguration).now();
> jobId = JsonPath.read(response.getString(), "$.id");
> } finally {
> if(response != null) {
> response.close();
> }
> }
> {code}
> instead of this:
> {code:java}
> String jobId;
> try (BasicResponse response =
> Workflow.submit(getHadoop()).text(xmlConfiguration).now()){
> jobId = JsonPath.read(response.getString(), "$.id");
> }
> {code}
> [Source|https://blog.layer4.fr/2016/12/06/knox-production-pitfalls-and-common-mistakes/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KNOX-828) Add truncate feature for HBase table in client
[ https://issues.apache.org/jira/browse/KNOX-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-828: --- Component/s: ClientDSL > Add truncate feature for HBase table in client > --- > > Key: KNOX-828 > URL: https://issues.apache.org/jira/browse/KNOX-828 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Attachments: KNOX-828.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-835: --- Component/s: ClientDSL > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-835: --- Attachment: KNOX-835.patch > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Attachments: KNOX-835.patch > > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KNOX-815) Add list in Oozie for ClientDSL
[
https://issues.apache.org/jira/browse/KNOX-815?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778207#comment-15778207
]
Vincent Devillers commented on KNOX-815:
Duplicate with KNOX-835
> Add list in Oozie for ClientDSL
> ---
>
> Key: KNOX-815
> URL: https://issues.apache.org/jira/browse/KNOX-815
> Project: Apache Knox
> Issue Type: Improvement
> Components: ClientDSL
>Affects Versions: 0.7.0
>Reporter: Khanh Maudoux
>Priority: Minor
> Labels: KIP-4
> Fix For: 0.12.0
>
>
> The ClientDSL for Oozie shoud provide the list method.
> It could be easily done with something like that (in a java 8 style):
> {code:xml}
> public static class Request extends AbstractRequest {
> private List filter = new ArrayList<>();
> private String type;
> Request( Hadoop session ) {
> super( session );
> }
> public Request filter( String filter ) {
> this.filter.add( filter );
> return this;
> }
> public Request type( String type ) {
> this.type = type;
> return this;
> }
> protected Callable callable() {
> return new Callable() {
> @Override
> public Status.Response call() throws Exception {
> String comma = URLEncoder.encode(";");
> String filters = Request.this.filter.stream().map(e ->
> URLEncoder.encode(e)).collect(Collectors.joining(comma));
> URIBuilder uri = null;
> if (StringUtils.isEmpty(filters)) {
> uri = Request.this.uri(new
> String[]{Workflow.SERVICE_PATH, "/jobs/"});
> } else {
> uri = Request.this.uri(new
> String[]{Workflow.SERVICE_PATH, "/jobs/", "?filter=" + filters});
> }
> if (StringUtils.isNotEmpty(type)) {
> uri.addParameter("jobtype", type);
> }
> HttpGet request = new HttpGet(uri.build());
> return new
> Status.Response((Request.this.execute(request)));
> }
> };
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KNOX-831) Unmarshall raw Json/XML response into POJO
[ https://issues.apache.org/jira/browse/KNOX-831?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-831: --- Labels: KIP-4 (was: ) > Unmarshall raw Json/XML response into POJO > -- > > Key: KNOX-831 > URL: https://issues.apache.org/jira/browse/KNOX-831 > Project: Apache Knox > Issue Type: Improvement >Reporter: Vincent Devillers >Priority: Minor > Labels: KIP-4 > > This issue is about the parsing of the various responses from the Knox Server > into the Knox Shell client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-829) HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate
[ https://issues.apache.org/jira/browse/KNOX-829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-829: --- Labels: KIP-4 (was: ) > HDFS commands for client: > append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate > --- > > Key: KNOX-829 > URL: https://issues.apache.org/jira/browse/KNOX-829 > Project: Apache Knox > Issue Type: Improvement >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Attachments: KNOX-829.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-835) Improvements for Oozie in the ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-835: --- Labels: KIP-4 (was: ) > Improvements for Oozie in the ClientDSL > --- > > Key: KNOX-835 > URL: https://issues.apache.org/jira/browse/KNOX-835 > Project: Apache Knox > Issue Type: Improvement >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > > Implements: list, kill, change, info, definition, rerun, resume, log, start, > dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KNOX-814) Add kill in Oozie for ClientDSL
[
https://issues.apache.org/jira/browse/KNOX-814?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15778206#comment-15778206
]
Vincent Devillers commented on KNOX-814:
Duplicate with KNOX-835
> Add kill in Oozie for ClientDSL
> ---
>
> Key: KNOX-814
> URL: https://issues.apache.org/jira/browse/KNOX-814
> Project: Apache Knox
> Issue Type: Improvement
> Components: ClientDSL
>Affects Versions: 0.7.0
>Reporter: Khanh Maudoux
>Priority: Minor
> Labels: KIP-4
> Fix For: 0.12.0
>
>
> The ClientDSL for Oozie shoud provide the kill method.
> It could be easily done with something like that :
> {code:java}
> protected Callable callable() {
> return new Callable() {
> @Override
> public Kill.Response call() throws Exception {
> URIBuilder uri = uri( Workflow.SERVICE_PATH, "/job/",
> jobId );
> uri.addParameter("action", "kill");
> HttpPut request = new HttpPut( uri.build() );
> return new Kill.Response( execute( request ) );
> }
> };
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KNOX-835) Improvements for Oozie in the ClientDSL
Vincent Devillers created KNOX-835: -- Summary: Improvements for Oozie in the ClientDSL Key: KNOX-835 URL: https://issues.apache.org/jira/browse/KNOX-835 Project: Apache Knox Issue Type: Improvement Reporter: Vincent Devillers Assignee: Vincent Devillers Priority: Minor Implements: list, kill, change, info, definition, rerun, resume, log, start, dryrun -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KNOX-831) Unmarshall raw Json/XML response into POJO
Vincent Devillers created KNOX-831: -- Summary: Unmarshall raw Json/XML response into POJO Key: KNOX-831 URL: https://issues.apache.org/jira/browse/KNOX-831 Project: Apache Knox Issue Type: Improvement Reporter: Vincent Devillers Priority: Minor This issue is about the parsing of the various responses from the Knox Server into the Knox Shell client. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-807) Missing resource deallocation
[
https://issues.apache.org/jira/browse/KNOX-807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Devillers updated KNOX-807:
---
Attachment: KNOX-807.patch
> Missing resource deallocation
> -
>
> Key: KNOX-807
> URL: https://issues.apache.org/jira/browse/KNOX-807
> Project: Apache Knox
> Issue Type: Bug
>Reporter: Vincent Devillers
>Assignee: Vincent Devillers
> Labels: KIP-4
> Fix For: 0.12.0
>
> Attachments: KNOX-807.patch
>
>
> h2. Release connection to pool after HTTP call
> The close() method in BasicResponse does only
> EntityUtils.consumeQuietly(response.getEntity()):
> {code}
> public void close() {
> this.consume();
> }
> public void consume() {
> if(!this.consumed) {
> EntityUtils.consumeQuietly(this.response.getEntity());
> this.consumed = true;
> }
> }
> {code}
> The underlying HTTP connection is still held. In order to ensure correct
> deallocation of OS resources we need to call CloseableHttpResponse#close()
> like this:
> {code}
> public void close() {
> try {
> this.consume();
> } finally {
> if (response instanceof CloseableHttpResponse) {
> try {
> ((CloseableHttpResponse) response).close();
> } catch (IOException e) {
> throw Throwables.propagate(e);
> }
> }
> }
> }
> {code}
> h2. Shutdown connection pool
> Also when the Hadoop session is no more used, the only method present by to
> be call is shutdown(), but it close only the ExecutorService. When a
> CloseableHttpClient instance is no longer needed, we have to shut down the
> connection manager to ensure immediate deallocation of all OS resources.
> {code}
> public void close() {
> try {
> executor.shutdownNow();
> } catch(Exception e) {
> // log something here
> }
> try {
> client.close();
> // client.close() should call getConnectionManager().shutdown();
> } catch(Exception e) {
> // log something here
> }
> }
> {code}
> [Source|https://blog.layer4.fr/2016/12/06/knox-production-pitfalls-and-common-mistakes/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (KNOX-813) Add rename in HDFS for ClientDSL
[
https://issues.apache.org/jira/browse/KNOX-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Devillers reassigned KNOX-813:
--
Assignee: Vincent Devillers
> Add rename in HDFS for ClientDSL
>
>
> Key: KNOX-813
> URL: https://issues.apache.org/jira/browse/KNOX-813
> Project: Apache Knox
> Issue Type: Improvement
> Components: ClientDSL
>Affects Versions: 0.7.0
>Reporter: Khanh Maudoux
>Assignee: Vincent Devillers
>Priority: Trivial
> Labels: KIP-4
> Fix For: 0.12.0
>
> Attachments: KNOX-813.patch
>
>
> The ClientDSL for HDFS shoud provide the rename method.
> It could be easily done with something like that :
> {code:java}
> protected Callable callable() {
> return new Callable() {
> @Override
> public Rename.Response call() throws Exception {
> URIBuilder uri = uri( Hdfs.SERVICE_PATH, file );
> addQueryParam( uri, "op", "RENAME" );
> addQueryParam( uri, "destination", to );
> HttpPut request = new HttpPut( uri.build() );
> return new Rename.Response( execute( request ) );
> }
> };
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (KNOX-828) Add truncate feature for HBase table in client
[ https://issues.apache.org/jira/browse/KNOX-828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers reassigned KNOX-828: -- Assignee: Vincent Devillers > Add truncate feature for HBase table in client > --- > > Key: KNOX-828 > URL: https://issues.apache.org/jira/browse/KNOX-828 > Project: Apache Knox > Issue Type: Improvement >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-829) HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate
[ https://issues.apache.org/jira/browse/KNOX-829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-829: --- Attachment: KNOX-829.patch > HDFS commands for client: > append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate > --- > > Key: KNOX-829 > URL: https://issues.apache.org/jira/browse/KNOX-829 > Project: Apache Knox > Issue Type: Improvement >Reporter: Vincent Devillers >Priority: Minor > Attachments: KNOX-829.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KNOX-829) HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate
[ https://issues.apache.org/jira/browse/KNOX-829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers reassigned KNOX-829: -- Assignee: Vincent Devillers > HDFS commands for client: > append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate > --- > > Key: KNOX-829 > URL: https://issues.apache.org/jira/browse/KNOX-829 > Project: Apache Knox > Issue Type: Improvement >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Attachments: KNOX-829.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KNOX-829) HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate
Vincent Devillers created KNOX-829: -- Summary: HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate Key: KNOX-829 URL: https://issues.apache.org/jira/browse/KNOX-829 Project: Apache Knox Issue Type: Improvement Reporter: Vincent Devillers Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-813) Add rename in HDFS for ClientDSL
[
https://issues.apache.org/jira/browse/KNOX-813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Devillers updated KNOX-813:
---
Attachment: KNOX-813.patch
> Add rename in HDFS for ClientDSL
>
>
> Key: KNOX-813
> URL: https://issues.apache.org/jira/browse/KNOX-813
> Project: Apache Knox
> Issue Type: Improvement
> Components: ClientDSL
>Affects Versions: 0.7.0
>Reporter: Khanh Maudoux
>Priority: Trivial
> Labels: KIP-4
> Fix For: 0.12.0
>
> Attachments: KNOX-813.patch
>
>
> The ClientDSL for HDFS shoud provide the rename method.
> It could be easily done with something like that :
> {code:java}
> protected Callable callable() {
> return new Callable() {
> @Override
> public Rename.Response call() throws Exception {
> URIBuilder uri = uri( Hdfs.SERVICE_PATH, file );
> addQueryParam( uri, "op", "RENAME" );
> addQueryParam( uri, "destination", to );
> HttpPut request = new HttpPut( uri.build() );
> return new Rename.Response( execute( request ) );
> }
> };
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KNOX-828) Add truncate feature for HBase table in client
Vincent Devillers created KNOX-828: -- Summary: Add truncate feature for HBase table in client Key: KNOX-828 URL: https://issues.apache.org/jira/browse/KNOX-828 Project: Apache Knox Issue Type: Improvement Reporter: Vincent Devillers Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KNOX-805) Allow defining a custom HttpClient when creating a new session
[ https://issues.apache.org/jira/browse/KNOX-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15772454#comment-15772454 ] Vincent Devillers commented on KNOX-805: Ok, I've decoupled the configuration and manage different sections for socket, pooling and connection configuration. This will allow differents protocols or frameworks to be used. I've also update the class used for HttpClient and replaced the deprecated ones. > Allow defining a custom HttpClient when creating a new session > -- > > Key: KNOX-805 > URL: https://issues.apache.org/jira/browse/KNOX-805 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-805.patch > > > When creating a new session, I think it should be possible to specify a > custom CloseableHttpClient in parameter of the static method > Hadoop.login(url, user,password). This will allow to prepare a well defined > CloaseableHttpClient if you want to add headers, limit the length of the > message send, define specific strategy on dns resolution or hostname > verification, size the connection pool… -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-805) Allow defining a custom HttpClient when creating a new session
[ https://issues.apache.org/jira/browse/KNOX-805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-805: --- Attachment: KNOX-805.patch > Allow defining a custom HttpClient when creating a new session > -- > > Key: KNOX-805 > URL: https://issues.apache.org/jira/browse/KNOX-805 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-805.patch > > > When creating a new session, I think it should be possible to specify a > custom CloseableHttpClient in parameter of the static method > Hadoop.login(url, user,password). This will allow to prepare a well defined > CloaseableHttpClient if you want to add headers, limit the length of the > message send, define specific strategy on dns resolution or hostname > verification, size the connection pool… -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KNOX-809) Add copyFromLocal along with the put in HDFS for ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15745862#comment-15745862 ] Vincent Devillers commented on KNOX-809: Indeed, in WebHDFS there is only a Put command, the copyFromLocal is only present in the underlying libs. > Add copyFromLocal along with the put in HDFS for ClientDSL > -- > > Key: KNOX-809 > URL: https://issues.apache.org/jira/browse/KNOX-809 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-809.patch > > > By default, Knox has the “put” endpoint for HDFS. It copies one or multiple > sources from local file system to the destination file system. The > copyFromLocal is very similar to the put command, except that you have an > option to overwrite the destination if it already exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-809) Add copyFromLocal along with the put in HDFS for ClientDSL
[ https://issues.apache.org/jira/browse/KNOX-809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-809: --- Attachment: KNOX-809.patch > Add copyFromLocal along with the put in HDFS for ClientDSL > -- > > Key: KNOX-809 > URL: https://issues.apache.org/jira/browse/KNOX-809 > Project: Apache Knox > Issue Type: Improvement > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers >Priority: Minor > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-809.patch > > > By default, Knox has the “put” endpoint for HDFS. It copies one or multiple > sources from local file system to the destination file system. The > copyFromLocal is very similar to the put command, except that you have an > option to overwrite the destination if it already exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-808) Multi-put for HBase
[ https://issues.apache.org/jira/browse/KNOX-808?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Devillers updated KNOX-808: --- Attachment: KNOX-808.patch > Multi-put for HBase > --- > > Key: KNOX-808 > URL: https://issues.apache.org/jira/browse/KNOX-808 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > Labels: KIP-4 > Fix For: 0.12.0 > > Attachments: KNOX-808.patch > > > If you want to insert some data in HBase the only available method is > store(), which allow you to put only one row in a table. That means if you > want to put 10k rows, you have to do 10k HTTP calls… The HBase REST API > provides a multi-put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-806) Implement Closeable for deallocable resources
[
https://issues.apache.org/jira/browse/KNOX-806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Devillers updated KNOX-806:
---
Attachment: KNOX-806.patch
> Implement Closeable for deallocable resources
> -
>
> Key: KNOX-806
> URL: https://issues.apache.org/jira/browse/KNOX-806
> Project: Apache Knox
> Issue Type: Improvement
> Components: ClientDSL
>Reporter: Vincent Devillers
>Assignee: Vincent Devillers
>Priority: Minor
> Labels: KIP-4
> Fix For: 0.12.0
>
> Attachments: KNOX-806.patch
>
>
> org.apache.hadoop.gateway.shell.BasicResponse and
> org.apache.hadoop.gateway.shell.Hadoop are not java.io.Closeable, which means
> that we have to do this:
> {code:java}
> BasicResponse response = null;
> String jobId;
> try {
> response = Workflow.submit(getHadoop()).text(xmlConfiguration).now();
> jobId = JsonPath.read(response.getString(), "$.id");
> } finally {
> if(response != null) {
> response.close();
> }
> }
> {code}
> instead of this:
> {code:java}
> String jobId;
> try (BasicResponse response =
> Workflow.submit(getHadoop()).text(xmlConfiguration).now()){
> jobId = JsonPath.read(response.getString(), "$.id");
> }
> {code}
> [Source|https://blog.layer4.fr/2016/12/06/knox-production-pitfalls-and-common-mistakes/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KNOX-805) Allow defining a custom HttpClient when creating a new session
[ https://issues.apache.org/jira/browse/KNOX-805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15740200#comment-15740200 ] Vincent Devillers commented on KNOX-805: Sure, that's a good idea! Maybe instead of passing an httpclient in Hadoop.login(url, user,password), we could pass a HttpClientConfiguration holding some properties like: - timeouts (read, connect...) - connection pool (sizing, thread factory name...) - socket attributes - custom headers The method Hadoop.login(url, user,password) will call Hadoop.login(url, user,password, httpclientconfiguration) using a default httpclientconfiguration. What do you think? > Allow defining a custom HttpClient when creating a new session > -- > > Key: KNOX-805 > URL: https://issues.apache.org/jira/browse/KNOX-805 > Project: Apache Knox > Issue Type: Bug > Components: ClientDSL >Reporter: Vincent Devillers >Assignee: Vincent Devillers > > When creating a new session, I think it should be possible to specify a > custom CloseableHttpClient in parameter of the static method > Hadoop.login(url, user,password). This will allow to prepare a well defined > CloaseableHttpClient if you want to add headers, limit the length of the > message send, define specific strategy on dns resolution or hostname > verification, size the connection pool… -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KNOX-810) Add exists in HDFS for ClientDSL
Vincent Devillers created KNOX-810: -- Summary: Add exists in HDFS for ClientDSL Key: KNOX-810 URL: https://issues.apache.org/jira/browse/KNOX-810 Project: Apache Knox Issue Type: Improvement Components: ClientDSL Reporter: Vincent Devillers Priority: Minor The ClientDSL for HDFS provides various methods like "list" or "get", but the "exists" may be very useful too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KNOX-809) Add copyFromLocal along with the put in HDFS for ClientDSL
Vincent Devillers created KNOX-809: -- Summary: Add copyFromLocal along with the put in HDFS for ClientDSL Key: KNOX-809 URL: https://issues.apache.org/jira/browse/KNOX-809 Project: Apache Knox Issue Type: Improvement Components: ClientDSL Reporter: Vincent Devillers Priority: Minor By default, Knox has the “put” endpoint for HDFS. It copies one or multiple sources from local file system to the destination file system. The copyFromLocal is very similar to the put command, except that you have an option to overwrite the destination if it already exists. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KNOX-808) Multi-put for HBase
Vincent Devillers created KNOX-808: -- Summary: Multi-put for HBase Key: KNOX-808 URL: https://issues.apache.org/jira/browse/KNOX-808 Project: Apache Knox Issue Type: Bug Components: ClientDSL Reporter: Vincent Devillers If you want to insert some data in HBase the only available method is store(), which allow you to put only one row in a table. That means if you want to put 10k rows, you have to do 10k HTTP calls… The HBase REST API provides a multi-put -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KNOX-806) Implement Closeable for deallocable resources
[
https://issues.apache.org/jira/browse/KNOX-806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vincent Devillers updated KNOX-806:
---
Description:
org.apache.hadoop.gateway.shell.BasicResponse and
org.apache.hadoop.gateway.shell.Hadoop are not java.io.Closeable, which means
that we have to do this:
{code:java}
BasicResponse response = null;
String jobId;
try {
response = Workflow.submit(getHadoop()).text(xmlConfiguration).now();
jobId = JsonPath.read(response.getString(), "$.id");
} finally {
if(response != null) {
response.close();
}
}
{code}
instead of this:
{code:java}
String jobId;
try (BasicResponse response =
Workflow.submit(getHadoop()).text(xmlConfiguration).now()){
jobId = JsonPath.read(response.getString(), "$.id");
}
{code}
[Source|https://blog.layer4.fr/2016/12/06/knox-production-pitfalls-and-common-mistakes/]
was:
org.apache.hadoop.gateway.shell.BasicResponse and
org.apache.hadoop.gateway.shell.Hadoop are not java.io.Closeable, which means
that we have to do this:
{code:java}
BasicResponse response = null;
String jobId;
try {
response = Workflow.submit(getHadoop()).text(xmlConfiguration).now();
jobId = JsonPath.read(response.getString(), "$.id");
} finally {
if(response != null) {
response.close();
}
}
{code}
instead of this:
{code:java}
String jobId;
try (BasicResponse response =
Workflow.submit(getHadoop()).text(xmlConfiguration).now()){
jobId = JsonPath.read(response.getString(), "$.id");
}
{code}
> Implement Closeable for deallocable resources
> -
>
> Key: KNOX-806
> URL: https://issues.apache.org/jira/browse/KNOX-806
> Project: Apache Knox
> Issue Type: Improvement
> Components: ClientDSL
>Reporter: Vincent Devillers
>Priority: Minor
>
> org.apache.hadoop.gateway.shell.BasicResponse and
> org.apache.hadoop.gateway.shell.Hadoop are not java.io.Closeable, which means
> that we have to do this:
> {code:java}
> BasicResponse response = null;
> String jobId;
> try {
> response = Workflow.submit(getHadoop()).text(xmlConfiguration).now();
> jobId = JsonPath.read(response.getString(), "$.id");
> } finally {
> if(response != null) {
> response.close();
> }
> }
> {code}
> instead of this:
> {code:java}
> String jobId;
> try (BasicResponse response =
> Workflow.submit(getHadoop()).text(xmlConfiguration).now()){
> jobId = JsonPath.read(response.getString(), "$.id");
> }
> {code}
> [Source|https://blog.layer4.fr/2016/12/06/knox-production-pitfalls-and-common-mistakes/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KNOX-807) Missing resource deallocation
Vincent Devillers created KNOX-807:
--
Summary: Missing resource deallocation
Key: KNOX-807
URL: https://issues.apache.org/jira/browse/KNOX-807
Project: Apache Knox
Issue Type: Bug
Reporter: Vincent Devillers
h2. Release connection to pool after HTTP call
The close() method in BasicResponse does only
EntityUtils.consumeQuietly(response.getEntity()):
{code}
public void close() {
this.consume();
}
public void consume() {
if(!this.consumed) {
EntityUtils.consumeQuietly(this.response.getEntity());
this.consumed = true;
}
}
{code}
The underlying HTTP connection is still held. In order to ensure correct
deallocation of OS resources we need to call CloseableHttpResponse#close() like
this:
{code}
public void close() {
try {
this.consume();
} finally {
if (response instanceof CloseableHttpResponse) {
try {
((CloseableHttpResponse) response).close();
} catch (IOException e) {
throw Throwables.propagate(e);
}
}
}
}
{code}
h2. Shutdown connection pool
Also when the Hadoop session is no more used, the only method present by to be
call is shutdown(), but it close only the ExecutorService. When a
CloseableHttpClient instance is no longer needed, we have to shut down the
connection manager to ensure immediate deallocation of all OS resources.
{code}
public void close() {
try {
executor.shutdownNow();
} catch(Exception e) {
// log something here
}
try {
client.close();
// client.close() should call getConnectionManager().shutdown();
} catch(Exception e) {
// log something here
}
}
{code}
[Source|https://blog.layer4.fr/2016/12/06/knox-production-pitfalls-and-common-mistakes/]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KNOX-806) Implement Closeable for deallocable resources
Vincent Devillers created KNOX-806:
--
Summary: Implement Closeable for deallocable resources
Key: KNOX-806
URL: https://issues.apache.org/jira/browse/KNOX-806
Project: Apache Knox
Issue Type: Improvement
Components: ClientDSL
Reporter: Vincent Devillers
Priority: Minor
org.apache.hadoop.gateway.shell.BasicResponse and
org.apache.hadoop.gateway.shell.Hadoop are not java.io.Closeable, which means
that we have to do this:
{code:java}
BasicResponse response = null;
String jobId;
try {
response = Workflow.submit(getHadoop()).text(xmlConfiguration).now();
jobId = JsonPath.read(response.getString(), "$.id");
} finally {
if(response != null) {
response.close();
}
}
{code}
instead of this:
{code:java}
String jobId;
try (BasicResponse response =
Workflow.submit(getHadoop()).text(xmlConfiguration).now()){
jobId = JsonPath.read(response.getString(), "$.id");
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (KNOX-805) Allow defining a custom HttpClient when creating a new session
Vincent Devillers created KNOX-805: -- Summary: Allow defining a custom HttpClient when creating a new session Key: KNOX-805 URL: https://issues.apache.org/jira/browse/KNOX-805 Project: Apache Knox Issue Type: Bug Components: ClientDSL Reporter: Vincent Devillers When creating a new session, I think it should be possible to specify a custom CloseableHttpClient in parameter of the static method Hadoop.login(url, user,password). This will allow to prepare a well defined CloaseableHttpClient if you want to add headers, limit the length of the message send, define specific strategy on dns resolution or hostname verification, size the connection pool… -- This message was sent by Atlassian JIRA (v6.3.4#6332)
