[jira] [Comment Edited] (KNOX-829) HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate

2018-01-16 Thread Rajesh Chandramohan (JIRA) 

[ 
https://issues.apache.org/jira/browse/KNOX-829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328188#comment-16328188
 ] 

Rajesh Chandramohan edited comment on KNOX-829 at 1/17/18 2:23 AM:
---

[~lmccay] ,

We can use this patch for our streaming use case right. Hope it will be merged 
with upcoming releases.

 

 


was (Author: rajeshhadoop):
[~lmccay] ,

We can use this path for our streaming use case right. Hope it will be merged 
with upcoming releases.

 

 

> HDFS commands for client: 
> append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate
> ---
>
> Key: KNOX-829
> URL: https://issues.apache.org/jira/browse/KNOX-829
> Project: Apache Knox
>  Issue Type: Improvement
>  Components: ClientDSL
>Reporter: Vincent Devillers
>Assignee: Vincent Devillers
>Priority: Minor
>  Labels: KIP-4
> Fix For: Future
>
> Attachments: KNOX-829.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KNOX-829) HDFS commands for client: append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate

2018-01-16 Thread Rajesh Chandramohan (JIRA) 

[ 
https://issues.apache.org/jira/browse/KNOX-829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16328188#comment-16328188
 ] 

Rajesh Chandramohan commented on KNOX-829:
--

[~lmccay] ,

We can use this path for our streaming use case right. Hope it will be merged 
with upcoming releases.

 

 

> HDFS commands for client: 
> append,checksum,concat,homedir,chown,chmod,touch,symlink,truncate
> ---
>
> Key: KNOX-829
> URL: https://issues.apache.org/jira/browse/KNOX-829
> Project: Apache Knox
>  Issue Type: Improvement
>  Components: ClientDSL
>Reporter: Vincent Devillers
>Assignee: Vincent Devillers
>Priority: Minor
>  Labels: KIP-4
> Fix For: Future
>
> Attachments: KNOX-829.patch
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KNOX-1147) KNOX Audit Log time not configurable or not supporting TZ

2017-12-13 Thread Rajesh Chandramohan (JIRA) 

[ 
https://issues.apache.org/jira/browse/KNOX-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16290033#comment-16290033
 ] 

Rajesh Chandramohan edited comment on KNOX-1147 at 12/13/17 10:38 PM:
--

[~lmccay]
Please Review the attached changes , Can work on enhanced version to merge with 
v15 too.


was (Author: rajeshhadoop):
Please check the attached changes.

> KNOX Audit Log time not configurable or not supporting TZ
> -
>
> Key: KNOX-1147
> URL: https://issues.apache.org/jira/browse/KNOX-1147
> Project: Apache Knox
>  Issue Type: Improvement
>Affects Versions: 0.10.0, 0.13.0, 0.14.0
>Reporter: Rajesh Chandramohan
> Fix For: 0.15.0
>
> Attachments: KNOX-1147.patch
>
>
>  The KNOX Log4J configuration for Knoxaudit log uses  
> or.hadoop.gateway.audit.log4j.layout.AuditLayout , so when the system is 
> using PST tz , The auditLog is logging with 1hr ahead of current PST time. Is 
> there a way to log audit event matching with current time in system and not 
> break with DST change?
> Even in latest version it is having hardcoded time pattern , not able to add 
> Time zone. Was thinking of adding  
> setDateFormat(DATE_FORMAT, TimeZone.getTimeZone(TZ));  But is there any other 
> better way to get around this issue?
> +
> public class AuditLayout extends DateLayout {
>   
>   private static final String DATE_FORMAT = "yy/MM/dd HH:mm:ss";
>   private static final String SEPARATOR = "|";
>   private StringBuffer sb = new StringBuffer();
> Override
> public void activateOptions() {
>   setDateFormat( DATE_FORMAT );
> }
> +
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KNOX-1147) KNOX Audit Log time not configurable or not supporting TZ

2017-12-13 Thread Rajesh Chandramohan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KNOX-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Chandramohan updated KNOX-1147:
--
Status: Patch Available  (was: Open)

Please check the attached changes.

> KNOX Audit Log time not configurable or not supporting TZ
> -
>
> Key: KNOX-1147
> URL: https://issues.apache.org/jira/browse/KNOX-1147
> Project: Apache Knox
>  Issue Type: Improvement
>Affects Versions: 0.10.0, 0.13.0, 0.14.0
>Reporter: Rajesh Chandramohan
> Fix For: 0.15.0
>
> Attachments: KNOX-1147.patch
>
>
>  The KNOX Log4J configuration for Knoxaudit log uses  
> or.hadoop.gateway.audit.log4j.layout.AuditLayout , so when the system is 
> using PST tz , The auditLog is logging with 1hr ahead of current PST time. Is 
> there a way to log audit event matching with current time in system and not 
> break with DST change?
> Even in latest version it is having hardcoded time pattern , not able to add 
> Time zone. Was thinking of adding  
> setDateFormat(DATE_FORMAT, TimeZone.getTimeZone(TZ));  But is there any other 
> better way to get around this issue?
> +
> public class AuditLayout extends DateLayout {
>   
>   private static final String DATE_FORMAT = "yy/MM/dd HH:mm:ss";
>   private static final String SEPARATOR = "|";
>   private StringBuffer sb = new StringBuffer();
> Override
> public void activateOptions() {
>   setDateFormat( DATE_FORMAT );
> }
> +
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KNOX-1147) KNOX Audit Log time not configurable or not supporting TZ

2017-12-13 Thread Rajesh Chandramohan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KNOX-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Chandramohan updated KNOX-1147:
--
Attachment: KNOX-1147.patch

Adding a patch,  which helps with this case. We can extend further by giving 
the Configurable variable for layout through log4j.properties  as like 
log4j.EnhancedPatternLayout

> KNOX Audit Log time not configurable or not supporting TZ
> -
>
> Key: KNOX-1147
> URL: https://issues.apache.org/jira/browse/KNOX-1147
> Project: Apache Knox
>  Issue Type: Improvement
>Affects Versions: 0.10.0, 0.13.0, 0.14.0
>Reporter: Rajesh Chandramohan
> Fix For: 0.15.0
>
> Attachments: KNOX-1147.patch
>
>
>  The KNOX Log4J configuration for Knoxaudit log uses  
> or.hadoop.gateway.audit.log4j.layout.AuditLayout , so when the system is 
> using PST tz , The auditLog is logging with 1hr ahead of current PST time. Is 
> there a way to log audit event matching with current time in system and not 
> break with DST change?
> Even in latest version it is having hardcoded time pattern , not able to add 
> Time zone. Was thinking of adding  
> setDateFormat(DATE_FORMAT, TimeZone.getTimeZone(TZ));  But is there any other 
> better way to get around this issue?
> +
> public class AuditLayout extends DateLayout {
>   
>   private static final String DATE_FORMAT = "yy/MM/dd HH:mm:ss";
>   private static final String SEPARATOR = "|";
>   private StringBuffer sb = new StringBuffer();
> Override
> public void activateOptions() {
>   setDateFormat( DATE_FORMAT );
> }
> +
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Issue Comment Deleted] (KNOX-1147) KNOX Audit Log time not configurable or not supporting TZ

2017-12-13 Thread Rajesh Chandramohan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KNOX-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Chandramohan updated KNOX-1147:
--
Comment: was deleted

(was: diff --git a/gateway-util-common/pom.xml b/gateway-util-common/pom.xml
index 867b58f..3be0d3a 100644
--- a/gateway-util-common/pom.xml
+++ b/gateway-util-common/pom.xml
@@ -22,7 +22,7 @@
 
 org.apache.knox
 gateway
-0.10.0
+0.10.1
 
 gateway-util-common

diff --git 
a/gateway-util-common/src/main/java/org/apache/hadoop/gateway/audit/log4j/layout/AuditLayout.java
 
b/gateway-util-common/src/main/java/org/apache/hadoop/gateway/audit/log4j/layout/AuditLayout.java
index 09f827c..ee92e0f 100644
--- 
a/gateway-util-common/src/main/java/org/apache/hadoop/gateway/audit/log4j/layout/AuditLayout.java
+++ 
b/gateway-util-common/src/main/java/org/apache/hadoop/gateway/audit/log4j/layout/AuditLayout.java
@@ -25,6 +25,8 @@ import 
org.apache.hadoop.gateway.audit.log4j.correlation.Log4jCorrelationService
 import org.apache.log4j.helpers.DateLayout;
 import org.apache.log4j.spi.LoggingEvent;

+import java.util.TimeZone;
+
 /**
  * Formats audit record to following output:
  * date time 
root_request_id|parent_request_id|request_id|channel|target_service|username|proxy_username|system_username|action|resource_type|resource_name|outcome|message
@@ -33,11 +35,17 @@ public class AuditLayout extends DateLayout {

   private static final String DATE_FORMAT = "yy/MM/dd HH:mm:ss";
   private static final String SEPARATOR = "|";
+  private static final String TZ = "PST";
   private StringBuffer sb = new StringBuffer();

-  @Override
-  public void activateOptions() {
+  /*@Override
+   public void activateOptions() {
 setDateFormat( DATE_FORMAT );
+  } */
+
+  @Override
+  public void activateOptions(){
+setDateFormat(DATE_FORMAT, TimeZone.getTimeZone(TZ));
   }

   @Override
diff --git a/pom.xml b/pom.xml
index a8e9983..efd0422 100644
--- a/pom.xml
+++ b/pom.xml
@@ -26,7 +26,7 @@
 
 org.apache.knox
 gateway
-0.10.0
+0.10.1
 pom

 gateway
)

> KNOX Audit Log time not configurable or not supporting TZ
> -
>
> Key: KNOX-1147
> URL: https://issues.apache.org/jira/browse/KNOX-1147
> Project: Apache Knox
>  Issue Type: Improvement
>Affects Versions: 0.10.0, 0.13.0, 0.14.0
>Reporter: Rajesh Chandramohan
> Fix For: 0.15.0
>
>
>  The KNOX Log4J configuration for Knoxaudit log uses  
> or.hadoop.gateway.audit.log4j.layout.AuditLayout , so when the system is 
> using PST tz , The auditLog is logging with 1hr ahead of current PST time. Is 
> there a way to log audit event matching with current time in system and not 
> break with DST change?
> Even in latest version it is having hardcoded time pattern , not able to add 
> Time zone. Was thinking of adding  
> setDateFormat(DATE_FORMAT, TimeZone.getTimeZone(TZ));  But is there any other 
> better way to get around this issue?
> +
> public class AuditLayout extends DateLayout {
>   
>   private static final String DATE_FORMAT = "yy/MM/dd HH:mm:ss";
>   private static final String SEPARATOR = "|";
>   private StringBuffer sb = new StringBuffer();
> Override
> public void activateOptions() {
>   setDateFormat( DATE_FORMAT );
> }
> +
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KNOX-1147) KNOX Audit Log time not configurable or not supporting TZ

2017-12-13 Thread Rajesh Chandramohan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KNOX-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Chandramohan updated KNOX-1147:
--
Status: Open  (was: Patch Available)

diff --git a/gateway-util-common/pom.xml b/gateway-util-common/pom.xml
index 867b58f..3be0d3a 100644
--- a/gateway-util-common/pom.xml
+++ b/gateway-util-common/pom.xml
@@ -22,7 +22,7 @@
 
 org.apache.knox
 gateway
-0.10.0
+0.10.1
 
 gateway-util-common

diff --git 
a/gateway-util-common/src/main/java/org/apache/hadoop/gateway/audit/log4j/layout/AuditLayout.java
 
b/gateway-util-common/src/main/java/org/apache/hadoop/gateway/audit/log4j/layout/AuditLayout.java
index 09f827c..ee92e0f 100644
--- 
a/gateway-util-common/src/main/java/org/apache/hadoop/gateway/audit/log4j/layout/AuditLayout.java
+++ 
b/gateway-util-common/src/main/java/org/apache/hadoop/gateway/audit/log4j/layout/AuditLayout.java
@@ -25,6 +25,8 @@ import 
org.apache.hadoop.gateway.audit.log4j.correlation.Log4jCorrelationService
 import org.apache.log4j.helpers.DateLayout;
 import org.apache.log4j.spi.LoggingEvent;

+import java.util.TimeZone;
+
 /**
  * Formats audit record to following output:
  * date time 
root_request_id|parent_request_id|request_id|channel|target_service|username|proxy_username|system_username|action|resource_type|resource_name|outcome|message
@@ -33,11 +35,17 @@ public class AuditLayout extends DateLayout {

   private static final String DATE_FORMAT = "yy/MM/dd HH:mm:ss";
   private static final String SEPARATOR = "|";
+  private static final String TZ = "PST";
   private StringBuffer sb = new StringBuffer();

-  @Override
-  public void activateOptions() {
+  /*@Override
+   public void activateOptions() {
 setDateFormat( DATE_FORMAT );
+  } */
+
+  @Override
+  public void activateOptions(){
+setDateFormat(DATE_FORMAT, TimeZone.getTimeZone(TZ));
   }

   @Override
diff --git a/pom.xml b/pom.xml
index a8e9983..efd0422 100644
--- a/pom.xml
+++ b/pom.xml
@@ -26,7 +26,7 @@
 
 org.apache.knox
 gateway
-0.10.0
+0.10.1
 pom

 gateway


> KNOX Audit Log time not configurable or not supporting TZ
> -
>
> Key: KNOX-1147
> URL: https://issues.apache.org/jira/browse/KNOX-1147
> Project: Apache Knox
>  Issue Type: Improvement
>Affects Versions: 0.10.0, 0.13.0, 0.14.0
>Reporter: Rajesh Chandramohan
> Fix For: 0.15.0
>
>
>  The KNOX Log4J configuration for Knoxaudit log uses  
> or.hadoop.gateway.audit.log4j.layout.AuditLayout , so when the system is 
> using PST tz , The auditLog is logging with 1hr ahead of current PST time. Is 
> there a way to log audit event matching with current time in system and not 
> break with DST change?
> Even in latest version it is having hardcoded time pattern , not able to add 
> Time zone. Was thinking of adding  
> setDateFormat(DATE_FORMAT, TimeZone.getTimeZone(TZ));  But is there any other 
> better way to get around this issue?
> +
> public class AuditLayout extends DateLayout {
>   
>   private static final String DATE_FORMAT = "yy/MM/dd HH:mm:ss";
>   private static final String SEPARATOR = "|";
>   private StringBuffer sb = new StringBuffer();
> Override
> public void activateOptions() {
>   setDateFormat( DATE_FORMAT );
> }
> +
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KNOX-1147) KNOX Audit Log time not configurable or not supporting TZ

2017-12-13 Thread Rajesh Chandramohan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KNOX-1147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Chandramohan updated KNOX-1147:
--
Status: Patch Available  (was: Open)

> KNOX Audit Log time not configurable or not supporting TZ
> -
>
> Key: KNOX-1147
> URL: https://issues.apache.org/jira/browse/KNOX-1147
> Project: Apache Knox
>  Issue Type: Improvement
>Affects Versions: 0.10.0, 0.13.0, 0.14.0
>Reporter: Rajesh Chandramohan
> Fix For: 0.15.0
>
>
>  The KNOX Log4J configuration for Knoxaudit log uses  
> or.hadoop.gateway.audit.log4j.layout.AuditLayout , so when the system is 
> using PST tz , The auditLog is logging with 1hr ahead of current PST time. Is 
> there a way to log audit event matching with current time in system and not 
> break with DST change?
> Even in latest version it is having hardcoded time pattern , not able to add 
> Time zone. Was thinking of adding  
> setDateFormat(DATE_FORMAT, TimeZone.getTimeZone(TZ));  But is there any other 
> better way to get around this issue?
> +
> public class AuditLayout extends DateLayout {
>   
>   private static final String DATE_FORMAT = "yy/MM/dd HH:mm:ss";
>   private static final String SEPARATOR = "|";
>   private StringBuffer sb = new StringBuffer();
> Override
> public void activateOptions() {
>   setDateFormat( DATE_FORMAT );
> }
> +
> Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Created] (KNOX-1147) KNOX Audit Log time not configurable or not supporting TZ

2017-12-12 Thread Rajesh Chandramohan (JIRA) 
Rajesh Chandramohan created KNOX-1147:
-

 Summary: KNOX Audit Log time not configurable or not supporting TZ
 Key: KNOX-1147
 URL: https://issues.apache.org/jira/browse/KNOX-1147
 Project: Apache Knox
  Issue Type: Improvement
Affects Versions: 0.10.0, 0.13.0, 0.14.0
Reporter: Rajesh Chandramohan


 The KNOX Log4J configuration for Knoxaudit log uses  
or.hadoop.gateway.audit.log4j.layout.AuditLayout , so when the system is using 
PST tz , The auditLog is logging with 1hr ahead of current PST time. Is there a 
way to log audit event matching with current time in system and not break with 
DST change?

Even in latest version it is having hardcoded time pattern , not able to add 
Time zone. Was thinking of adding  
setDateFormat(DATE_FORMAT, TimeZone.getTimeZone(TZ));  But is there any other 
better way to get around this issue?

+
public class AuditLayout extends DateLayout {
  
  private static final String DATE_FORMAT = "yy/MM/dd HH:mm:ss";
  private static final String SEPARATOR = "|";
  private StringBuffer sb = new StringBuffer();
Override
public void activateOptions() {
  setDateFormat( DATE_FORMAT );
}


+


Thanks



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Updated] (KNOX-1093) KNOX Not Handling safemode state of one of the NameNode In HA state

2017-10-31 Thread Rajesh Chandramohan (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KNOX-1093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajesh Chandramohan updated KNOX-1093:
--
Description: 
 per your code WebHdfsHaDispatch.java , When Safemode exception happened it 
calls the retryRequest() method. which also calls executeRequest() method as 
like failover request but the namenode info is not changing for the thread for 
all of its iteration until maxRetryAttempts=300 
and retrySleep=1000 ( 1 sec ) 
After Max 5 minutes , client retries should pick the right namenode atleast in 
next attempt.
 But in this case if we need to copy a set of files in stipulated time there is 
X% of connections falls into these namenode and fails. Can we handle that better

{code:java}
try {
 inboundResponse = executeOutboundRequest(outboundRequest);
 writeOutboundResponse(outboundRequest, inboundRequest, 
outboundResponse, inboundResponse);
  } catch (StandbyException e) {
 LOG.errorReceivedFromStandbyNode(e);
 failoverRequest(outboundRequest, inboundRequest, outboundResponse, 
inboundResponse, e);
  } catch (SafeModeException e) {
 LOG.errorReceivedFromSafeModeNode(e);
 retryRequest(outboundRequest, inboundRequest, outboundResponse, 
inboundResponse, e);
  } catch (IOException e) {
 LOG.errorConnectingToServer(outboundRequest.getURI().toString(), e);
 failoverRequest(outboundRequest, inboundRequest, outboundResponse, 
inboundResponse, e);
  }
   }
{code}


Need to change the logic in SafeModeexception state in  KNOX HADispatch code to 
flag the namenode which is stuck in safemode  and maintain don't try queue and 
redirect all further connection only to healthy active namenode . This way X5 
of failures we can handle. What do we think

  was:

 per your code WebHdfsHaDispatch.java , When Safemode exception happened it 
calls the retryRequest() method. which also calls executeRequest() method as 
like failover request but the namenode info is not changing for the thread for 
all of its iteration until maxRetryAttempts=300 
and retrySleep=1000 ( 1 sec ) 
After Max 5 minutes , client retries should pick the right namenode atleast in 
next attempt.
 But in this case if we need to copy a set of files in stipulated time there is 
X% os connections falls into these namenode and fails. Can we candle that better

{code:java}
try {
 inboundResponse = executeOutboundRequest(outboundRequest);
 writeOutboundResponse(outboundRequest, inboundRequest, 
outboundResponse, inboundResponse);
  } catch (StandbyException e) {
 LOG.errorReceivedFromStandbyNode(e);
 failoverRequest(outboundRequest, inboundRequest, outboundResponse, 
inboundResponse, e);
  } catch (SafeModeException e) {
 LOG.errorReceivedFromSafeModeNode(e);
 retryRequest(outboundRequest, inboundRequest, outboundResponse, 
inboundResponse, e);
  } catch (IOException e) {
 LOG.errorConnectingToServer(outboundRequest.getURI().toString(), e);
 failoverRequest(outboundRequest, inboundRequest, outboundResponse, 
inboundResponse, e);
  }
   }
{code}


Need to change the logic in SafeModeexception state in  KNOX HADispatch code to 
flag the namenode which is stuck in safemode  and maintain don't try queue and 
redirect all further connection only to healthy active namenode . This way X5 
of failures we can handle. What do we think


> KNOX Not Handling safemode state of one of the NameNode In HA state 
> 
>
> Key: KNOX-1093
> URL: https://issues.apache.org/jira/browse/KNOX-1093
> Project: Apache Knox
>  Issue Type: Bug
>  Components: Server
>Affects Versions: 0.10.0
>Reporter: Rajesh Chandramohan
>
>  per your code WebHdfsHaDispatch.java , When Safemode exception happened it 
> calls the retryRequest() method. which also calls executeRequest() method as 
> like failover request but the namenode info is not changing for the thread 
> for all of its iteration until maxRetryAttempts=300 
> and retrySleep=1000 ( 1 sec ) 
> After Max 5 minutes , client retries should pick the right namenode atleast 
> in next attempt.
>  But in this case if we need to copy a set of files in stipulated time there 
> is X% of connections falls into these namenode and fails. Can we handle that 
> better
> {code:java}
> try {
>  inboundResponse = executeOutboundRequest(outboundRequest);
>  writeOutboundResponse(outboundRequest, inboundRequest, 
> outboundResponse, inboundResponse);
>   } catch (StandbyException e) {
>  LOG.errorReceivedFromStandbyNode(e);
>  failoverRequest(outboundRequest, inboundRequest, outboundResponse, 
> inboundResponse, e);
>   } catch (SafeModeException e) {
>  LOG.errorReceivedFromSafeModeNode(e);
>

[jira] [Created] (KNOX-1093) KNOX Not Handling safemode state of one of the NameNode In HA state

2017-10-31 Thread Rajesh Chandramohan (JIRA) 
Rajesh Chandramohan created KNOX-1093:
-

 Summary: KNOX Not Handling safemode state of one of the NameNode 
In HA state 
 Key: KNOX-1093
 URL: https://issues.apache.org/jira/browse/KNOX-1093
 Project: Apache Knox
  Issue Type: Bug
  Components: Server
Affects Versions: 0.10.0
Reporter: Rajesh Chandramohan



 per your code WebHdfsHaDispatch.java , When Safemode exception happened it 
calls the retryRequest() method. which also calls executeRequest() method as 
like failover request but the namenode info is not changing for the thread for 
all of its iteration until maxRetryAttempts=300 
and retrySleep=1000 ( 1 sec ) 
After Max 5 minutes , client retries should pick the right namenode atleast in 
next attempt.
 But in this case if we need to copy a set of files in stipulated time there is 
X% os connections falls into these namenode and fails. Can we candle that better

{code:java}
try {
 inboundResponse = executeOutboundRequest(outboundRequest);
 writeOutboundResponse(outboundRequest, inboundRequest, 
outboundResponse, inboundResponse);
  } catch (StandbyException e) {
 LOG.errorReceivedFromStandbyNode(e);
 failoverRequest(outboundRequest, inboundRequest, outboundResponse, 
inboundResponse, e);
  } catch (SafeModeException e) {
 LOG.errorReceivedFromSafeModeNode(e);
 retryRequest(outboundRequest, inboundRequest, outboundResponse, 
inboundResponse, e);
  } catch (IOException e) {
 LOG.errorConnectingToServer(outboundRequest.getURI().toString(), e);
 failoverRequest(outboundRequest, inboundRequest, outboundResponse, 
inboundResponse, e);
  }
   }
{code}


Need to change the logic in SafeModeexception state in  KNOX HADispatch code to 
flag the namenode which is stuck in safemode  and maintain don't try queue and 
redirect all further connection only to healthy active namenode . This way X5 
of failures we can handle. What do we think



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (KNOX-822) In Knox can we capture Bytes transfered info in logs

2017-05-15 Thread Rajesh Chandramohan (JIRA) 

[ 
https://issues.apache.org/jira/browse/KNOX-822?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16011058#comment-16011058
 ] 

Rajesh Chandramohan commented on KNOX-822:
--

 Kevin Minder 
To dev@knox.apache.org Rajesh Chandramohan
CC u...@knox.apache.org
15/12/16 at 6:35 AM
Actually if you enable the properties shown in gateway-log4j.properties and 
restart you should get request/response sizes in a separate access log but 
unfortunately not the authenticated/mapped user.  Keep in mind that the request 
length might be based on the request Content-Length header.  The code involved 
is in gateway-server/src/main/java/org/apache/hadoop/gateway/trace and 
AccessHandler.java in particular.

#log4j.logger.org.apache.hadoop.gateway.access=TRACE,httpaccess
#log4j.additivity.org.apache.hadoop.gateway.access=false

#log4j.appender.httpaccess=org.apache.log4j.DailyRollingFileAppender
#log4j.appender.httpaccess.File=${app.log.dir}/${launcher.name}-http-access.log
#log4j.appender.httpaccess.DatePattern=.-MM-dd
#log4j.appender.httpaccess.layout=org.apache.log4j.PatternLayout
#log4j.appender.httpaccess.layout.ConversionPattern=%d{ISO8601}|%t|%m%n

> On Dec 14, 2016, at 11:15 PM, larry mccay  wrote:
>
> Hello Rajesh -
>
> I don't think that we have any way to do this today.
> There may be extension points in our metrics or in our audit logging to 
> capture payload sizes.
> I imagine that the interesting place to do this would be in the dispatch 
> classes to capture what is being sent to and received from the backend 
> services.
>
> This would require some investigation to determine whether we can gather the 
> sizes and how best to capture them.
>
> I would suggest filing an improvement JIRA and continuing the discussion 
> there.
>
> Thanks for your interest in Knox!
>
> --larry

> In Knox can we capture Bytes transfered info in logs
> 
>
> Key: KNOX-822
> URL: https://issues.apache.org/jira/browse/KNOX-822
> Project: Apache Knox
>  Issue Type: Wish
>  Components: Server
>Affects Versions: 0.6.0, 0.7.0, 0.9.0, 0.10.0
>Reporter: Rajesh Chandramohan
>
> Hi Knox-team,
> In knox can we capture the bytes transferred information along with file name 
> etc. This would help us to track how much transaction each users are doing  
> via knox , Any method or thoughts on this?
>  We make use of audit logs to capture each user how many transaction doing 
> per hr per day. But we like to capture the details of how much bytes per 
> transaction users are doing which will help us for various cases to manage 
> Hadoop and Knox better. Any thoughts on this ? 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)