[jira] [Commented] (HADOOP-15928) libhdfs logs errors when opened FS doesn't support ByteBufferReadable
[
https://issues.apache.org/jira/browse/HADOOP-15928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16687451#comment-16687451
]
Xiao Chen commented on HADOOP-15928:
Thanks for working on this Pranay. Code seems fine. I can't understand this
part of the comment - could you update it?
{code}
//..., it means that this is the test direct read
// that is called from hdfsOpenFIleImpl().
{code}
I don't have better ideas of unit testing this, and can live with the manual
test. Please move the jira to HDFS as Steve suggested, by clicking {{More}}
then {{Move}}.
> libhdfs logs errors when opened FS doesn't support ByteBufferReadable
> -
>
> Key: HADOOP-15928
> URL: https://issues.apache.org/jira/browse/HADOOP-15928
> Project: Hadoop Common
> Issue Type: Improvement
> Components: hdfs-client
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Major
> Labels: libhdfs
> Fix For: 3.0.3
>
> Attachments: HADOOP-15928.001.patch
>
>
> Problem:
>
> There is excessive error logging when a file is opened by libhdfs
> (DFSClient/HDFS) in S3 environment, this issue is caused because buffered
> read is not supported in S3 environment, HADOOP-14603 "S3A input stream to
> support ByteBufferReadable"
> The following message is printed repeatedly in the error log/ to STDERR:
> --
> UnsupportedOperationException: Byte-buffer read unsupported by input
> streamjava.lang.UnsupportedOperationException: Byte-buffer read unsupported
> by input stream
> at
> org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:150)
> Root cause
>
> After investigating the issue, it appears that the above exception is printed
> because
> when a file is opened via hdfsOpenFileImpl() calls readDirect() which is
> hitting this
> exception.
> Fix:
>
> Since the hdfs client is not initiating the byte buffered read but is
> happening in a implicit manner, we should not be generating the error log
> during open of a file.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-11391) Enabling HVE/node awareness does not rebalance replicas on data that existed prior to topology changes.
[
https://issues.apache.org/jira/browse/HADOOP-11391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673912#comment-16673912
]
Xiao Chen commented on HADOOP-11391:
+1 pending fixing checkstyle (Agreed the \{{NamenodeFsck}} one is invalid)
> Enabling HVE/node awareness does not rebalance replicas on data that existed
> prior to topology changes.
>
>
> Key: HADOOP-11391
> URL: https://issues.apache.org/jira/browse/HADOOP-11391
> Project: Hadoop Common
> Issue Type: Bug
> Environment: VMWare w/ local storage
>Reporter: ellen johansen
>Assignee: Hrishikesh Gadre
>Priority: Major
> Attachments: HADOOP-11391-001.patch, HADOOP-11391-002.patch,
> HADOOP-11391-003.patch, HADOOP-11391-004.patch, HADOOP-11391-005.patch
>
>
> Enabling HVE/node awareness does not rebalance replicas on data that existed
> prior to topology changes.
> [root@vmw-d10-001 jenkins]# more /opt/cloudera/topology.data
> 10.20.xxx.161 /rack1/nodegroup1
> 10.20.xxx.162 /rack1/nodegroup1
> 10.20.xxx.163 /rack3/nodegroup1
> 10.20.xxx.164 /rack3/nodegroup1
> 172.17.xxx.71 /rack2/nodegroup1
> 172.17.xxx.72 /rack2/nodegroup1
> before HVE:
> /user/impalauser/tpcds/store_sales
> /user/impalauser/tpcds/store_sales/store_sales.dat 1180463121 bytes, 9
> block(s): OK
> 0. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742xxx_1382
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.161:20002,
> 10.20.xxx.163:20002]
> 1. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742213_1389
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.161:20002]
> 2. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742214_1390
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 3. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742215_1391
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 4. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742216_1392
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.161:20002,
> 172.17.xxx.72:20002]
> 5. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742217_1393
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 6. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742220_1396
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.162:20002,
> 10.20.xxx.163:20002]
> 7. BP-1184748135-172.17.xxx.71-1418235396548:blk_107374_1398
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.163:20002,
> 10.20.xxx.161:20002]
> 8. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742224_1400
> len=106721297 repl=3 [10.20.xxx.164:20002, 10.20.xxx.162:20002,
> 172.17.xxx.72:20002]
> -
> Before enabling HVE:
> Status: HEALTHY
> Total size: 1648156454 B (Total open files size: 498 B)
> Total dirs: 138
> Total files: 384
> Total symlinks: 0 (Files currently being written: 6)
> Total blocks (validated):390 (avg. block size 4226042 B) (Total open
> file blocks (not validated): 6)
> Minimally replicated blocks: 390 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 1 (0.25641027 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 3
> Average block replication: 2.8564103
> Corrupt blocks: 0
> Missing replicas:5 (0.44682753 %)
> Number of data-nodes:5
> Number of racks: 1
> FSCK ended at Wed Dec 10 14:04:35 EST 2014 in 50 milliseconds
> The filesystem under path '/' is HEALTHY
> --
> after HVE (and NN restart):
> /user/impalauser/tpcds/store_sales
> /user/impalauser/tpcds/store_sales/store_sales.dat 1180463121 bytes, 9
> block(s): OK
> 0. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742xxx_1382
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.163:20002,
> 10.20.xxx.161:20002]
> 1. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742213_1389
> len=134217728 repl=3 [172.17.xxx.72:20002, 10.20.xxx.164:20002,
> 10.20.xxx.161:20002]
> 2. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742214_1390
> len=134217728 repl=3 [172.17.xxx.72:20002, 10.20.xxx.164:20002,
> 10.20.xxx.163:20002]
> 3. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742215_1391
> len=134217728 repl=3 [172.17.xxx.72:20002, 10.20.xxx.164:20002,
> 10.20.xxx.163:20002]
> 4. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742216_1392
> len=134217728 repl=3 [172.17.xxx.72:20002, 10.20.xxx.164:20002,
> 10.20.xxx.161:20002]
> 5. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742217_1393
> len=134217728 repl=3 [172.17.xxx.72:20002, 10.20.xxx.164:20002,
> 10.20.xxx.163:20002]
> 6.
[jira] [Commented] (HADOOP-11391) Enabling HVE/node awareness does not rebalance replicas on data that existed prior to topology changes.
[
https://issues.apache.org/jira/browse/HADOOP-11391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16673468#comment-16673468
]
Xiao Chen commented on HADOOP-11391:
Thanks for revving Hrishikesh! Latest patch looks pretty good to me.
Some further comments, mostly nits:
- I see {{misReplicatedBlocks}} in {{NamenodeFsck#collectBlocksSummary}} is an
ArrayList, and in BlockManager we iterate through it via index. But it is
populated by adding {{stroedBlock}} one by one. I think a LinkedList and
iterate through iterator would be more light-weight on NN GC.
- BlockManager: can we add an info log in case of {{InterruptedException}}
caught?
- NamenodeFsck: the new warn log can look slightly better if we use
parameterized logging
- Test: Can we expose the existing {{runFsck}} from {{TestFsck}} as a shared
util method (e.g. to {{DFSTestUtil}}) and reuse it?
- Test: MiniDFSCluster can use try-with-resources to auto-close.
- Nitty nit: there's still 1 mention of 'data nodes': {{Now restart the data
nodes stopped earlier}}
> Enabling HVE/node awareness does not rebalance replicas on data that existed
> prior to topology changes.
>
>
> Key: HADOOP-11391
> URL: https://issues.apache.org/jira/browse/HADOOP-11391
> Project: Hadoop Common
> Issue Type: Bug
> Environment: VMWare w/ local storage
>Reporter: ellen johansen
>Assignee: Hrishikesh Gadre
>Priority: Major
> Attachments: HADOOP-11391-001.patch, HADOOP-11391-002.patch,
> HADOOP-11391-003.patch, HADOOP-11391-004.patch
>
>
> Enabling HVE/node awareness does not rebalance replicas on data that existed
> prior to topology changes.
> [root@vmw-d10-001 jenkins]# more /opt/cloudera/topology.data
> 10.20.xxx.161 /rack1/nodegroup1
> 10.20.xxx.162 /rack1/nodegroup1
> 10.20.xxx.163 /rack3/nodegroup1
> 10.20.xxx.164 /rack3/nodegroup1
> 172.17.xxx.71 /rack2/nodegroup1
> 172.17.xxx.72 /rack2/nodegroup1
> before HVE:
> /user/impalauser/tpcds/store_sales
> /user/impalauser/tpcds/store_sales/store_sales.dat 1180463121 bytes, 9
> block(s): OK
> 0. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742xxx_1382
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.161:20002,
> 10.20.xxx.163:20002]
> 1. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742213_1389
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.161:20002]
> 2. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742214_1390
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 3. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742215_1391
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 4. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742216_1392
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.161:20002,
> 172.17.xxx.72:20002]
> 5. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742217_1393
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 6. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742220_1396
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.162:20002,
> 10.20.xxx.163:20002]
> 7. BP-1184748135-172.17.xxx.71-1418235396548:blk_107374_1398
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.163:20002,
> 10.20.xxx.161:20002]
> 8. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742224_1400
> len=106721297 repl=3 [10.20.xxx.164:20002, 10.20.xxx.162:20002,
> 172.17.xxx.72:20002]
> -
> Before enabling HVE:
> Status: HEALTHY
> Total size: 1648156454 B (Total open files size: 498 B)
> Total dirs: 138
> Total files: 384
> Total symlinks: 0 (Files currently being written: 6)
> Total blocks (validated):390 (avg. block size 4226042 B) (Total open
> file blocks (not validated): 6)
> Minimally replicated blocks: 390 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 1 (0.25641027 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 3
> Average block replication: 2.8564103
> Corrupt blocks: 0
> Missing replicas:5 (0.44682753 %)
> Number of data-nodes:5
> Number of racks: 1
> FSCK ended at Wed Dec 10 14:04:35 EST 2014 in 50 milliseconds
> The filesystem under path '/' is HEALTHY
> --
> after HVE (and NN restart):
> /user/impalauser/tpcds/store_sales
> /user/impalauser/tpcds/store_sales/store_sales.dat 1180463121 bytes, 9
> block(s): OK
> 0. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742xxx_1382
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.163:20002,
> 10.20.xxx.161:20002]
> 1. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742213_1389
>
[jira] [Commented] (HADOOP-11391) Enabling HVE/node awareness does not rebalance replicas on data that existed prior to topology changes.
[
https://issues.apache.org/jira/browse/HADOOP-11391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672207#comment-16672207
]
Xiao Chen commented on HADOOP-11391:
[~hgadre] reached out to me offline about the questions. I overlooked the rack
topology on the description that there are actually 3 racks, so means after
restart {{Mis-replicated blocks}} after restart is expected. I was mistakenly
assuming 10.20* are all on rack, and 172.17* are on another.
> Enabling HVE/node awareness does not rebalance replicas on data that existed
> prior to topology changes.
>
>
> Key: HADOOP-11391
> URL: https://issues.apache.org/jira/browse/HADOOP-11391
> Project: Hadoop Common
> Issue Type: Bug
> Environment: VMWare w/ local storage
>Reporter: ellen johansen
>Assignee: Hrishikesh Gadre
>Priority: Major
> Attachments: HADOOP-11391-001.patch, HADOOP-11391-002.patch
>
>
> Enabling HVE/node awareness does not rebalance replicas on data that existed
> prior to topology changes.
> [root@vmw-d10-001 jenkins]# more /opt/cloudera/topology.data
> 10.20.xxx.161 /rack1/nodegroup1
> 10.20.xxx.162 /rack1/nodegroup1
> 10.20.xxx.163 /rack3/nodegroup1
> 10.20.xxx.164 /rack3/nodegroup1
> 172.17.xxx.71 /rack2/nodegroup1
> 172.17.xxx.72 /rack2/nodegroup1
> before HVE:
> /user/impalauser/tpcds/store_sales
> /user/impalauser/tpcds/store_sales/store_sales.dat 1180463121 bytes, 9
> block(s): OK
> 0. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742xxx_1382
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.161:20002,
> 10.20.xxx.163:20002]
> 1. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742213_1389
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.161:20002]
> 2. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742214_1390
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 3. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742215_1391
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 4. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742216_1392
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.161:20002,
> 172.17.xxx.72:20002]
> 5. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742217_1393
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 6. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742220_1396
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.162:20002,
> 10.20.xxx.163:20002]
> 7. BP-1184748135-172.17.xxx.71-1418235396548:blk_107374_1398
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.163:20002,
> 10.20.xxx.161:20002]
> 8. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742224_1400
> len=106721297 repl=3 [10.20.xxx.164:20002, 10.20.xxx.162:20002,
> 172.17.xxx.72:20002]
> -
> Before enabling HVE:
> Status: HEALTHY
> Total size: 1648156454 B (Total open files size: 498 B)
> Total dirs: 138
> Total files: 384
> Total symlinks: 0 (Files currently being written: 6)
> Total blocks (validated):390 (avg. block size 4226042 B) (Total open
> file blocks (not validated): 6)
> Minimally replicated blocks: 390 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 1 (0.25641027 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 3
> Average block replication: 2.8564103
> Corrupt blocks: 0
> Missing replicas:5 (0.44682753 %)
> Number of data-nodes:5
> Number of racks: 1
> FSCK ended at Wed Dec 10 14:04:35 EST 2014 in 50 milliseconds
> The filesystem under path '/' is HEALTHY
> --
> after HVE (and NN restart):
> /user/impalauser/tpcds/store_sales
> /user/impalauser/tpcds/store_sales/store_sales.dat 1180463121 bytes, 9
> block(s): OK
> 0. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742xxx_1382
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.163:20002,
> 10.20.xxx.161:20002]
> 1. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742213_1389
> len=134217728 repl=3 [172.17.xxx.72:20002, 10.20.xxx.164:20002,
> 10.20.xxx.161:20002]
> 2. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742214_1390
> len=134217728 repl=3 [172.17.xxx.72:20002, 10.20.xxx.164:20002,
> 10.20.xxx.163:20002]
> 3. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742215_1391
> len=134217728 repl=3 [172.17.xxx.72:20002, 10.20.xxx.164:20002,
> 10.20.xxx.163:20002]
> 4. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742216_1392
> len=134217728 repl=3 [172.17.xxx.72:20002, 10.20.xxx.164:20002,
> 10.20.xxx.161:20002]
> 5. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742217_1393
>
[jira] [Commented] (HADOOP-11391) Enabling HVE/node awareness does not rebalance replicas on data that existed prior to topology changes.
[
https://issues.apache.org/jira/browse/HADOOP-11391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16672158#comment-16672158
]
Xiao Chen commented on HADOOP-11391:
Thanks Hrishikesh for working on this. Thanks [~ejohansen] for reporting and
Allen for commenting.
Patch makes sense to me. I have 1 question and a few review comments.
Question:
Did you try to verify if what's reported in the description is still happening
on trunk? I don't recall which specific code would trigger it, but I feel the
behavior is still the same as of now. Specifically, {{Mis-replicated blocks}}
in description is still be 0 after a restart.
Review comments:
- BlockManager: we need to break out the writelock into finer-grained
configurable batches since the input {{blocks}} could be huge. Probably ok to
reuse {{numBlocksPerIteration}}.
- BlockManager: the log about {{processMisReplicatedBlocks}} should be debug or
trace
- Would be good to have the test to verify changing from 'never been
multi-rack' to multi-rack, and then -replicate.
- Test: trivial, let's be consistent about naming in the comment. 'datanodes'
should be fine, don't need 'data nodes' .
> Enabling HVE/node awareness does not rebalance replicas on data that existed
> prior to topology changes.
>
>
> Key: HADOOP-11391
> URL: https://issues.apache.org/jira/browse/HADOOP-11391
> Project: Hadoop Common
> Issue Type: Bug
> Environment: VMWare w/ local storage
>Reporter: ellen johansen
>Assignee: Hrishikesh Gadre
>Priority: Major
> Attachments: HADOOP-11391-001.patch, HADOOP-11391-002.patch
>
>
> Enabling HVE/node awareness does not rebalance replicas on data that existed
> prior to topology changes.
> [root@vmw-d10-001 jenkins]# more /opt/cloudera/topology.data
> 10.20.xxx.161 /rack1/nodegroup1
> 10.20.xxx.162 /rack1/nodegroup1
> 10.20.xxx.163 /rack3/nodegroup1
> 10.20.xxx.164 /rack3/nodegroup1
> 172.17.xxx.71 /rack2/nodegroup1
> 172.17.xxx.72 /rack2/nodegroup1
> before HVE:
> /user/impalauser/tpcds/store_sales
> /user/impalauser/tpcds/store_sales/store_sales.dat 1180463121 bytes, 9
> block(s): OK
> 0. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742xxx_1382
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.161:20002,
> 10.20.xxx.163:20002]
> 1. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742213_1389
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.161:20002]
> 2. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742214_1390
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 3. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742215_1391
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 4. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742216_1392
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.161:20002,
> 172.17.xxx.72:20002]
> 5. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742217_1393
> len=134217728 repl=3 [10.20.xxx.164:20002, 172.17.xxx.72:20002,
> 10.20.xxx.163:20002]
> 6. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742220_1396
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.162:20002,
> 10.20.xxx.163:20002]
> 7. BP-1184748135-172.17.xxx.71-1418235396548:blk_107374_1398
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.163:20002,
> 10.20.xxx.161:20002]
> 8. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742224_1400
> len=106721297 repl=3 [10.20.xxx.164:20002, 10.20.xxx.162:20002,
> 172.17.xxx.72:20002]
> -
> Before enabling HVE:
> Status: HEALTHY
> Total size: 1648156454 B (Total open files size: 498 B)
> Total dirs: 138
> Total files: 384
> Total symlinks: 0 (Files currently being written: 6)
> Total blocks (validated):390 (avg. block size 4226042 B) (Total open
> file blocks (not validated): 6)
> Minimally replicated blocks: 390 (100.0 %)
> Over-replicated blocks: 0 (0.0 %)
> Under-replicated blocks: 1 (0.25641027 %)
> Mis-replicated blocks: 0 (0.0 %)
> Default replication factor: 3
> Average block replication: 2.8564103
> Corrupt blocks: 0
> Missing replicas:5 (0.44682753 %)
> Number of data-nodes:5
> Number of racks: 1
> FSCK ended at Wed Dec 10 14:04:35 EST 2014 in 50 milliseconds
> The filesystem under path '/' is HEALTHY
> --
> after HVE (and NN restart):
> /user/impalauser/tpcds/store_sales
> /user/impalauser/tpcds/store_sales/store_sales.dat 1180463121 bytes, 9
> block(s): OK
> 0. BP-1184748135-172.17.xxx.71-1418235396548:blk_1073742xxx_1382
> len=134217728 repl=3 [10.20.xxx.164:20002, 10.20.xxx.163:20002,
> 10.20.xxx.161:20002]
> 1.
[jira] [Updated] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
Closing the Jira now for tracking purposes. Will reopen when doing branch-2
patches (currently buried in various ec stuff..)
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.addemdum.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15873) Add JavaBeans Activation Framework API to LICENSE.txt
[
https://issues.apache.org/jira/browse/HADOOP-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15873:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
+1, committed to trunk.
Lowered the priority to major, as I believe the hbase failure should be updated
in hbase. Thanks Wei-Chiu for filing the jira and Akira for the quick turnaround
> Add JavaBeans Activation Framework API to LICENSE.txt
> -
>
> Key: HADOOP-15873
> URL: https://issues.apache.org/jira/browse/HADOOP-15873
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: Akira Ajisaka
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HADOOP-15873.01.patch, HADOOP-15873.02.patch,
> HADOOP-15873.03.patch
>
>
> When I compile HBase master trunk against Hadoop trunk
> {code}
> $ mvn package -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.0-SNAPSHOT
> -DskipTests=true
> {code}
> , it fails to compile:
> {quote}
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (check-aggregate-license) @
> hbase-shaded-client ---
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed
> with message:
> License errors detected, for more detail find ERROR in
>
> /Users/weichiu/sandbox/hbase/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE
> {quote}
> Checking the error in the file, I saw
> {quote}
> --
> This product includes JavaBeans Activation Framework API jar licensed under
> the CDDL/GPLv2+CE.
> CDDL or GPL version 2 plus the Classpath Exception
> ERROR: Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> javax.activation
> javax.activation-api
> 1.2.0
> {quote}
> It looks like the dependency was added in HADOOP-15775. [~ajisakaa]
> [~tasanuma0829] could you check? We'll need to understand the license of this
> library
> https://github.com/javaee/activation/blob/master/LICENSE.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15873) Add JavaBeans Activation Framework API to LICENSE.txt
[
https://issues.apache.org/jira/browse/HADOOP-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15873:
---
Summary: Add JavaBeans Activation Framework API to LICENSE.txt (was:
JavaBeans Activation Framework API uses CDDL/GPLv2+CE license)
> Add JavaBeans Activation Framework API to LICENSE.txt
> -
>
> Key: HADOOP-15873
> URL: https://issues.apache.org/jira/browse/HADOOP-15873
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: Akira Ajisaka
>Priority: Blocker
> Attachments: HADOOP-15873.01.patch, HADOOP-15873.02.patch,
> HADOOP-15873.03.patch
>
>
> When I compile HBase master trunk against Hadoop trunk
> {code}
> $ mvn package -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.0-SNAPSHOT
> -DskipTests=true
> {code}
> , it fails to compile:
> {quote}
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (check-aggregate-license) @
> hbase-shaded-client ---
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed
> with message:
> License errors detected, for more detail find ERROR in
>
> /Users/weichiu/sandbox/hbase/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE
> {quote}
> Checking the error in the file, I saw
> {quote}
> --
> This product includes JavaBeans Activation Framework API jar licensed under
> the CDDL/GPLv2+CE.
> CDDL or GPL version 2 plus the Classpath Exception
> ERROR: Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> javax.activation
> javax.activation-api
> 1.2.0
> {quote}
> It looks like the dependency was added in HADOOP-15775. [~ajisakaa]
> [~tasanuma0829] could you check? We'll need to understand the license of this
> library
> https://github.com/javaee/activation/blob/master/LICENSE.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15873) Add JavaBeans Activation Framework API to LICENSE.txt
[
https://issues.apache.org/jira/browse/HADOOP-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15873:
---
Priority: Major (was: Blocker)
> Add JavaBeans Activation Framework API to LICENSE.txt
> -
>
> Key: HADOOP-15873
> URL: https://issues.apache.org/jira/browse/HADOOP-15873
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: Akira Ajisaka
>Priority: Major
> Attachments: HADOOP-15873.01.patch, HADOOP-15873.02.patch,
> HADOOP-15873.03.patch
>
>
> When I compile HBase master trunk against Hadoop trunk
> {code}
> $ mvn package -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.0-SNAPSHOT
> -DskipTests=true
> {code}
> , it fails to compile:
> {quote}
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (check-aggregate-license) @
> hbase-shaded-client ---
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed
> with message:
> License errors detected, for more detail find ERROR in
>
> /Users/weichiu/sandbox/hbase/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE
> {quote}
> Checking the error in the file, I saw
> {quote}
> --
> This product includes JavaBeans Activation Framework API jar licensed under
> the CDDL/GPLv2+CE.
> CDDL or GPL version 2 plus the Classpath Exception
> ERROR: Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> javax.activation
> javax.activation-api
> 1.2.0
> {quote}
> It looks like the dependency was added in HADOOP-15775. [~ajisakaa]
> [~tasanuma0829] could you check? We'll need to understand the license of this
> library
> https://github.com/javaee/activation/blob/master/LICENSE.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15873) JavaBeans Activation Framework API uses CDDL/GPLv2+CE license
[
https://issues.apache.org/jira/browse/HADOOP-15873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660127#comment-16660127
]
Xiao Chen commented on HADOOP-15873:
Thanks for the work here folks.
IIUC dual license means either-or, not and. So in this case you can just add it
under the existing CDDL 1.1 section at
https://github.com/apache/hadoop/blob/trunk/LICENSE.txt#L1278
> JavaBeans Activation Framework API uses CDDL/GPLv2+CE license
> -
>
> Key: HADOOP-15873
> URL: https://issues.apache.org/jira/browse/HADOOP-15873
> Project: Hadoop Common
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Wei-Chiu Chuang
>Assignee: Akira Ajisaka
>Priority: Blocker
> Attachments: HADOOP-15873.01.patch, HADOOP-15873.02.patch
>
>
> When I compile HBase master trunk against Hadoop trunk
> {code}
> $ mvn package -Dhadoop.profile=3.0 -Dhadoop-three.version=3.3.0-SNAPSHOT
> -DskipTests=true
> {code}
> , it fails to compile:
> {quote}
> [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce (check-aggregate-license) @
> hbase-shaded-client ---
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.EvaluateBeanshell failed
> with message:
> License errors detected, for more detail find ERROR in
>
> /Users/weichiu/sandbox/hbase/hbase-shaded/hbase-shaded-client/target/maven-shared-archive-resources/META-INF/LICENSE
> {quote}
> Checking the error in the file, I saw
> {quote}
> --
> This product includes JavaBeans Activation Framework API jar licensed under
> the CDDL/GPLv2+CE.
> CDDL or GPL version 2 plus the Classpath Exception
> ERROR: Please check this License for acceptability here:
> https://www.apache.org/legal/resolved
> If it is okay, then update the list named 'non_aggregate_fine' in the
> LICENSE.vm file.
> If it isn't okay, then revert the change that added the dependency.
> More info on the dependency:
> javax.activation
> javax.activation-api
> 1.2.0
> {quote}
> It looks like the dependency was added in HADOOP-15775. [~ajisakaa]
> [~tasanuma0829] could you check? We'll need to understand the license of this
> library
> https://github.com/javaee/activation/blob/master/LICENSE.txt
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15414) Job submit not work well on HDFS Federation with Transparent Encryption feature
[
https://issues.apache.org/jira/browse/HADOOP-15414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16655596#comment-16655596
]
Xiao Chen commented on HADOOP-15414:
Thanks for following up [~ste...@apache.org]. I didn't test to verify my
statement, but I believe HADOOP-14445 addresses this. [~hexiaoqiao] would you
be able to verify?
> Job submit not work well on HDFS Federation with Transparent Encryption
> feature
> ---
>
> Key: HADOOP-15414
> URL: https://issues.apache.org/jira/browse/HADOOP-15414
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs
>Reporter: He Xiaoqiao
>Priority: Major
> Attachments: HADOOP-15414-trunk.001.patch,
> HADOOP-15414-trunk.002.patch
>
>
> When submit sample MapReduce job WordCount which read/write path under
> encryption zone on HDFS Federation in security mode to YARN, task throws
> exception as below:
> {code:java}
> 18/04/26 16:07:26 INFO mapreduce.Job: Task Id : attempt_JOBID_m_TASKID_0,
> Status : FAILED
> Error: java.io.IOException:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:489)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:776)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:388)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:1468)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:1538)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:306)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:300)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:300)
> at org.apache.hadoop.fs.FilterFileSystem.open(FilterFileSystem.java:161)
> at
> org.apache.hadoop.fs.viewfs.ChRootedFileSystem.open(ChRootedFileSystem.java:258)
> at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.open(ViewFileSystem.java:424)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:793)
> at
> org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:85)
> at
> org.apache.hadoop.mapred.MapTask$NewTrackingRecordReader.initialize(MapTask.java:552)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:823)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by:
> org.apache.hadoop.security.authentication.client.AuthenticationException:
> GSSException: No valid credentials provided (Mechanism level: Failed to find
> any Kerberos tgt)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.doSpnegoSequence(KerberosAuthenticator.java:332)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:205)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:128)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:215)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:483)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1690)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:478)
> ... 21 more
> Caused by: GSSException: No valid credentials provided (Mechanism level:
> Failed to find any Kerberos tgt)
> at
> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)
> at
>
[jira] [Updated] (HADOOP-11100) Support to configure ftpClient.setControlKeepAliveTimeout
[
https://issues.apache.org/jira/browse/HADOOP-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-11100:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
+1, committed to trunk.
Thanks [~adam.antal] for the fix, [~knanasi] for the review and [~proveindia]
for filing the jira.
> Support to configure ftpClient.setControlKeepAliveTimeout
> --
>
> Key: HADOOP-11100
> URL: https://issues.apache.org/jira/browse/HADOOP-11100
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.3.0
>Reporter: Krishnamoorthy Dharmalingam
>Assignee: Adam Antal
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: HADOOP-11100.002.patch, HADOOP-11100.003.patch,
> HADOOP-11100.004.patch, HDFS-11000.001.patch
>
>
> In FTPFilesystem or Configuration, timeout is not possible to configure.
> It is very straight forward to configure, in FTPFilesystem.connect() method.
> ftpClient.setControlKeepAliveTimeout
> Like
> private FTPClient connect() throws IOException {
> ...
> String timeout = conf.get("fs.ftp.timeout." + host);
> ...
> ftpClient.setControlKeepAliveTimeout(new Integer(300));
>
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-11100) Support to configure ftpClient.setControlKeepAliveTimeout
[
https://issues.apache.org/jira/browse/HADOOP-11100?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-11100:
---
Summary: Support to configure ftpClient.setControlKeepAliveTimeout (was:
Support to configure ftpClient.setControlKeepAliveTimeout )
> Support to configure ftpClient.setControlKeepAliveTimeout
> --
>
> Key: HADOOP-11100
> URL: https://issues.apache.org/jira/browse/HADOOP-11100
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs
>Affects Versions: 2.3.0
>Reporter: Krishnamoorthy Dharmalingam
>Assignee: Adam Antal
>Priority: Minor
> Attachments: HADOOP-11100.002.patch, HADOOP-11100.003.patch,
> HADOOP-11100.004.patch, HDFS-11000.001.patch
>
>
> In FTPFilesystem or Configuration, timeout is not possible to configure.
> It is very straight forward to configure, in FTPFilesystem.connect() method.
> ftpClient.setControlKeepAliveTimeout
> Like
> private FTPClient connect() throws IOException {
> ...
> String timeout = conf.get("fs.ftp.timeout." + host);
> ...
> ftpClient.setControlKeepAliveTimeout(new Integer(300));
>
> }
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15861) Move DelegationTokenIssuer to the right path
[ https://issues.apache.org/jira/browse/HADOOP-15861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16653841#comment-16653841 ] Xiao Chen commented on HADOOP-15861: Thanks Wei-Chiu and Steve! FWIW I think it's also fair to move this back if there is release scheduling conflicts (e.g. 3.2.1). This shouldn't be rushed for any releases. It was an overlook from me on trunk reviews, which Wei-Chiu nicely caught during backport reviews. Appreciate Steve's commit and comment. :) > Move DelegationTokenIssuer to the right path > > > Key: HADOOP-15861 > URL: https://issues.apache.org/jira/browse/HADOOP-15861 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.2.0, 3.0.4, 3.1.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15861.001.patch > > > The addendum path of HADOOP-14445 updated the package name of > DelegationTokenIssuer, but it didn't update the patch. > It is currently under > hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/org/apache/hadoop/security/token/DelegationTokenIssuer.java > We should move it under > hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/DelegationTokenIssuer.java > File this as a separate jira instead of muddling through HADOOP-14445. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15861) Move DelegationTokenIssuer to the right path
[ https://issues.apache.org/jira/browse/HADOOP-15861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652973#comment-16652973 ] Xiao Chen commented on HADOOP-15861: Good catch. +1 pending > Move DelegationTokenIssuer to the right path > > > Key: HADOOP-15861 > URL: https://issues.apache.org/jira/browse/HADOOP-15861 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 3.2.0, 3.0.4, 3.1.2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Attachments: HADOOP-15861.001.patch > > > The addendum path of HADOOP-14445 updated the package name of > DelegationTokenIssuer, but it didn't update the patch. > It is currently under > hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/org/apache/hadoop/security/token/DelegationTokenIssuer.java > We should move it under > hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/DelegationTokenIssuer.java > File this as a separate jira instead of muddling through HADOOP-14445. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651073#comment-16651073
]
Xiao Chen commented on HADOOP-14445:
Forgot to mention: addendum patch committed to relevant branches (trunk,
branch-3.[0-2]). Thanks again Daryn.
Branch-2 is still good to have, but I fear I'll be preempted at least for this
week.
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.addemdum.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649691#comment-16649691
]
Xiao Chen edited comment on HADOOP-14445 at 10/15/18 3:45 AM:
--
[~daryn] Do you mind another review? Sadly this needs an addendum for 2 things:
* {{DelegationTokenIssuer}} class was recursively
'org/apache/hadoop/security/token' package twice... sorry didn't catch this
during review
* It caused 2 test failures in TestEncryptionZones. Pre-commit smartly skipped
hadoop-hdfs (only ran hadoop-hdfs-client and hadoop-common), and it's caught
when I try to backport to CDH where a full unit test was carried out. Out of
the 2 failures, {{testDelegationToken}} needs to update the way it's mocked,
and {{addMockKmsToken}} (another test method) caused mockito to give up,
refusing to call the method on interface...
(For thoroughness, internal pre-commit also complained about API compat, saying
{{addDelegationTokens}} is removed from FileSystem and DistributedFileSystem;
it also noted the same method is added to DelegationTokenIssuer, but not able
to use the latter as a clue to cross off the former. So this part is clearly to
be overruled)
was (Author: xiaochen):
[~daryn] sadly this needs an addendum for 2 things:
* {{DelegationTokenIssuer}} class was recursively
'org/apache/hadoop/security/token' package twice... sorry didn't catch this
during review
* It caused 2 test failures in TestEncryptionZones. Pre-commit smartly skipped
hadoop-hdfs (only ran hadoop-hdfs-client and hadoop-common), and it's caught
when I try to backport to CDH where a full unit test was carried out. Out of
the 2 failures, {{testDelegationToken}} needs to update the way it's mocked,
and {{addMockKmsToken}} (another test method) caused mockito to give up,
refusing to call the method on interface...
(For thoroughness, internal pre-commit also complained about API compat, saying
{{addDelegationTokens}} is removed from FileSystem and DistributedFileSystem;
it also noted the same method is added to DelegationTokenIssuer, but not able
to use the latter as a clue to cross off the former. So this part is clearly to
be overruled)
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.addemdum.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
>
[jira] [Commented] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649691#comment-16649691
]
Xiao Chen commented on HADOOP-14445:
[~daryn] sadly this needs an addendum for 2 things:
* {{DelegationTokenIssuer}} class was recursively
'org/apache/hadoop/security/token' package twice... sorry didn't catch this
during review
* It caused 2 test failures in TestEncryptionZones. Pre-commit smartly skipped
hadoop-hdfs (only ran hadoop-hdfs-client and hadoop-common), and it's caught
when I try to backport to CDH where a full unit test was carried out. Out of
the 2 failures, {{testDelegationToken}} needs to update the way it's mocked,
and {{addMockKmsToken}} (another test method) caused mockito to give up,
refusing to call the method on interface...
(For thoroughness, internal pre-commit also complained about API compat, saying
{{addDelegationTokens}} is removed from FileSystem and DistributedFileSystem;
it also noted the same method is added to DelegationTokenIssuer, but not able
to use the latter as a clue to cross off the former. So this part is clearly to
be overruled)
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.addemdum.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.addemdum.patch
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.addemdum.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15849) Upgrade netty version to 3.10.6
[ https://issues.apache.org/jira/browse/HADOOP-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15849: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) Committed to trunk. Thank you, [~arpitagarwal]. > Upgrade netty version to 3.10.6 > > > Key: HADOOP-15849 > URL: https://issues.apache.org/jira/browse/HADOOP-15849 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Fix For: 3.3.0 > > Attachments: HADOOP-15849.01.patch > > > We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 > release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15849) Upgrade netty version to 3.10.6
[ https://issues.apache.org/jira/browse/HADOOP-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15849: --- Summary: Upgrade netty version to 3.10.6 (was: Upgrade netty version) > Upgrade netty version to 3.10.6 > > > Key: HADOOP-15849 > URL: https://issues.apache.org/jira/browse/HADOOP-15849 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15849.01.patch > > > We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 > release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Fix Version/s: 3.1.2
3.0.4
3.2.0
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Fix For: 3.2.0, 3.0.4, 3.1.2
>
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648326#comment-16648326
]
Xiao Chen commented on HADOOP-14445:
Pushed this to branch-3.2 (clean backport), branch-3.1 (minor conflict), and
branch-3.0 (patched above).
Will work on the branch-2 in the recent future...
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15849) Upgrade netty version
[ https://issues.apache.org/jira/browse/HADOOP-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15849: --- Attachment: HADOOP-15849.01.patch > Upgrade netty version > - > > Key: HADOOP-15849 > URL: https://issues.apache.org/jira/browse/HADOOP-15849 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Priority: Major > Attachments: HADOOP-15849.01.patch > > > We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 > release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-15849) Upgrade netty version
[ https://issues.apache.org/jira/browse/HADOOP-15849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen reassigned HADOOP-15849: -- Assignee: Xiao Chen > Upgrade netty version > - > > Key: HADOOP-15849 > URL: https://issues.apache.org/jira/browse/HADOOP-15849 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Xiao Chen >Assignee: Xiao Chen >Priority: Major > Attachments: HADOOP-15849.01.patch > > > We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 > release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Created] (HADOOP-15849) Upgrade netty version
Xiao Chen created HADOOP-15849: -- Summary: Upgrade netty version Key: HADOOP-15849 URL: https://issues.apache.org/jira/browse/HADOOP-15849 Project: Hadoop Common Issue Type: Improvement Reporter: Xiao Chen We're currently at 3.10.5. It'd be good to upgrade to the latest 3.10.6 release. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Release Note:
This patch improves the KMS delegation token issuing and authentication logic,
to enable tokens to authenticate with a set of KMS servers. The change is
backport compatible, in that it keeps the existing authentication logic as a
fall back.
Historically, KMS delegation tokens have ip:port as service, making KMS clients
only able to use the token to authenticate with the KMS server specified as
ip:port, even though the token is shared among all KMS servers at server-side.
After this patch, newly created tokens will have the KMS URL as service.
A `DelegationTokenIssuer` interface is introduced for token creation.
was:
This patch improves the KMS delegation token issuing and authentication logic,
to enable tokens to authenticate with a set of KMS servers. The change is
backport compatible, in that it keeps the existing authentication logic as a
fall back.
Historically, KMS delegation tokens have ip:port as service, making KMS clients
only able to use the token to authenticate with the KMS server specified as
ip:port, even though the token is shared among all KMS servers at server-side.
After this patch, newly created tokens will have the KMS URL as service.
A DelegationTokenIssuer interface is introduced for token creation.
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Release Note:
This patch improves the KMS delegation token issuing and authentication logic,
to enable tokens to authenticate with a set of KMS servers. The change is
backport compatible, in that it keeps the existing authentication logic as a
fall back.
Historically, KMS delegation tokens have ip:port as service, making KMS clients
only able to use the token to authenticate with the KMS server specified as
ip:port, even though the token is shared among all KMS servers at server-side.
After this patch, newly created tokens will have the KMS URL as service.
A DelegationTokenIssuer interface is introduced for token creation.
was:
A new configuration, `hadoop.security.kms.client.token.use.uri.format`, is
introduced in the KMS clients to control the service field of the delegation
tokens fetched from the KMS. Historically KMS delegation tokens have ip:port as
service, making KMS clients only able to use the token to authenticate with 1
KMS server, even though the token is shared among all KMS servers at
server-side. The default value of this configuration is false, to be compatible
with existing behavior.
When the configuration is set to true, KMS delegation token will use uri as its
service. This way, the clients can use it to authenticate with all KMS servers.
Note that this should only be set to true if ALL clients and renewers are
running software that contains HADOOP-14445. Clients running on software
without HADOOP-14445 will fail to authenticate if the token is in uri format.
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.branch-3.0.001.patch
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.branch-3.0.001.patch, HADOOP-14445.compat.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648100#comment-16648100
]
Xiao Chen commented on HADOOP-14445:
Thanks Daryn for the elaboration, good catch.
I have committed this to trunk. Will update release notes and attach backport
patches for pre-commit. Also put Rushabh's name in it since he worked on the
earlier patch, and some tests were written by him. :)
[~daryn]: are you interested in having this in branch-2.8? We'd need to modify
the DelegationTokenIssuer to compile with java7. I'm happy to do it, but would
like to check it's what you want (and produce less conflict for Yahoo's
internal version, if there's one).
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Use DelegationTokenIssuer to create KMS delegation tokens that can authenticate to all KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Summary: Use DelegationTokenIssuer to create KMS delegation tokens that can
authenticate to all KMS instances (was: Delegation tokens are not shared
between KMS instances)
> Use DelegationTokenIssuer to create KMS delegation tokens that can
> authenticate to all KMS instances
>
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647304#comment-16647304
]
Xiao Chen commented on HADOOP-14445:
[^HADOOP-14445.20.patch]
Thanks [~daryn] for the additional review.
According to [slf4j doc|https://www.slf4j.org/faq.html#logging_performance],
when the log is disabled by level, "The one and two argument variants" of the
logging statement doesn't incur the object construction cost. But to make the
+1 valid, I have gone ahead and wrapped it. :)
Logging also changed, tuned down the extras to debug and left only 1 token
creation at info level, to be inline with DFSClient.
Will let pre-commit kick-in and commit on tomorrow if no more comments.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.20.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch, HADOOP-14445.20.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15676) Cleanup TestSSLHttpServer
[ https://issues.apache.org/jira/browse/HADOOP-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15676: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) Committed to trunk. Thank you for the contribution [~snemeth], nice clean up here. :) > Cleanup TestSSLHttpServer > - > > Key: HADOOP-15676 > URL: https://issues.apache.org/jira/browse/HADOOP-15676 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 2.6.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-15676.001.patch, HADOOP-15676.002.patch, > HADOOP-15676.003.patch, HADOOP-15676.004.patch, HADOOP-15676.005.patch > > > This issue will fix: > * Several typos in this class > * Code is not very well readable in some of the places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15676) Cleanup TestSSLHttpServer
[ https://issues.apache.org/jira/browse/HADOOP-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647097#comment-16647097 ] Xiao Chen commented on HADOOP-15676: Thanks for revving [~snemeth]! +1, committing this > Cleanup TestSSLHttpServer > - > > Key: HADOOP-15676 > URL: https://issues.apache.org/jira/browse/HADOOP-15676 > Project: Hadoop Common > Issue Type: Improvement > Components: common >Affects Versions: 2.6.0 >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: HADOOP-15676.001.patch, HADOOP-15676.002.patch, > HADOOP-15676.003.patch, HADOOP-15676.004.patch, HADOOP-15676.005.patch > > > This issue will fix: > * Several typos in this class > * Code is not very well readable in some of the places. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15717) TGT renewal thread does not log IOException
[
https://issues.apache.org/jira/browse/HADOOP-15717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647068#comment-16647068
]
Xiao Chen commented on HADOOP-15717:
Thanks for the discussion [~rkanter] and [~snemeth].
https://www.slf4j.org/apidocs/org/slf4j/Logger.html has a dedicated section for
this actually. :)
{quote}
Be sure to read the FAQ entry relating to parameterized logging. Note that
logging statements can be parameterized in presence of an exception/throwable.
{quote}
which links to https://www.slf4j.org/faq.html#paramException .
I also confirmed with {{TestUGI#testKerberosTicketIsDestroyedChecked}} that the
behavior is the same.
But as the slf4j page explains, this behavior is post slf4j 1.6.0, I think it's
probably a good idea to commit the patch as-is, so we don't change behavior
based on slf4j's version. +1 on patch 2 from me.
> TGT renewal thread does not log IOException
> ---
>
> Key: HADOOP-15717
> URL: https://issues.apache.org/jira/browse/HADOOP-15717
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: HADOOP-15717.001.patch, HADOOP-15717.002.patch
>
>
> I came across a case where tgt.getEndTime() was returned null and it resulted
> in an NPE, this observation was popped out of a test suite execution on a
> cluster. The reason for logging the {{IOException}} is that it helps to
> troubleshoot what caused the exception, as it can come from two different
> calls from the try-catch.
> I can see that [~gabor.bota] handled this with HADOOP-15593, but apart from
> logging the fact that the ticket's {{endDate}} was null, we have not logged
> the exception at all.
> With the current code, the exception is swallowed and the thread terminates
> in case the ticket's {{endDate}} is null.
> As this can happen with OpenJDK for example, it is required to print the
> exception (stack trace, message) to the log.
> The code should be updated here:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L918
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15717) TGT renewal thread does not log IOException
[
https://issues.apache.org/jira/browse/HADOOP-15717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645635#comment-16645635
]
Xiao Chen edited comment on HADOOP-15717 at 10/10/18 10:34 PM:
---
Thanks [~snemeth] for the new rev, and thanks [~rkanter] for reading my mind. :)
The logging _can_ be done with slf4j parameterized logging. See
https://www.slf4j.org/apidocs/org/slf4j/Logger.html
For example, the following will log ie into the message with its stacktrace.
{noformat}
LOG.error("TGT is destroyed. Aborting renew thread for {}.",
getUserName(), ie);
{noformat}
was (Author: xiaochen):
Thanks [~snemeth] for the new rev.
The logging _can_ be done with slf4j parameterized logging. See
https://www.slf4j.org/apidocs/org/slf4j/Logger.html
For example, the following will log ie into the message with its stacktrace.
{noformat}
LOG.error("TGT is destroyed. Aborting renew thread for {}.",
getUserName(), ie);
{noformat}
> TGT renewal thread does not log IOException
> ---
>
> Key: HADOOP-15717
> URL: https://issues.apache.org/jira/browse/HADOOP-15717
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: HADOOP-15717.001.patch, HADOOP-15717.002.patch
>
>
> I came across a case where tgt.getEndTime() was returned null and it resulted
> in an NPE, this observation was popped out of a test suite execution on a
> cluster. The reason for logging the {{IOException}} is that it helps to
> troubleshoot what caused the exception, as it can come from two different
> calls from the try-catch.
> I can see that [~gabor.bota] handled this with HADOOP-15593, but apart from
> logging the fact that the ticket's {{endDate}} was null, we have not logged
> the exception at all.
> With the current code, the exception is swallowed and the thread terminates
> in case the ticket's {{endDate}} is null.
> As this can happen with OpenJDK for example, it is required to print the
> exception (stack trace, message) to the log.
> The code should be updated here:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L918
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15717) TGT renewal thread does not log IOException
[
https://issues.apache.org/jira/browse/HADOOP-15717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645635#comment-16645635
]
Xiao Chen commented on HADOOP-15717:
Thanks [~snemeth] for the new rev.
The logging _can_ be done with slf4j parameterized logging. See
https://www.slf4j.org/apidocs/org/slf4j/Logger.html
For example, the following will log ie into the message with its stacktrace.
{noformat}
LOG.error("TGT is destroyed. Aborting renew thread for {}.",
getUserName(), ie);
{noformat}
> TGT renewal thread does not log IOException
> ---
>
> Key: HADOOP-15717
> URL: https://issues.apache.org/jira/browse/HADOOP-15717
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: HADOOP-15717.001.patch, HADOOP-15717.002.patch
>
>
> I came across a case where tgt.getEndTime() was returned null and it resulted
> in an NPE, this observation was popped out of a test suite execution on a
> cluster. The reason for logging the {{IOException}} is that it helps to
> troubleshoot what caused the exception, as it can come from two different
> calls from the try-catch.
> I can see that [~gabor.bota] handled this with HADOOP-15593, but apart from
> logging the fact that the ticket's {{endDate}} was null, we have not logged
> the exception at all.
> With the current code, the exception is swallowed and the thread terminates
> in case the ticket's {{endDate}} is null.
> As this can happen with OpenJDK for example, it is required to print the
> exception (stack trace, message) to the log.
> The code should be updated here:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L918
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15676) Cleanup TestSSLHttpServer
[
https://issues.apache.org/jira/browse/HADOOP-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16645310#comment-16645310
]
Xiao Chen commented on HADOOP-15676:
Thanks for revving, [~snemeth].
I understand it's existing code and you're just fixing a typo out of the
message, and thanks for doing that. But since we're changing it, let's try to
improve it for good.
IMO that try-catch-fail is bad, because only the exception message will be
logged without the actual stack trace. To me, that's less information because
otherwise one can just get to the failure point by looking at the stack trace -
now this has to be done by text search and log correlation. The message itself
doesn't seem to have substantial information either. If you'd like more
information, we can log the ciphers before calling into the function.
pre-commits are gone but I think the checkstyle is complaining about
{{oneEnabledCiphers}} and {{excludedCiphers}} not meeting the constant naming
regex (needs to be upper case)
> Cleanup TestSSLHttpServer
> -
>
> Key: HADOOP-15676
> URL: https://issues.apache.org/jira/browse/HADOOP-15676
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 2.6.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: HADOOP-15676.001.patch, HADOOP-15676.002.patch,
> HADOOP-15676.003.patch
>
>
> This issue will fix:
> * Several typos in this class
> * Code is not very well readable in some of the places.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15835) Reuse Object Mapper in KMSJSONWriter
[ https://issues.apache.org/jira/browse/HADOOP-15835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16644258#comment-16644258 ] Xiao Chen commented on HADOOP-15835: +1, thanks for the backport! > Reuse Object Mapper in KMSJSONWriter > > > Key: HADOOP-15835 > URL: https://issues.apache.org/jira/browse/HADOOP-15835 > Project: Hadoop Common > Issue Type: Bug >Reporter: Jonathan Eagles >Assignee: Jonathan Eagles >Priority: Major > Attachments: HADOOP-15835.001-branch-2.9.patch, > HADOOP-15835.001-branch-3.0.patch > > > In lie of HADOOP-15550 in branch-3.0, branch-2.9, branch-2.8. This patch will > provide some benefit of MapperObject reuse though not as complete as the > JsonSerialization util lazy loading fix. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16643813#comment-16643813
]
Xiao Chen commented on HADOOP-14445:
[^HADOOP-14445.19.patch]
Thanks for the review [~ajayydv].
bq. canonicalService field in LoadBalancingKMSClientProvider
The canonicalService is used for token selection. We depend on LBKMSCP and
KMSCP's canonicalService to handle all combinations of token look up, so both
are needed. You can try debug this with a token with client and a new client to
see how it works differently. :)
Good idea on the additional test coverage. Updated in patch 19.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.19.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.19.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15811) Optimizations for Java's TLS performance
[ https://issues.apache.org/jira/browse/HADOOP-15811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16642543#comment-16642543 ] Xiao Chen commented on HADOOP-15811: Catching up emails after PTO. Thanks [~daryn] for the ping, and great archeology here... Also thanks [~Steven Rand] for the pointer. [~knanasi] or [~jojochuang], is this something you're interested to test out? I'm happy to review if so. > Optimizations for Java's TLS performance > > > Key: HADOOP-15811 > URL: https://issues.apache.org/jira/browse/HADOOP-15811 > Project: Hadoop Common > Issue Type: Bug > Components: common >Affects Versions: 1.0.0 >Reporter: Daryn Sharp >Priority: Major > > Java defaults to using /dev/random and disables intrinsic methods used in hot > code paths. Both cause highly synchronized impls to be used that > significantly degrade performance. > * -Djava.security.egd=file:/dev/urandom > * -XX:+UseMontgomerySquareIntrinsic > * -XX:+UseMontgomeryMultiplyIntrinsic > * -XX:+UseSquareToLenIntrinsic > * -XX:+UseMultiplyToLenIntrinsic > These settings significantly boost KMS server performance. Under load, > threads are not jammed in the SSLEngine. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15717) TGT renewal thread does not log IOException
[
https://issues.apache.org/jira/browse/HADOOP-15717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16612398#comment-16612398
]
Xiao Chen commented on HADOOP-15717:
Thanks for the investigation [~snemeth], good find.
Would it suffice if we log the caught exception {{ie}} in the two existing
{{LOG.error}} statements? These 2, combining with the existing {{LOG.warn}},
would cover all possible cases in the catch block for now, and we don't have to
add another logging statement.
> TGT renewal thread does not log IOException
> ---
>
> Key: HADOOP-15717
> URL: https://issues.apache.org/jira/browse/HADOOP-15717
> Project: Hadoop Common
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: HADOOP-15717.001.patch
>
>
> I came across a case where tgt.getEndTime() was returned null and it resulted
> in an NPE, this observation was popped out of a test suite execution on a
> cluster. The reason for logging the {{IOException}} is that it helps to
> troubleshoot what caused the exception, as it can come from two different
> calls from the try-catch.
> I can see that [~gabor.bota] handled this with HADOOP-15593, but apart from
> logging the fact that the ticket's {{endDate}} was null, we have not logged
> the exception at all.
> With the current code, the exception is swallowed and the thread terminates
> in case the ticket's {{endDate}} is null.
> As this can happen with OpenJDK for example, it is required to print the
> exception (stack trace, message) to the log.
> The code should be updated here:
> https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L918
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15743) Jetty and SSL tunings to stabilize KMS performance
[ https://issues.apache.org/jira/browse/HADOOP-15743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611131#comment-16611131 ] Xiao Chen commented on HADOOP-15743: Thanks [~daryn] for the good summary and thanks for your week of life. :) FWIW we have also found JDK8 121+ (or said to be jdk9, untested) is more efficient in doing SSLHandShakes, mostly from https://bugs.openjdk.java.net/browse/JDK-8133070 and https://bugs.openjdk.java.net/browse/JDK-8129988. > Jetty and SSL tunings to stabilize KMS performance > --- > > Key: HADOOP-15743 > URL: https://issues.apache.org/jira/browse/HADOOP-15743 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.8.0 >Reporter: Daryn Sharp >Priority: Major > > The KMS has very low throughput with high client failure rates. The > following config options will "stabilize" the KMS under load: > # Disable ECDH algos because java's SSL engine is inexplicably HORRIBLE. > # Reduce SSL session cache size (unlimited) and ttl (24h). The memory cache > has very poor performance and causes extreme GC collection pressure. Load > balancing diminishes the effectiveness of the cache to 1/N-hosts anyway. > ** -Djavax.net.ssl.sessionCacheSize=1000 > ** -Djavax.net.ssl.sessionCacheTimeout=6 > # Completely disable thread LowResourceMonitor to stop jetty from > immediately closing incoming connections during connection bursts. Client > retries cause jetty to remain in a low resource state until many clients fail > and cause thousands of sockets to linger in various close related states. > # Set min/max threads to 4x processors. Jetty recommends only 50 to 500 > threads. Java's SSL engine has excessive synchronization that limits > performance anyway. > # Set https idle timeout to 6s. > # Significantly increase max fds to at least 128k. Recommend using a VIP > load balancer with a lower limit. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16611021#comment-16611021
]
Xiao Chen commented on HADOOP-14445:
bq. no functional changes were made?
Nope, besides cosmetic updates and adding debug logs / test cases, the only
'functional' change was the removal of the server defaults stuff in
WebHdfsFileSystem, which seems to fit better with another jira.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15676) Cleanup TestSSLHttpServer
[
https://issues.apache.org/jira/browse/HADOOP-15676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16608286#comment-16608286
]
Xiao Chen commented on HADOOP-15676:
Thanks [~snemeth] for the improvement here.
Looks like {{testOneEnabledCiphers}} and {{testExclusiveEnabledCiphers}} has
some duplicate codes.
And usually it's not recommended to:
{code}
try {
...
} catch (Exception e) {
fail()
}
{code}
One can expect that the exception can trigger the test failure, and that would
be enough for investigation.
+1 once that's fixed and precommit checkstyle is addressed.
> Cleanup TestSSLHttpServer
> -
>
> Key: HADOOP-15676
> URL: https://issues.apache.org/jira/browse/HADOOP-15676
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common
>Affects Versions: 2.6.0
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: HADOOP-15676.001.patch, HADOOP-15676.002.patch
>
>
> This issue will fix:
> * Several typos in this class
> * Code is not very well readable in some of the places.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606661#comment-16606661
]
Xiao Chen commented on HADOOP-14445:
Pre-commit failure seems intermittent, re-kicked.
{noformat}
[INFO] Running 'yarn install' in
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-ui/target/webapp
[INFO] yarn install v0.21.3
[INFO] [1/4] Resolving packages...
[INFO] [2/4] Fetching packages...
[ERROR] error An unexpected error occurred:
"https://registry.yarnpkg.com/sri-toolbox/-/sri-toolbox-0.2.0.tgz: Request
failed \"500 Internal Server Error\"".
{noformat}
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606403#comment-16606403
]
Xiao Chen commented on HADOOP-14445:
[^HADOOP-14445.18.patch] ready for review.
Fixed pre-commit complaints. Removed the server default configs which seems to
be an accidental change in the patch... Made {{getAdditionalTokenIssuers}} in
FileSystem to be {{InterfaceAudience.Private}} because I don't think it's
supposed to be used called by downstream.
[~daryn], want to do a semi-self-review? :)
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.18.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.18.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16605268#comment-16605268
]
Xiao Chen commented on HADOOP-14445:
Reviewed and tested in a test cluster, works nicely both ways. :) My
1-line-summary of the kms compat part would be: old client selects the token
based on the credential map's key (which is the service of the token), and not
the service of the token.
Attached patch 17 to kick a jenkins run. It has mostly small tweaks to make
this more debuggable, and polished up test cases. Removed a few end-to-end
tests that are no longer needed now that we don't use a config, and added a few
tests to verify KMSCP / LBKMSCP directly.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.17.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch, HADOOP-14445.17.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[ https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16604785#comment-16604785 ] Xiao Chen commented on HADOOP-15696: +1 on fixing this in 3.1/3.0 given the severity and impact of this issue > KMS performance regression due to too many open file descriptors after Jetty > migration > -- > > Key: HADOOP-15696 > URL: https://issues.apache.org/jira/browse/HADOOP-15696 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15696.001.patch, HADOOP-15696.002.patch, > HADOOP-15696.003.patch, HADOOP-15696.branch-3.1.001.patch, Screen Shot > 2018-08-22 at 11.36.16 AM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, > Screen Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.27.02 > PM.png, Screen Shot 2018-08-22 at 4.30.32 PM.png, Screen Shot 2018-08-22 at > 4.30.39 PM.png, Screen Shot 2018-08-24 at 7.08.16 PM.png > > > We recently found KMS performance regressed in Hadoop 3.0, possibly linking > to the migration from Tomcat to Jetty in HADOOP-13597. > Symptoms: > # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand > under stress, sometimes even exceeds 32K, which is the system limit, causing > failures for any access to encryption zones. Our internal testing shows the > openfd number was in the range of a few hundred in Hadoop 2.x, and it > increases by almost 100x in Hadoop 3. > # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same > heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are > temporary byte arrays associated with open SSL connections. > # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and > we observed up to 20% performance reduction due to GC. > A possible solution is to reduce the idle timeout setting in HttpServer2. It > is currently hard-coded 10 seconds. By setting it to 1 second, open fds > dropped from 20 thousand down to 3 thousand in my experiment. > File this jira to invite open discussion for a solution. > Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy; > [~xiaochen] for digging into this problem. > Screenshots: > CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.39 PM.png! > CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.32 PM.png! > CDH5 (Hadoop 2) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.26.51 PM.png! > CDH6 (Hadoop 3) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.27.02 PM.png! > JXray report > !Screen Shot 2018-08-22 at 11.36.16 AM.png! > open fd drops from 20 k down to 3k after the proposed change. > !Screen Shot 2018-08-24 at 7.08.16 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[ https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15696: --- Fix Version/s: 3.1.2 3.0.4 > KMS performance regression due to too many open file descriptors after Jetty > migration > -- > > Key: HADOOP-15696 > URL: https://issues.apache.org/jira/browse/HADOOP-15696 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15696.001.patch, HADOOP-15696.002.patch, > HADOOP-15696.003.patch, HADOOP-15696.branch-3.1.001.patch, Screen Shot > 2018-08-22 at 11.36.16 AM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, > Screen Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.27.02 > PM.png, Screen Shot 2018-08-22 at 4.30.32 PM.png, Screen Shot 2018-08-22 at > 4.30.39 PM.png, Screen Shot 2018-08-24 at 7.08.16 PM.png > > > We recently found KMS performance regressed in Hadoop 3.0, possibly linking > to the migration from Tomcat to Jetty in HADOOP-13597. > Symptoms: > # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand > under stress, sometimes even exceeds 32K, which is the system limit, causing > failures for any access to encryption zones. Our internal testing shows the > openfd number was in the range of a few hundred in Hadoop 2.x, and it > increases by almost 100x in Hadoop 3. > # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same > heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are > temporary byte arrays associated with open SSL connections. > # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and > we observed up to 20% performance reduction due to GC. > A possible solution is to reduce the idle timeout setting in HttpServer2. It > is currently hard-coded 10 seconds. By setting it to 1 second, open fds > dropped from 20 thousand down to 3 thousand in my experiment. > File this jira to invite open discussion for a solution. > Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy; > [~xiaochen] for digging into this problem. > Screenshots: > CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.39 PM.png! > CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.32 PM.png! > CDH5 (Hadoop 2) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.26.51 PM.png! > CDH6 (Hadoop 3) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.27.02 PM.png! > JXray report > !Screen Shot 2018-08-22 at 11.36.16 AM.png! > open fd drops from 20 k down to 3k after the proposed change. > !Screen Shot 2018-08-24 at 7.08.16 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[ https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603931#comment-16603931 ] Xiao Chen commented on HADOOP-15696: +1 pending jenkins > KMS performance regression due to too many open file descriptors after Jetty > migration > -- > > Key: HADOOP-15696 > URL: https://issues.apache.org/jira/browse/HADOOP-15696 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Attachments: HADOOP-15696.001.patch, HADOOP-15696.002.patch, > HADOOP-15696.003.patch, Screen Shot 2018-08-22 at 11.36.16 AM.png, Screen > Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, > Screen Shot 2018-08-22 at 4.27.02 PM.png, Screen Shot 2018-08-22 at 4.30.32 > PM.png, Screen Shot 2018-08-22 at 4.30.39 PM.png, Screen Shot 2018-08-24 at > 7.08.16 PM.png > > > We recently found KMS performance regressed in Hadoop 3.0, possibly linking > to the migration from Tomcat to Jetty in HADOOP-13597. > Symptoms: > # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand > under stress, sometimes even exceeds 32K, which is the system limit, causing > failures for any access to encryption zones. Our internal testing shows the > openfd number was in the range of a few hundred in Hadoop 2.x, and it > increases by almost 100x in Hadoop 3. > # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same > heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are > temporary byte arrays associated with open SSL connections. > # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and > we observed up to 20% performance reduction due to GC. > A possible solution is to reduce the idle timeout setting in HttpServer2. It > is currently hard-coded 10 seconds. By setting it to 1 second, open fds > dropped from 20 thousand down to 3 thousand in my experiment. > File this jira to invite open discussion for a solution. > Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy; > [~xiaochen] for digging into this problem. > Screenshots: > CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.39 PM.png! > CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.32 PM.png! > CDH5 (Hadoop 2) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.26.51 PM.png! > CDH6 (Hadoop 3) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.27.02 PM.png! > JXray report > !Screen Shot 2018-08-22 at 11.36.16 AM.png! > open fd drops from 20 k down to 3k after the proposed change. > !Screen Shot 2018-08-24 at 7.08.16 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[ https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603444#comment-16603444 ] Xiao Chen commented on HADOOP-15696: +1 on patch 2 pending pre-commit fixes. (whitebox seems should work using HADOOP-14188 at least on trunk, or exposing a test-only method on HttpServer2. I don't feel strongly either way.) Thanks for the work, Wei-Chiu! > KMS performance regression due to too many open file descriptors after Jetty > migration > -- > > Key: HADOOP-15696 > URL: https://issues.apache.org/jira/browse/HADOOP-15696 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Blocker > Attachments: HADOOP-15696.001.patch, HADOOP-15696.002.patch, Screen > Shot 2018-08-22 at 11.36.16 AM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, > Screen Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.27.02 > PM.png, Screen Shot 2018-08-22 at 4.30.32 PM.png, Screen Shot 2018-08-22 at > 4.30.39 PM.png, Screen Shot 2018-08-24 at 7.08.16 PM.png > > > We recently found KMS performance regressed in Hadoop 3.0, possibly linking > to the migration from Tomcat to Jetty in HADOOP-13597. > Symptoms: > # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand > under stress, sometimes even exceeds 32K, which is the system limit, causing > failures for any access to encryption zones. Our internal testing shows the > openfd number was in the range of a few hundred in Hadoop 2.x, and it > increases by almost 100x in Hadoop 3. > # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same > heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are > temporary byte arrays associated with open SSL connections. > # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and > we observed up to 20% performance reduction due to GC. > A possible solution is to reduce the idle timeout setting in HttpServer2. It > is currently hard-coded 10 seconds. By setting it to 1 second, open fds > dropped from 20 thousand down to 3 thousand in my experiment. > File this jira to invite open discussion for a solution. > Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy; > [~xiaochen] for digging into this problem. > Screenshots: > CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.39 PM.png! > CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor > chart. > !Screen Shot 2018-08-22 at 4.30.32 PM.png! > CDH5 (Hadoop 2) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.26.51 PM.png! > CDH6 (Hadoop 3) GC activities on the KMS process > !Screen Shot 2018-08-22 at 4.27.02 PM.png! > JXray report > !Screen Shot 2018-08-22 at 11.36.16 AM.png! > open fd drops from 20 k down to 3k after the proposed change. > !Screen Shot 2018-08-24 at 7.08.16 PM.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[
https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599503#comment-16599503
]
Xiao Chen commented on HADOOP-15696:
bq. Can you elaborate on "the connection object is created in relation to the
caller ugi" statement? What's the exact condition for when a connection can be
reused or not?
That's the {{conn = getActualUgi().doAs(...)}} part of the code, meaning that
if userA and userB get a hold of the same client object, their calls should not
reuse each other's connections. May be a little off topic, but the hadoop ipc
equivalent is
https://github.com/apache/hadoop/blob/branch-3.0.0/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ipc/Client.java#L1564
> KMS performance regression due to too many open file descriptors after Jetty
> migration
> --
>
> Key: HADOOP-15696
> URL: https://issues.apache.org/jira/browse/HADOOP-15696
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Attachments: HADOOP-15696.001.patch, HADOOP-15696.002.patch, Screen
> Shot 2018-08-22 at 11.36.16 AM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png,
> Screen Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.27.02
> PM.png, Screen Shot 2018-08-22 at 4.30.32 PM.png, Screen Shot 2018-08-22 at
> 4.30.39 PM.png, Screen Shot 2018-08-24 at 7.08.16 PM.png
>
>
> We recently found KMS performance regressed in Hadoop 3.0, possibly linking
> to the migration from Tomcat to Jetty in HADOOP-13597.
> Symptoms:
> # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand
> under stress, sometimes even exceeds 32K, which is the system limit, causing
> failures for any access to encryption zones. Our internal testing shows the
> openfd number was in the range of a few hundred in Hadoop 2.x, and it
> increases by almost 100x in Hadoop 3.
> # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same
> heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are
> temporary byte arrays associated with open SSL connections.
> # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and
> we observed up to 20% performance reduction due to GC.
> A possible solution is to reduce the idle timeout setting in HttpServer2. It
> is currently hard-coded 10 seconds. By setting it to 1 second, open fds
> dropped from 20 thousand down to 3 thousand in my experiment.
> File this jira to invite open discussion for a solution.
> Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy;
> [~xiaochen] for digging into this problem.
> Screenshots:
> CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor
> chart.
> !Screen Shot 2018-08-22 at 4.30.39 PM.png!
> CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor
> chart.
> !Screen Shot 2018-08-22 at 4.30.32 PM.png!
> CDH5 (Hadoop 2) GC activities on the KMS process
> !Screen Shot 2018-08-22 at 4.26.51 PM.png!
> CDH6 (Hadoop 3) GC activities on the KMS process
> !Screen Shot 2018-08-22 at 4.27.02 PM.png!
> JXray report
> !Screen Shot 2018-08-22 at 11.36.16 AM.png!
> open fd drops from 20 k down to 3k after the proposed change.
> !Screen Shot 2018-08-24 at 7.08.16 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15696) KMS performance regression due to too many open file descriptors after Jetty migration
[
https://issues.apache.org/jira/browse/HADOOP-15696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597076#comment-16597076
]
Xiao Chen commented on HADOOP-15696:
Thanks for the work here [~jojochuang] and [~mi...@cloudera.com], good find and
discussions!
Everything you said make sense. I'm not entirely sure, but I think originally
with tomcat, this way of connection setup didn't cause any issue probably
because the connection was polled by tomcat. This has proven to be problematic
in Jetty, and glad having the idle timeout dropped helped this situation.
How about we do this jira first to make it configurable, and continue the
connection reuse improvement as a follow-on? A tricky part for the connection
reuse is (at least on current trunk) the connection object is created in
relation to the caller ugi, so one would need to make sure the reuse doesn't
expose any security concerns. From use case perspective, I agree the connection
should be reused - and if not, the connection to be proactively closed by the
client.
Minor code review comment on patch 1:
- Instead of calling it {{HTTP_IDLE_TIMEOUT}} we may call it
{{HTTP_IDLE_TIMEOUT_MS}} for clarity about the millisecond unit
- It seems httpfs-default could also use such an update.
> KMS performance regression due to too many open file descriptors after Jetty
> migration
> --
>
> Key: HADOOP-15696
> URL: https://issues.apache.org/jira/browse/HADOOP-15696
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.0.0-alpha2
>Reporter: Wei-Chiu Chuang
>Assignee: Wei-Chiu Chuang
>Priority: Blocker
> Attachments: HADOOP-15696.001.patch, Screen Shot 2018-08-22 at
> 11.36.16 AM.png, Screen Shot 2018-08-22 at 4.26.51 PM.png, Screen Shot
> 2018-08-22 at 4.26.51 PM.png, Screen Shot 2018-08-22 at 4.27.02 PM.png,
> Screen Shot 2018-08-22 at 4.30.32 PM.png, Screen Shot 2018-08-22 at 4.30.39
> PM.png, Screen Shot 2018-08-24 at 7.08.16 PM.png
>
>
> We recently found KMS performance regressed in Hadoop 3.0, possibly linking
> to the migration from Tomcat to Jetty in HADOOP-13597.
> Symptoms:
> # Hadoop 3.x KMS open file descriptors quickly rises to more than 10 thousand
> under stress, sometimes even exceeds 32K, which is the system limit, causing
> failures for any access to encryption zones. Our internal testing shows the
> openfd number was in the range of a few hundred in Hadoop 2.x, and it
> increases by almost 100x in Hadoop 3.
> # Hadoop 3.x KMS as much as twice the heap size than in Hadoop 2.x. The same
> heap size can go OOM in Hadoop 3.x. Jxray analysis suggests most of them are
> temporary byte arrays associated with open SSL connections.
> # Due to the heap usage, Hadoop 3.x KMS has more frequent GC activities, and
> we observed up to 20% performance reduction due to GC.
> A possible solution is to reduce the idle timeout setting in HttpServer2. It
> is currently hard-coded 10 seconds. By setting it to 1 second, open fds
> dropped from 20 thousand down to 3 thousand in my experiment.
> File this jira to invite open discussion for a solution.
> Credit: [~mi...@cloudera.com] for the proposed Jetty idle timeout remedy;
> [~xiaochen] for digging into this problem.
> Screenshots:
> CDH5 (Hadoop 2) KMS CPU utilization, resident memory and file descriptor
> chart.
> !Screen Shot 2018-08-22 at 4.30.39 PM.png!
> CDH6 (Hadoop 3) KMS CPU utilization, resident memory and file descriptor
> chart.
> !Screen Shot 2018-08-22 at 4.30.32 PM.png!
> CDH5 (Hadoop 2) GC activities on the KMS process
> !Screen Shot 2018-08-22 at 4.26.51 PM.png!
> CDH6 (Hadoop 3) GC activities on the KMS process
> !Screen Shot 2018-08-22 at 4.27.02 PM.png!
> JXray report
> !Screen Shot 2018-08-22 at 11.36.16 AM.png!
> open fd drops from 20 k down to 3k after the proposed change.
> !Screen Shot 2018-08-24 at 7.08.16 PM.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15698) KMS log4j is not initialized properly at startup
[ https://issues.apache.org/jira/browse/HADOOP-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16597067#comment-16597067 ] Xiao Chen edited comment on HADOOP-15698 at 8/30/18 5:11 AM: - Committed to trunk as well as branch-3.[0-1] (since I believe this is a regression from HADOOP-13597). Thanks Kitti for the nice fix, and Ajay / Daniel for the review comments! was (Author: xiaochen): Committed to trunk as well as branch-3.[0-1] (since I believe this is a regression from HADOOP-13597). Thanks Kitti for the nice fix, and Ajay / Daniel for the comments! > KMS log4j is not initialized properly at startup > > > Key: HADOOP-15698 > URL: https://issues.apache.org/jira/browse/HADOOP-15698 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15698.001.patch > > > During KMs startup, log4j logs don't show up resulting in important logs > getting omitted. This happens because log4 initialisation only happens in > KMSWebApp#contextInitialized and logs written before that don't show up. > For example the following log never shows up: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java#L197-L199] > Another example is that the KMS startup message never shows up in the kms > logs. > Note that this works in the unit tests, because MiniKMS sets the log4j system > property. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15698) KMS log4j is not initialized properly at startup
[ https://issues.apache.org/jira/browse/HADOOP-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15698: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.2 3.0.4 3.2.0 Status: Resolved (was: Patch Available) Committed to trunk as well as branch-3.[0-1] (since I believe this is a regression from HADOOP-13597). Thanks Kitti for the nice fix, and Ajay / Daniel for the comments! > KMS log4j is not initialized properly at startup > > > Key: HADOOP-15698 > URL: https://issues.apache.org/jira/browse/HADOOP-15698 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15698.001.patch > > > During KMs startup, log4j logs don't show up resulting in important logs > getting omitted. This happens because log4 initialisation only happens in > KMSWebApp#contextInitialized and logs written before that don't show up. > For example the following log never shows up: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java#L197-L199] > Another example is that the KMS startup message never shows up in the kms > logs. > Note that this works in the unit tests, because MiniKMS sets the log4j system > property. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15698) KMS log4j is not initialized properly at startup
[ https://issues.apache.org/jira/browse/HADOOP-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15698: --- Summary: KMS log4j is not initialized properly at startup (was: KMS startup logs don't show) > KMS log4j is not initialized properly at startup > > > Key: HADOOP-15698 > URL: https://issues.apache.org/jira/browse/HADOOP-15698 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HADOOP-15698.001.patch > > > During KMs startup, log4j logs don't show up resulting in important logs > getting omitted. This happens because log4 initialisation only happens in > KMSWebApp#contextInitialized and logs written before that don't show up. > For example the following log never shows up: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java#L197-L199] > Another example is that the KMS startup message never shows up in the kms > logs. > Note that this works in the unit tests, because MiniKMS sets the log4j system > property. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Resolved] (HADOOP-14238) [Umbrella] Rechecking Guava's object is not exposed to user-facing API
[ https://issues.apache.org/jira/browse/HADOOP-14238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen resolved HADOOP-14238. Resolution: Fixed Closing as I don't see any more work for this at this time > [Umbrella] Rechecking Guava's object is not exposed to user-facing API > -- > > Key: HADOOP-14238 > URL: https://issues.apache.org/jira/browse/HADOOP-14238 > Project: Hadoop Common > Issue Type: Improvement >Reporter: Tsuyoshi Ozawa >Priority: Critical > > This is reported by [~hitesh] on HADOOP-10101. > At least, AMRMClient#waitFor takes Guava's Supplier instance as an instance. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595876#comment-16595876
]
Xiao Chen commented on HADOOP-14445:
Thanks Daryn. I'm reviewing the patch, will improve it and do some additional
testing. Should come back by end of this week.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.compat.patch, HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15698) KMS startup logs don't show
[ https://issues.apache.org/jira/browse/HADOOP-15698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16595579#comment-16595579 ] Xiao Chen commented on HADOOP-15698: Thanks Kitti for working on this, good find! Also thanks Ajay for the comment. +1 on patch 1. Thanks Ajay for bringing up the testing idea, and I agree we want tests with a fix in general. Giving the static initialization of things, IMO this fix can go with just the manual testing. I tried locally and can see the logs after a kms startup. Also good to know that Kitti tested this in a cluster where the issue was occurring before and fixed after. If no more concerns I plan to commit this by end of Wednesday. > KMS startup logs don't show > --- > > Key: HADOOP-15698 > URL: https://issues.apache.org/jira/browse/HADOOP-15698 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.1.0 >Reporter: Kitti Nanasi >Assignee: Kitti Nanasi >Priority: Major > Attachments: HADOOP-15698.001.patch > > > During KMs startup, log4j logs don't show up resulting in important logs > getting omitted. This happens because log4 initialisation only happens in > KMSWebApp#contextInitialized and logs written before that don't show up. > For example the following log never shows up: > [https://github.com/apache/hadoop/blob/a55d6bba71c81c1c4e9d8cd11f55c78f10a548b0/hadoop-common-project/hadoop-auth/src/main/java/org/apache/hadoop/security/authentication/util/ZKSignerSecretProvider.java#L197-L199] > Another example is that the KMS startup message never shows up in the kms > logs. > Note that this works in the unit tests, because MiniKMS sets the log4j system > property. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16590670#comment-16590670
]
Xiao Chen commented on HADOOP-14445:
Good to know, thanks for the update [~daryn].
I agree we can take advantage of the kind-less token lookup to build the silver
bullet. Was trying to make the new behavior correctly based on token kind in
the last patch, but looking forward to the magical patch before I really hold
any opinion.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16586915#comment-16586915
]
Xiao Chen commented on HADOOP-14445:
Thanks for the update [~daryn]. Happy to know you have a version too. Open to
simpler solutions. :)
Please let me know how I can help to move us forward.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15674) Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
[
https://issues.apache.org/jira/browse/HADOOP-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584824#comment-16584824
]
Xiao Chen commented on HADOOP-15674:
bq. Is there any step that I need to take?
Nope. If the conflicts were non-trivial, then you'd need to upload a patch. But
as mentioned, I have manually resolve the conflict at commit time, so we're
done here
> Test failure TestSSLHttpServer.testExcludedCiphers with
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
> --
>
> Key: HADOOP-15674
> URL: https://issues.apache.org/jira/browse/HADOOP-15674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.6.0
>Reporter: Gabor Bota
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 2.10.0, 2.9.2, 2.8.5, 2.7.8, 3.0.4, 3.2, 3.1.2
>
> Attachments: HADOOP-15674-branch-2.001.patch,
> HADOOP-15674-branch-2.002.patch, HADOOP-15674-branch-2.003.patch,
> HADOOP-15674-branch-3.0.0.001.patch, HADOOP-15674-branch-3.0.0.002.patch,
> HADOOP-15674-branch-3.0.0.003.patch, HADOOP-15674.001.patch,
> HADOOP-15674.002.patch, HADOOP-15674.003.patch, HADOOP-15674.004.patch
>
>
> Running {{hadoop/hadoop-common-project/hadoop-common# mvn test
> -Dtest="TestSSLHttpServer#testExcludedCiphers" -Dhttps.protocols=TLSv1.2
> -Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256}} fails with:
> {noformat}
> Error Message
> No Ciphers in common, SSLHandshake must fail.
> Stacktrace
> java.lang.AssertionError: No Ciphers in common, SSLHandshake must fail.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testExcludedCiphers(TestSSLHttpServer.java:178)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9214) Create a new touch command to allow modifying atime and mtime
[ https://issues.apache.org/jira/browse/HADOOP-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-9214: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) Committed to trunk. Thank you for the contribution, [~hgadre]! > Create a new touch command to allow modifying atime and mtime > - > > Key: HADOOP-9214 > URL: https://issues.apache.org/jira/browse/HADOOP-9214 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Affects Versions: 0.23.5 >Reporter: Brian Burton >Assignee: Hrishikesh Gadre >Priority: Minor > Fix For: 3.2.0 > > Attachments: HADOOP-9214-001.patch, HADOOP-9214-002.patch, > HADOOP-9214-003.patch, HADOOP-9214-004.patch, HADOOP-9214-005.patch, > HADOOP-9214-006.patch > > > Currently there is no way to set the mtime or atime of a file from the > "hadoop fs" command line. It would be useful if the 'hadoop fs -touchz' > command were updated to include this functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-9214) Create a new touch command to allow modifying atime and mtime
[ https://issues.apache.org/jira/browse/HADOOP-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-9214: -- Summary: Create a new touch command to allow modifying atime and mtime (was: Update touchz to allow modifying atime and mtime) > Create a new touch command to allow modifying atime and mtime > - > > Key: HADOOP-9214 > URL: https://issues.apache.org/jira/browse/HADOOP-9214 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Affects Versions: 0.23.5 >Reporter: Brian Burton >Assignee: Hrishikesh Gadre >Priority: Minor > Attachments: HADOOP-9214-001.patch, HADOOP-9214-002.patch, > HADOOP-9214-003.patch, HADOOP-9214-004.patch, HADOOP-9214-005.patch, > HADOOP-9214-006.patch > > > Currently there is no way to set the mtime or atime of a file from the > "hadoop fs" command line. It would be useful if the 'hadoop fs -touchz' > command were updated to include this functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9214) Update touchz to allow modifying atime and mtime
[ https://issues.apache.org/jira/browse/HADOOP-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584209#comment-16584209 ] Xiao Chen commented on HADOOP-9214: --- +1, committing this. Thanks for the great work here Hrishikesh, and thanks Daryn for the comment! > Update touchz to allow modifying atime and mtime > > > Key: HADOOP-9214 > URL: https://issues.apache.org/jira/browse/HADOOP-9214 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Affects Versions: 0.23.5 >Reporter: Brian Burton >Assignee: Hrishikesh Gadre >Priority: Minor > Attachments: HADOOP-9214-001.patch, HADOOP-9214-002.patch, > HADOOP-9214-003.patch, HADOOP-9214-004.patch, HADOOP-9214-005.patch, > HADOOP-9214-006.patch > > > Currently there is no way to set the mtime or atime of a file from the > "hadoop fs" command line. It would be useful if the 'hadoop fs -touchz' > command were updated to include this functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15674) Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
[
https://issues.apache.org/jira/browse/HADOOP-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15674:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.2
3.2
3.0.4
2.7.8
2.8.5
2.9.2
2.10.0
Status: Resolved (was: Patch Available)
Committed to trunk, branch-3[0-1], branch-2, branch-2.[7-9].
There was a minor conflict about log4j in branch-2.8 that needs to be taken
care of.
Verified the test passes locally on branch-2 and branch-2.8 before pushing.
> Test failure TestSSLHttpServer.testExcludedCiphers with
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
> --
>
> Key: HADOOP-15674
> URL: https://issues.apache.org/jira/browse/HADOOP-15674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.6.0
>Reporter: Gabor Bota
>Assignee: Szilard Nemeth
>Priority: Major
> Fix For: 2.10.0, 2.9.2, 2.8.5, 2.7.8, 3.0.4, 3.2, 3.1.2
>
> Attachments: HADOOP-15674-branch-2.001.patch,
> HADOOP-15674-branch-2.002.patch, HADOOP-15674-branch-2.003.patch,
> HADOOP-15674-branch-3.0.0.001.patch, HADOOP-15674-branch-3.0.0.002.patch,
> HADOOP-15674-branch-3.0.0.003.patch, HADOOP-15674.001.patch,
> HADOOP-15674.002.patch, HADOOP-15674.003.patch, HADOOP-15674.004.patch
>
>
> Running {{hadoop/hadoop-common-project/hadoop-common# mvn test
> -Dtest="TestSSLHttpServer#testExcludedCiphers" -Dhttps.protocols=TLSv1.2
> -Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256}} fails with:
> {noformat}
> Error Message
> No Ciphers in common, SSLHandshake must fail.
> Stacktrace
> java.lang.AssertionError: No Ciphers in common, SSLHandshake must fail.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testExcludedCiphers(TestSSLHttpServer.java:178)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15674) Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
[
https://issues.apache.org/jira/browse/HADOOP-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16584173#comment-16584173
]
Xiao Chen commented on HADOOP-15674:
+1, committing. Thanks for the working on this through the finish line
[~snemeth].
> Test failure TestSSLHttpServer.testExcludedCiphers with
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
> --
>
> Key: HADOOP-15674
> URL: https://issues.apache.org/jira/browse/HADOOP-15674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.6.0
>Reporter: Gabor Bota
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: HADOOP-15674-branch-2.001.patch,
> HADOOP-15674-branch-2.002.patch, HADOOP-15674-branch-2.003.patch,
> HADOOP-15674-branch-3.0.0.001.patch, HADOOP-15674-branch-3.0.0.002.patch,
> HADOOP-15674-branch-3.0.0.003.patch, HADOOP-15674.001.patch,
> HADOOP-15674.002.patch, HADOOP-15674.003.patch, HADOOP-15674.004.patch
>
>
> Running {{hadoop/hadoop-common-project/hadoop-common# mvn test
> -Dtest="TestSSLHttpServer#testExcludedCiphers" -Dhttps.protocols=TLSv1.2
> -Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256}} fails with:
> {noformat}
> Error Message
> No Ciphers in common, SSLHandshake must fail.
> Stacktrace
> java.lang.AssertionError: No Ciphers in common, SSLHandshake must fail.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testExcludedCiphers(TestSSLHttpServer.java:178)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15655) Enhance KMS client retry behavior
[
https://issues.apache.org/jira/browse/HADOOP-15655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15655:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks [~knanasi]!
> Enhance KMS client retry behavior
> -
>
> Key: HADOOP-15655
> URL: https://issues.apache.org/jira/browse/HADOOP-15655
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.1.0
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Critical
> Fix For: 3.2
>
> Attachments: HADOOP-15655.001.patch, HADOOP-15655.002.patch,
> HADOOP-15655.003.patch
>
>
> KMS doesn't retry upon SocketTimeoutException (the ssl connection was
> established, but the handshake timed out).
> {noformat}
> 6:08:55.315 PMWARNKMSClientProvider
> Failed to connect to example.com:16000
> 6:08:55.317 PMWARNLoadBalancingKMSClientProvider
> KMS provider at [https://example.com:16000/kms/v1/] threw an IOException:
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:140)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:473)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:472)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:788)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:288)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:124)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:949)
> at org.apache.hadoop.hbase.util.FSUtils.getVersion(FSUtils.java:338)
> at
[jira] [Updated] (HADOOP-15655) Enhance KMS client retry behavior
[
https://issues.apache.org/jira/browse/HADOOP-15655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15655:
---
Summary: Enhance KMS client retry behavior (was: KMS should retry upon
SocketTimeoutException)
> Enhance KMS client retry behavior
> -
>
> Key: HADOOP-15655
> URL: https://issues.apache.org/jira/browse/HADOOP-15655
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.1.0
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Critical
> Attachments: HADOOP-15655.001.patch, HADOOP-15655.002.patch,
> HADOOP-15655.003.patch
>
>
> KMS doesn't retry upon SocketTimeoutException (the ssl connection was
> established, but the handshake timed out).
> {noformat}
> 6:08:55.315 PMWARNKMSClientProvider
> Failed to connect to example.com:16000
> 6:08:55.317 PMWARNLoadBalancingKMSClientProvider
> KMS provider at [https://example.com:16000/kms/v1/] threw an IOException:
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:140)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:473)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:472)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:788)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:288)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:124)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
> at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:949)
> at org.apache.hadoop.hbase.util.FSUtils.getVersion(FSUtils.java:338)
> at org.apache.hadoop.hbase.util.FSUtils.checkVersion(FSUtils.java:423)
> at
>
[jira] [Commented] (HADOOP-15674) Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
[
https://issues.apache.org/jira/browse/HADOOP-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16583010#comment-16583010
]
Xiao Chen commented on HADOOP-15674:
Thanks for the work here [~snemeth] and [~gabor.bota] for reviewing!
Unfortunately, patch 3 looks to have accidentally included FSNamesystem.java
changes. Could you pull it out?
Since we will need a new rev, I have another minor comment on the patch:
I think the standard way to unset a property is
{code:java}
System.clearProperty(key);{code}
rather than
{code:java}
System.setProperty(key, "");{code}
Lastly, a patch specific to a branch is only needed if there is backport
conflicts. I tried the trunk patch, and it can be cleanly backported all the
way to branch-2.9 (and have conflicts when backporting to branch-2.8). So, no
need to generate branch-3 and branch-2 patches. If the conflicts were
non-trivial I would ask for a branch-2.8 patch, but given it is pretty easy to
resolve, I'll just take care of it at commit time.
> Test failure TestSSLHttpServer.testExcludedCiphers with
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
> --
>
> Key: HADOOP-15674
> URL: https://issues.apache.org/jira/browse/HADOOP-15674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.6.0
>Reporter: Gabor Bota
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: HADOOP-15674-branch-2.001.patch,
> HADOOP-15674-branch-2.002.patch, HADOOP-15674-branch-2.003.patch,
> HADOOP-15674-branch-3.0.0.001.patch, HADOOP-15674-branch-3.0.0.002.patch,
> HADOOP-15674-branch-3.0.0.003.patch, HADOOP-15674.001.patch,
> HADOOP-15674.002.patch, HADOOP-15674.003.patch
>
>
> Running {{hadoop/hadoop-common-project/hadoop-common# mvn test
> -Dtest="TestSSLHttpServer#testExcludedCiphers" -Dhttps.protocols=TLSv1.2
> -Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256}} fails with:
> {noformat}
> Error Message
> No Ciphers in common, SSLHandshake must fail.
> Stacktrace
> java.lang.AssertionError: No Ciphers in common, SSLHandshake must fail.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testExcludedCiphers(TestSSLHttpServer.java:178)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581894#comment-16581894
]
Xiao Chen commented on HADOOP-14445:
Thanks all for reviewing this. Look forward to Daryn's active reviews, will
address comments together..
Clarifications on the questions:
{quote}If the order of the kms instances change between renew for the long
running process? Will the token selector still be able to find the matching
token (AbstractDelegationTokenSelector#selectToken()) to renew? Should we
define a logical kms uri to represent KMS HA instances to avoid this?
{quote}
Good question. You're right that when the url changes, the selection wouldn't
work. My collection from the earlier discussion was that this was deemed an
acceptable behavior. See discussions above, until [this
comment|https://issues.apache.org/jira/browse/HADOOP-14445?focusedCommentId=16033492=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16033492].
{quote}
L347 in DelegationTokenAuthenticatedURL#selectDelegationToken will return a
service in ip:port or host:port format (not uri). So creds.getToken(service) in
next line may not return the HA KMS DT as service name will be in URI format.
{quote}
The goal of this jira is to add this new format, while maintaining backwards
compatibility. This means if the new software runs with old software (when the
config {{hadoop.security.kms.client.token.use.uri.format}} isn't changed), it
would behave the same. Hence the
{{DelegationTokenAuthenticatedURL#selectDelegationToken}} is simply a code
refactor.
The reason for that refactor is that, we can override this behavior in KMSCP
and work our way for the new format. The reason it falls back to DTAURL to
select token is for compat.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15655) KMS should retry upon SocketTimeoutException
[
https://issues.apache.org/jira/browse/HADOOP-15655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581880#comment-16581880
]
Xiao Chen commented on HADOOP-15655:
Thanks [~knanasi] for filing the jira and doing the good work here! Also thanks
Ajay and Gabor for reviewing. +1 on patch 3.
bq. I'd also like to point out that we don't have a unit test class for
RetryPolicies such as TestRetryPolicies
One way to figure out whether a class has tests is to look at the git history
of the class - in this case, the tests seems to be in {{TestRetryProxy}}.
Will commit this by end of Thursday if no comments from others.
> KMS should retry upon SocketTimeoutException
>
>
> Key: HADOOP-15655
> URL: https://issues.apache.org/jira/browse/HADOOP-15655
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 3.1.0
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Critical
> Attachments: HADOOP-15655.001.patch, HADOOP-15655.002.patch,
> HADOOP-15655.003.patch
>
>
> KMS doesn't retry upon SocketTimeoutException (the ssl connection was
> established, but the handshake timed out).
> {noformat}
> 6:08:55.315 PMWARNKMSClientProvider
> Failed to connect to example.com:16000
> 6:08:55.317 PMWARNLoadBalancingKMSClientProvider
> KMS provider at [https://example.com:16000/kms/v1/] threw an IOException:
> java.net.SocketTimeoutException: Read timed out
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
> at java.net.SocketInputStream.read(SocketInputStream.java:171)
> at java.net.SocketInputStream.read(SocketInputStream.java:141)
> at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
> at sun.security.ssl.InputRecord.read(InputRecord.java:503)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.connect(HttpsURLConnectionImpl.java:153)
> at
> org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:186)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:140)
> at
> org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:348)
> at
> org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:333)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:478)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:473)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1685)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:472)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:788)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:288)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:124)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
> at
>
[jira] [Commented] (HADOOP-9214) Update touchz to allow modifying atime and mtime
[ https://issues.apache.org/jira/browse/HADOOP-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581717#comment-16581717 ] Xiao Chen commented on HADOOP-9214: --- Thanks for the new rev [~hgadre]! LGTM, but will wait for a couple days to see if [~daryn] has comments. > Update touchz to allow modifying atime and mtime > > > Key: HADOOP-9214 > URL: https://issues.apache.org/jira/browse/HADOOP-9214 > Project: Hadoop Common > Issue Type: Improvement > Components: tools >Affects Versions: 0.23.5 >Reporter: Brian Burton >Assignee: Hrishikesh Gadre >Priority: Minor > Attachments: HADOOP-9214-001.patch, HADOOP-9214-002.patch, > HADOOP-9214-003.patch, HADOOP-9214-004.patch, HADOOP-9214-005.patch, > HADOOP-9214-006.patch > > > Currently there is no way to set the mtime or atime of a file from the > "hadoop fs" command line. It would be useful if the 'hadoop fs -touchz' > command were updated to include this functionality. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15674) Test failure TestSSLHttpServer.testExcludedCiphers with TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
[
https://issues.apache.org/jira/browse/HADOOP-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581565#comment-16581565
]
Xiao Chen commented on HADOOP-15674:
Thanks [~gabor.bota] for filing the jira and [~snemeth] for providing a fix.
Looks good to me overall. Some minor comments:
- log4j's parameterized messages can be used for logging. (See
https://logging.apache.org/log4j/2.0/manual/messages.html).
- In Javadoc I'd remove the mention of the jira number. The description seems
clear enough, and one can always git blame to find the jira number if needed.
We generally don't encourage fixing stuff unrelated to the jira to reduce the
chance of conflicts. But since this is a test class and the fix is only 1 typo,
I'll let this one slide.
> Test failure TestSSLHttpServer.testExcludedCiphers with
> TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 cipher suite
> --
>
> Key: HADOOP-15674
> URL: https://issues.apache.org/jira/browse/HADOOP-15674
> Project: Hadoop Common
> Issue Type: Bug
> Components: common
>Affects Versions: 2.6.0
>Reporter: Gabor Bota
>Assignee: Szilard Nemeth
>Priority: Major
> Attachments: HADOOP-15674-branch-2.001.patch,
> HADOOP-15674-branch-3.0.0.001.patch, HADOOP-15674.001.patch
>
>
> Running {{hadoop/hadoop-common-project/hadoop-common# mvn test
> -Dtest="TestSSLHttpServer#testExcludedCiphers" -Dhttps.protocols=TLSv1.2
> -Dhttps.cipherSuites=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256}} fails with:
> {noformat}
> Error Message
> No Ciphers in common, SSLHandshake must fail.
> Stacktrace
> java.lang.AssertionError: No Ciphers in common, SSLHandshake must fail.
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.hadoop.http.TestSSLHttpServer.testExcludedCiphers(TestSSLHttpServer.java:178)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-15675) checkstyle fails to execute
[
https://issues.apache.org/jira/browse/HADOOP-15675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581340#comment-16581340
]
Xiao Chen edited comment on HADOOP-15675 at 8/15/18 4:50 PM:
-
Thanks Allen for the lightning triage! Agreed Yetus should rather fail so
people can notice...
I have pinged HDDFS-119 for the checkstyle fix as well.
was (Author: xiaochen):
Thanks Allen for the lightning triage! Agreed Yetus should rather fail so
people can notice...
I have pinged HDDFS-119 about this as well.
> checkstyle fails to execute
> ---
>
> Key: HADOOP-15675
> URL: https://issues.apache.org/jira/browse/HADOOP-15675
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Kitti Nanasi
>Priority: Minor
>
> If a patch is created with checkstyle errors, for example when a modified
> line is longer than 80 characters, then running checkstyle with the
> test-patch script runs to success (though it should fail and show an error
> about the long line).
> {code:java}
> dev-support/bin/test-patch --plugins="-checkstyle" test.patch{code}
> However it does show the error (so works correctly) when running it with the
> IDEA checkstyle plugin.
>
> I only tried it out it for patches with too long lines and wrong indentation,
> but I assume that it can be a more general problem.
> We realised this when reviewing HDFS-13217, where patch 004 has a "too long
> line" checkstyle error. In the first build for that patch, the checkstyle
> report was showing the error, but when it was ran again with the same patch,
> the error disappeared. So probably the checkstyle checking stopped working on
> trunk somewhere between April and July 2018.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15675) checkstyle fails to execute
[
https://issues.apache.org/jira/browse/HADOOP-15675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16581340#comment-16581340
]
Xiao Chen commented on HADOOP-15675:
Thanks Allen for the lightning triage! Agreed Yetus should rather fail so
people can notice...
I have pinged HDDFS-119 about this as well.
> checkstyle fails to execute
> ---
>
> Key: HADOOP-15675
> URL: https://issues.apache.org/jira/browse/HADOOP-15675
> Project: Hadoop Common
> Issue Type: Bug
> Components: build
>Reporter: Kitti Nanasi
>Priority: Minor
>
> If a patch is created with checkstyle errors, for example when a modified
> line is longer than 80 characters, then running checkstyle with the
> test-patch script runs to success (though it should fail and show an error
> about the long line).
> {code:java}
> dev-support/bin/test-patch --plugins="-checkstyle" test.patch{code}
> However it does show the error (so works correctly) when running it with the
> IDEA checkstyle plugin.
>
> I only tried it out it for patches with too long lines and wrong indentation,
> but I assume that it can be a more general problem.
> We realised this when reviewing HDFS-13217, where patch 004 has a "too long
> line" checkstyle error. In the first build for that patch, the checkstyle
> report was showing the error, but when it was ran again with the same patch,
> the error disappeared. So probably the checkstyle checking stopped working on
> trunk somewhere between April and July 2018.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15665) Checkstyle shows false positive report
[
https://issues.apache.org/jira/browse/HADOOP-15665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15665:
---
Affects Version/s: (was: 3.1.0)
3.0.4
2.7.8
2.8.5
2.9.2
3.1.1
3.2.0
2.10.0
> Checkstyle shows false positive report
> --
>
> Key: HADOOP-15665
> URL: https://issues.apache.org/jira/browse/HADOOP-15665
> Project: Hadoop Common
> Issue Type: Bug
> Components: precommit, yetus
>Affects Versions: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 2.8.5, 2.7.8, 3.0.4
>Reporter: Kitti Nanasi
>Priority: Blocker
>
> If a patch is created with checkstyle errors, for example when a modified
> line is longer than 80 characters, then running checkstyle with the
> test-patch script runs to success (though it should fail and show an error
> about the long line).
> {code:java}
> dev-support/bin/test-patch --plugins="-checkstyle" test.patch{code}
> However it does show the error (so works correctly) when running it with the
> IDEA checkstyle plugin.
>
> I only tried it out it for patches with too long lines and wrong indentation,
> but I assume that it can be a more general problem.
> We realised this when reviewing HDFS-13217, where patch 004 has a "too long
> line" checkstyle error. In the first build for that patch, the checkstyle
> report was showing the error, but when it was ran again with the same patch,
> the error disappeared. So probably the checkstyle checking stopped working on
> trunk somewhere between April and July 2018.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-9214) Update touchz to allow modifying atime and mtime
[
https://issues.apache.org/jira/browse/HADOOP-9214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578892#comment-16578892
]
Xiao Chen commented on HADOOP-9214:
---
Thanks for working on this [~hgadre]. Looks pretty good overall!
I have 2 comments on the patch:
- It could theoretically be considered a change in behavior when touchz
(without any flags) a file which already exists: before this it's expected for
an new file (with CreateFlag == {{OVERWRITE}}) which will have a different
inode; now it will just update the time on the existing file. Let's please keep
the existing behavior of re-creating a file when no flags provided to touchz
for compatibility. When a flag is provided, I think it's perfectly reasonable
to just update the time as the patch is doing.
- In Touch.java, trivial but I think we should name the variables
{{changeModTimeOnly}} and {{changeAccessTimeOnly}} without 'only'. They simply
maps to the options that could be provided, and it seems we can check the
'only' part just by logic.
> Update touchz to allow modifying atime and mtime
>
>
> Key: HADOOP-9214
> URL: https://issues.apache.org/jira/browse/HADOOP-9214
> Project: Hadoop Common
> Issue Type: Improvement
> Components: tools
>Affects Versions: 0.23.5
>Reporter: Brian Burton
>Assignee: Hrishikesh Gadre
>Priority: Minor
> Attachments: HADOOP-9214-001.patch, HADOOP-9214-002.patch,
> HADOOP-9214-003.patch
>
>
> Currently there is no way to set the mtime or atime of a file from the
> "hadoop fs" command line. It would be useful if the 'hadoop fs -touchz'
> command were updated to include this functionality.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15638) KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x
[ https://issues.apache.org/jira/browse/HADOOP-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-15638: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.1.2 3.0.4 3.2.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-3.[0-1]. Thanks Wei-Chiu! > KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x > --- > > Key: HADOOP-15638 > URL: https://issues.apache.org/jira/browse/HADOOP-15638 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Fix For: 3.2.0, 3.0.4, 3.1.2 > > Attachments: HADOOP-15638.001.patch > > > HADOOP-13597 migrated KMS from Tomcat to Jetty. > In Hadoop 2.x, the Accept Queue Size default value was set by environment > variable KMS_ACCEPT_QUEUE with default value 500. > But after the migration, Hadoop 3.x now uses the default HttpServer2 > configuration to set accept queue size (hadoop.http.socket.backlog.size), > which is 128. > Should we restore the default accept queue size in kms-default.xml? Should we > document it as a known issue? After all, Hadoop 2 --> 3 allows breaking > changing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15638) KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x
[ https://issues.apache.org/jira/browse/HADOOP-15638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16578677#comment-16578677 ] Xiao Chen commented on HADOOP-15638: +1, thanks Wei-Chiu and Kitti! Committing this. > KMS Accept Queue Size default changed from 500 to 128 in Hadoop 3.x > --- > > Key: HADOOP-15638 > URL: https://issues.apache.org/jira/browse/HADOOP-15638 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 3.0.0-alpha2 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > Attachments: HADOOP-15638.001.patch > > > HADOOP-13597 migrated KMS from Tomcat to Jetty. > In Hadoop 2.x, the Accept Queue Size default value was set by environment > variable KMS_ACCEPT_QUEUE with default value 500. > But after the migration, Hadoop 3.x now uses the default HttpServer2 > configuration to set accept queue size (hadoop.http.socket.backlog.size), > which is 128. > Should we restore the default accept queue size in kms-default.xml? Should we > document it as a known issue? After all, Hadoop 2 --> 3 allows breaking > changing. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen reassigned HADOOP-14521: -- Assignee: Rushabh S Shah (was: Xiao Chen) > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Rushabh S Shah >Priority: Major > Fix For: 2.9.0, 2.8.3, 3.0.0 > > Attachments: HADOOP-14521-branch-2.8.002.patch, > HADOOP-14521-branch-2.8.2.patch, HADOOP-14521-trunk-10.patch, > HADOOP-14521.09.patch, HADOOP-14521.11.patch, HDFS-11804-branch-2.8.patch, > HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, HDFS-11804-trunk-3.patch, > HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, HDFS-11804-trunk-6.patch, > HDFS-11804-trunk-7.patch, HDFS-11804-trunk-8.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Assigned] (HADOOP-14521) KMS client needs retry logic
[ https://issues.apache.org/jira/browse/HADOOP-14521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen reassigned HADOOP-14521: -- Assignee: Xiao Chen (was: Rushabh S Shah) > KMS client needs retry logic > > > Key: HADOOP-14521 > URL: https://issues.apache.org/jira/browse/HADOOP-14521 > Project: Hadoop Common > Issue Type: Improvement >Affects Versions: 2.6.0 >Reporter: Rushabh S Shah >Assignee: Xiao Chen >Priority: Major > Fix For: 2.9.0, 2.8.3, 3.0.0 > > Attachments: HADOOP-14521-branch-2.8.002.patch, > HADOOP-14521-branch-2.8.2.patch, HADOOP-14521-trunk-10.patch, > HADOOP-14521.09.patch, HADOOP-14521.11.patch, HDFS-11804-branch-2.8.patch, > HDFS-11804-trunk-1.patch, HDFS-11804-trunk-2.patch, HDFS-11804-trunk-3.patch, > HDFS-11804-trunk-4.patch, HDFS-11804-trunk-5.patch, HDFS-11804-trunk-6.patch, > HDFS-11804-trunk-7.patch, HDFS-11804-trunk-8.patch, HDFS-11804-trunk.patch > > > The kms client appears to have no retry logic – at all. It's completely > decoupled from the ipc retry logic. This has major impacts if the KMS is > unreachable for any reason, including but not limited to network connection > issues, timeouts, the +restart during an upgrade+. > This has some major ramifications: > # Jobs may fail to submit, although oozie resubmit logic should mask it > # Non-oozie launchers may experience higher rates if they do not already have > retry logic. > # Tasks reading EZ files will fail, probably be masked by framework reattempts > # EZ file creation fails after creating a 0-length file – client receives > EDEK in the create response, then fails when decrypting the EDEK > # Bulk hadoop fs copies, and maybe distcp, will prematurely fail -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16567454#comment-16567454
]
Xiao Chen commented on HADOOP-14445:
Thanks again Wei-Chiu and Ajay.
I agree more testing is a good thing in general. For this case though, the KMS
HA behavior is orthogonal to the authentication method used. Specifically,
{{LBKMSCP#doOp}} (the HA behavior) does not depend on the config. So I didn't
add the test for now, because it wouldn't add more coverage. But feel free to
elaborate and convince me otherwise.
It seems [~daryn] isn't around for the rest of the week, I'll wait for a few
more days, and commit on next Wednesday (8/8) if no more comments. :)
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16563164#comment-16563164
]
Xiao Chen commented on HADOOP-14445:
Thanks [~jojochuang] and [~ajayydv].
New patch fixed pre-commit (which was a conflict resolution error I made when
rebasing) and Ajay's comments 1&2. For #3, the test case is testing KMSCP's
loadbalancing on failures. This patch does not change that code path and is
unrelated to client retry behavior, but on the delegation token side. So I
don't think adding the config would add reasonable value here.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.16.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.16.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15593) UserGroupInformation TGT renewer throws NPE
[
https://issues.apache.org/jira/browse/HADOOP-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16558998#comment-16558998
]
Xiao Chen commented on HADOOP-15593:
Hi [~eyang], do you have any comments on patch 5, or are you interested in
committing this if not?
> UserGroupInformation TGT renewer throws NPE
> ---
>
> Key: HADOOP-15593
> URL: https://issues.apache.org/jira/browse/HADOOP-15593
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Blocker
> Attachments: HADOOP-15593.001.patch, HADOOP-15593.002.patch,
> HADOOP-15593.003.patch, HADOOP-15593.004.patch, HADOOP-15593.005.patch
>
>
> Found the following NPE thrown in UGI tgt renewer. The NPE was thrown within
> an exception handler so the original exception was hidden, though it's likely
> caused by expired tgt.
> {noformat}
> 18/07/02 10:30:57 ERROR util.SparkUncaughtExceptionHandler: Uncaught
> exception in thread Thread[TGT Renewer for f...@example.com,5,main]
> java.lang.NullPointerException
> at
> javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
> at
> org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:894)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Suspect it's related to [https://bugs.openjdk.java.net/browse/JDK-8154889].
> The relevant code was added in HADOOP-13590. File this jira to handle the
> exception better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15593) UserGroupInformation TGT renewer throws NPE
[
https://issues.apache.org/jira/browse/HADOOP-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556850#comment-16556850
]
Xiao Chen commented on HADOOP-15593:
+1 on patch 5 from me. Thanks all.
> UserGroupInformation TGT renewer throws NPE
> ---
>
> Key: HADOOP-15593
> URL: https://issues.apache.org/jira/browse/HADOOP-15593
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Blocker
> Attachments: HADOOP-15593.001.patch, HADOOP-15593.002.patch,
> HADOOP-15593.003.patch, HADOOP-15593.004.patch, HADOOP-15593.005.patch
>
>
> Found the following NPE thrown in UGI tgt renewer. The NPE was thrown within
> an exception handler so the original exception was hidden, though it's likely
> caused by expired tgt.
> {noformat}
> 18/07/02 10:30:57 ERROR util.SparkUncaughtExceptionHandler: Uncaught
> exception in thread Thread[TGT Renewer for f...@example.com,5,main]
> java.lang.NullPointerException
> at
> javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
> at
> org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:894)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Suspect it's related to [https://bugs.openjdk.java.net/browse/JDK-8154889].
> The relevant code was added in HADOOP-13590. File this jira to handle the
> exception better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-14445:
---
Attachment: HADOOP-14445.15.patch
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556846#comment-16556846
]
Xiao Chen commented on HADOOP-14445:
Thanks for the clarification, didn't realize it was referring to #2 You're
absolutely right. [^HADOOP-14445.15.patch] rebases on latest trunk and
addressed the comments.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.15.patch, HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556371#comment-16556371
]
Xiao Chen commented on HADOOP-14445:
Thanks for the review [~jojochuang]. Will rebase on the next patch.
bq. 1
Good catch, you're right, will update in next rev.
bq. 2
Would {{KMSUtiltestInjector}} suffice? Or what any other suggestions?
bq. 3
Could you clarify which line?
bq. 4
I feel that's the goal of release notes. The last sentence of the release note
seems clear to me that the flag should not be set arbitrarily. If we raise the
incompatible flag, we theoretically cannot backport this to 2.x and 3.x...
Open to any recommendations though.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HADOOP-14445) Delegation tokens are not shared between KMS instances
[
https://issues.apache.org/jira/browse/HADOOP-14445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16556371#comment-16556371
]
Xiao Chen edited comment on HADOOP-14445 at 7/25/18 10:51 PM:
--
Thanks for the review [~jojochuang]. Will rebase on the next patch.
1. Good catch, you're right, will update in next rev.
2. Would {{KMSUtiltestInjector}} suffice? Or any other suggestions?
3. Could you clarify which line?
bq. 4
I feel that's the goal of release notes. The last sentence of the release note
seems clear to me that the flag should not be set arbitrarily. If we raise the
incompatible flag, we theoretically cannot backport this to 2.x and 3.x...
Open to any recommendations though.
was (Author: xiaochen):
Thanks for the review [~jojochuang]. Will rebase on the next patch.
bq. 1
Good catch, you're right, will update in next rev.
bq. 2
Would {{KMSUtiltestInjector}} suffice? Or what any other suggestions?
bq. 3
Could you clarify which line?
bq. 4
I feel that's the goal of release notes. The last sentence of the release note
seems clear to me that the flag should not be set arbitrarily. If we raise the
incompatible flag, we theoretically cannot backport this to 2.x and 3.x...
Open to any recommendations though.
> Delegation tokens are not shared between KMS instances
> --
>
> Key: HADOOP-14445
> URL: https://issues.apache.org/jira/browse/HADOOP-14445
> Project: Hadoop Common
> Issue Type: Bug
> Components: kms
>Affects Versions: 2.8.0, 3.0.0-alpha1
> Environment: CDH5.7.4, Kerberized, SSL, KMS-HA, at rest encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Xiao Chen
>Priority: Major
> Attachments: HADOOP-14445-branch-2.8.002.patch,
> HADOOP-14445-branch-2.8.patch, HADOOP-14445.002.patch,
> HADOOP-14445.003.patch, HADOOP-14445.004.patch, HADOOP-14445.05.patch,
> HADOOP-14445.06.patch, HADOOP-14445.07.patch, HADOOP-14445.08.patch,
> HADOOP-14445.09.patch, HADOOP-14445.10.patch, HADOOP-14445.11.patch,
> HADOOP-14445.12.patch, HADOOP-14445.13.patch, HADOOP-14445.14.patch,
> HADOOP-14445.branch-2.000.precommit.patch,
> HADOOP-14445.branch-2.001.precommit.patch, HADOOP-14445.branch-2.01.patch,
> HADOOP-14445.branch-2.02.patch, HADOOP-14445.branch-2.03.patch,
> HADOOP-14445.branch-2.04.patch, HADOOP-14445.branch-2.05.patch,
> HADOOP-14445.branch-2.06.patch, HADOOP-14445.branch-2.8.003.patch,
> HADOOP-14445.branch-2.8.004.patch, HADOOP-14445.branch-2.8.005.patch,
> HADOOP-14445.branch-2.8.006.patch, HADOOP-14445.branch-2.8.revert.patch,
> HADOOP-14445.revert.patch
>
>
> As discovered in HADOOP-14441, KMS HA using LoadBalancingKMSClientProvider do
> not share delegation tokens. (a client uses KMS address/port as the key for
> delegation token)
> {code:title=DelegationTokenAuthenticatedURL#openConnection}
> if (!creds.getAllTokens().isEmpty()) {
> InetSocketAddress serviceAddr = new InetSocketAddress(url.getHost(),
> url.getPort());
> Text service = SecurityUtil.buildTokenService(serviceAddr);
> dToken = creds.getToken(service);
> {code}
> But KMS doc states:
> {quote}
> Delegation Tokens
> Similar to HTTP authentication, KMS uses Hadoop Authentication for delegation
> tokens too.
> Under HA, A KMS instance must verify the delegation token given by another
> KMS instance, by checking the shared secret used to sign the delegation
> token. To do this, all KMS instances must be able to retrieve the shared
> secret from ZooKeeper.
> {quote}
> We should either update the KMS documentation, or fix this code to share
> delegation tokens.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15593) UserGroupInformation TGT renewer throws NPE
[
https://issues.apache.org/jira/browse/HADOOP-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555154#comment-16555154
]
Xiao Chen commented on HADOOP-15593:
Thanks for the clarification [~eyang].
According to the JDK issue, I think we should treat null endTime the same as
tgt destroyed here.
{code}
- * @return the expiration time for this ticket's validity period.
+ * @return the expiration time for this ticket's validity period,
+ * or {@code null} if destroyed.
*/
public final java.util.Date getEndTime() {
-return (Date) endTime.clone();
+return (endTime == null) ? null : (Date) endTime.clone();
}
{code}
> UserGroupInformation TGT renewer throws NPE
> ---
>
> Key: HADOOP-15593
> URL: https://issues.apache.org/jira/browse/HADOOP-15593
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Blocker
> Attachments: HADOOP-15593.001.patch, HADOOP-15593.002.patch,
> HADOOP-15593.003.patch, HADOOP-15593.004.patch
>
>
> Found the following NPE thrown in UGI tgt renewer. The NPE was thrown within
> an exception handler so the original exception was hidden, though it's likely
> caused by expired tgt.
> {noformat}
> 18/07/02 10:30:57 ERROR util.SparkUncaughtExceptionHandler: Uncaught
> exception in thread Thread[TGT Renewer for f...@example.com,5,main]
> java.lang.NullPointerException
> at
> javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
> at
> org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:894)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Suspect it's related to [https://bugs.openjdk.java.net/browse/JDK-8154889].
> The relevant code was added in HADOOP-13590. File this jira to handle the
> exception better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-15609) Retry KMS calls when SSLHandshakeException occurs
[
https://issues.apache.org/jira/browse/HADOOP-15609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HADOOP-15609:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.2
3.0.4
3.1.1
Status: Resolved (was: Patch Available)
Committed to trunk and branch-3.[0-1].
Thanks [~knanasi] for the work here, and all for reviews / comments!
> Retry KMS calls when SSLHandshakeException occurs
> -
>
> Key: HADOOP-15609
> URL: https://issues.apache.org/jira/browse/HADOOP-15609
> Project: Hadoop Common
> Issue Type: Improvement
> Components: common, kms
>Affects Versions: 3.1.0
>Reporter: Kitti Nanasi
>Assignee: Kitti Nanasi
>Priority: Major
> Fix For: 3.1.1, 3.0.4, 3.2
>
> Attachments: HADOOP-15609.001.patch, HADOOP-15609.002.patch,
> HADOOP-15609.003.patch, HADOOP-15609.004.patch
>
>
> KMS call should retry when javax.net.ssl.SSLHandshakeException occurs and
> FailoverOnNetworkExceptionRetry policy is used.
> For example in the following stack trace, we can see that the KMS Provider's
> connection is lost, an SSLHandshakeException is thrown and the operation is
> not retried:
> {code}
> W0711 18:19:50.213472 1508 LoadBalancingKMSClientProvider.java:132] KMS
> provider at [https://example.com:16000/kms/v1/] threw an IOException:
> Java exception follows:
> javax.net.ssl.SSLHandshakeException: Remote host closed connection during
> handshake
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1002)
> at
> sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1385)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1413)
> at
> sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1397)
> at
> sun.net.www.protocol.https.HttpsClient.afterConnect(HttpsClient.java:559)
> at
> sun.net.www.protocol.https.AbstractDelegateHttpsURLConnection.connect(AbstractDelegateHttpsURLConnection.java:185)
> at
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream0(HttpURLConnection.java:1316)
> at
> sun.net.www.protocol.http.HttpURLConnection.getOutputStream(HttpURLConnection.java:1291)
> at
> sun.net.www.protocol.https.HttpsURLConnectionImpl.getOutputStream(HttpsURLConnectionImpl.java:250)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:512)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.call(KMSClientProvider.java:502)
> at
> org.apache.hadoop.crypto.key.kms.KMSClientProvider.decryptEncryptedKey(KMSClientProvider.java:791)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:288)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$5.call(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:124)
> at
> org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.decryptEncryptedKey(LoadBalancingKMSClientProvider.java:284)
> at
> org.apache.hadoop.crypto.key.KeyProviderCryptoExtension.decryptEncryptedKey(KeyProviderCryptoExtension.java:532)
> at
> org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(DFSClient.java:927)
> at
> org.apache.hadoop.hdfs.DFSClient.createWrappedInputStream(DFSClient.java:946)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:316)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$4.doCall(DistributedFileSystem.java:311)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:323)
> Caused by: java.io.EOFException: SSL peer shut down incorrectly
> at sun.security.ssl.InputRecord.read(InputRecord.java:505)
> at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)
> ... 22 more
> W0711 18:19:50.239328 1508 LoadBalancingKMSClientProvider.java:149] Aborting
> since the Request has failed with all KMS providers(depending on
> hadoop.security.kms.client.failover.max.retries=1 setting and numProviders=1)
> in the group OR the exception is not recoverable
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15593) UserGroupInformation TGT renewer throws NPE
[
https://issues.apache.org/jira/browse/HADOOP-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16555116#comment-16555116
]
Xiao Chen commented on HADOOP-15593:
If we did what's proposed in [my previous
comment|https://issues.apache.org/jira/browse/HADOOP-15593?focusedCommentId=16554585=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16554585],
the case when tgt is destroyed will be handled by the {{return}} statement.
In the rare race that the tgt gets destroyed after the code has gone after
those lines, the rest of the logic including the nextRefresh part you pointed
out does not depend on tgt anymore (it only depends on the local var
{{tgtEndTime}}). We should be fine just it retry one more time and return on
the tgt null check next time it enters the while loop.
So we don't need to change {{now > nextRefresh}} part of code. Did I miss
anything?
> UserGroupInformation TGT renewer throws NPE
> ---
>
> Key: HADOOP-15593
> URL: https://issues.apache.org/jira/browse/HADOOP-15593
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Blocker
> Attachments: HADOOP-15593.001.patch, HADOOP-15593.002.patch,
> HADOOP-15593.003.patch, HADOOP-15593.004.patch
>
>
> Found the following NPE thrown in UGI tgt renewer. The NPE was thrown within
> an exception handler so the original exception was hidden, though it's likely
> caused by expired tgt.
> {noformat}
> 18/07/02 10:30:57 ERROR util.SparkUncaughtExceptionHandler: Uncaught
> exception in thread Thread[TGT Renewer for f...@example.com,5,main]
> java.lang.NullPointerException
> at
> javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
> at
> org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:894)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Suspect it's related to [https://bugs.openjdk.java.net/browse/JDK-8154889].
> The relevant code was added in HADOOP-13590. File this jira to handle the
> exception better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Commented] (HADOOP-15593) UserGroupInformation TGT renewer throws NPE
[
https://issues.apache.org/jira/browse/HADOOP-15593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16554680#comment-16554680
]
Xiao Chen commented on HADOOP-15593:
It makes sense for the best-effort retries in general, so the renewal thread
doesn't abort prematurely due to intermittent failures.
But could you clarify a little more? If a tgt is destroyed, how can it be
renewed?
Looks to me KDC outage would result in relogin failure and possibly getTGT()
being null without an exception, after which the current code just does a null
check on tgt and return without retries. IMO we should be consistent with it
and just return.
I don't feel strongly that having the last try as patch 4 is a big problem, but
it's not clear under which scenario this could possibly succeed.
> UserGroupInformation TGT renewer throws NPE
> ---
>
> Key: HADOOP-15593
> URL: https://issues.apache.org/jira/browse/HADOOP-15593
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 3.0.0
>Reporter: Wei-Chiu Chuang
>Assignee: Gabor Bota
>Priority: Blocker
> Attachments: HADOOP-15593.001.patch, HADOOP-15593.002.patch,
> HADOOP-15593.003.patch, HADOOP-15593.004.patch
>
>
> Found the following NPE thrown in UGI tgt renewer. The NPE was thrown within
> an exception handler so the original exception was hidden, though it's likely
> caused by expired tgt.
> {noformat}
> 18/07/02 10:30:57 ERROR util.SparkUncaughtExceptionHandler: Uncaught
> exception in thread Thread[TGT Renewer for f...@example.com,5,main]
> java.lang.NullPointerException
> at
> javax.security.auth.kerberos.KerberosTicket.getEndTime(KerberosTicket.java:482)
> at
> org.apache.hadoop.security.UserGroupInformation$1.run(UserGroupInformation.java:894)
> at java.lang.Thread.run(Thread.java:748){noformat}
> Suspect it's related to [https://bugs.openjdk.java.net/browse/JDK-8154889].
> The relevant code was added in HADOOP-13590. File this jira to handle the
> exception better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
