[jira] [Created] (HADOOP-8560) Change TestFTPFileSystem to use non-SNAPSHOT dependencies
Luke Lu created HADOOP-8560: --- Summary: Change TestFTPFileSystem to use non-SNAPSHOT dependencies Key: HADOOP-8560 URL: https://issues.apache.org/jira/browse/HADOOP-8560 Project: Hadoop Common Issue Type: Improvement Components: fs Affects Versions: 1.0.3 Reporter: Luke Lu Assignee: Yu Gao Priority: Minor It would good if the stable hadoop release don't depend on SNAPSHOT ftpserver artifacts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8561) Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes
Luke Lu created HADOOP-8561: --- Summary: Introduce HADOOP_PROXY_USER for secure impersonation in child hadoop client processes Key: HADOOP-8561 URL: https://issues.apache.org/jira/browse/HADOOP-8561 Project: Hadoop Common Issue Type: Improvement Components: security Reporter: Luke Lu Assignee: Yu Gao To solve the problem for an authenticated user to type hadoop shell commands in a web console, we can introduce an HADOOP_PROXY_USER environment variable to allow proper impersonation in the child hadoop client processes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8559) PMML Support in Hadoop Cluster
[ https://issues.apache.org/jira/browse/HADOOP-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Joseph Evans resolved HADOOP-8559. - Resolution: Won't Fix PMML Support in Hadoop Cluster -- Key: HADOOP-8559 URL: https://issues.apache.org/jira/browse/HADOOP-8559 Project: Hadoop Common Issue Type: New Feature Components: util Environment: Software Platform Reporter: Duraimurugan Priority: Minor Labels: newbie Would like to request a support for PMML. With that once the predictive models are built and provided in PMML format, we should be able to import into hadoop cluster for scoring. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8530) Potential deadlock in IPC
[ https://issues.apache.org/jira/browse/HADOOP-8530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved HADOOP-8530. --- Resolution: Duplicate Fixed by HADOOP-8533. Potential deadlock in IPC - Key: HADOOP-8530 URL: https://issues.apache.org/jira/browse/HADOOP-8530 Project: Hadoop Common Issue Type: Bug Components: ipc Affects Versions: 1.0.3, 2.0.0-alpha Reporter: Tom White Attachments: 1_jcarder_result_0.dot.png This cycle (see attached image, and explanation here: http://www.jcarder.org/manual.html#analysis) was found with jcarder in branch-1 (affects trunk too). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-8414) Address problems related to localhost resolving to 127.0.0.1 on Windows
[ https://issues.apache.org/jira/browse/HADOOP-8414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sanjay Radia resolved HADOOP-8414. -- Resolution: Fixed Thanks Ivan, committed to branch-1 windows. Address problems related to localhost resolving to 127.0.0.1 on Windows --- Key: HADOOP-8414 URL: https://issues.apache.org/jira/browse/HADOOP-8414 Project: Hadoop Common Issue Type: Bug Components: fs, test Affects Versions: 1.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Attachments: HADOOP-8414-branch-1-win(2).patch, HADOOP-8414-branch-1-win(3).patch, HADOOP-8414-branch-1-win.patch, HADOOP-8414-branch-1-win.patch Localhost resolves to 127.0.0.1 on Windows and that causes the following tests to fail: - TestHarFileSystem - TestCLI - TestSaslRPC This Jira tracks fixing these tests and other possible places that have similar issue. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8562) Enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments
Bikas Saha created HADOOP-8562: -- Summary: Enhancements to Hadoop for Windows Server and Windows Azure development and runtime environments Key: HADOOP-8562 URL: https://issues.apache.org/jira/browse/HADOOP-8562 Project: Hadoop Common Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Bikas Saha Assignee: Bikas Saha This JIRA tracks the work that needs to be done on trunk to enable Hadoop to run on Windows Server and Azure environments. This incorporates porting relevant work from the similar effort on branch 1 tracked via HADOOP-8079. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8563) don't package hadoop-pipes examples/bin
Colin Patrick McCabe created HADOOP-8563: Summary: don't package hadoop-pipes examples/bin Key: HADOOP-8563 URL: https://issues.apache.org/jira/browse/HADOOP-8563 Project: Hadoop Common Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Let's not package hadoop-pipes examples/bin -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8564) Create a Windows native InputStream class to address datanode concurrent reading and writing issue
Chuan Liu created HADOOP-8564:
-
Summary: Create a Windows native InputStream class to address
datanode concurrent reading and writing issue
Key: HADOOP-8564
URL: https://issues.apache.org/jira/browse/HADOOP-8564
Project: Hadoop Common
Issue Type: Bug
Components: io
Affects Versions: 1-win
Reporter: Chuan Liu
Assignee: Chuan Liu
HDFS files are made up of blocks. First, let’s look at writing. When the data
is written to datanode, an active or temporary file is created to receive
packets. After the last packet for the block is received, we will finalize the
block. One step during finalization is to rename the block file to a new
directory. The relevant code can be found via the call sequence:
FSDataSet.finalizeBlockInternal - FSDir.addBlock.
{code}
if ( ! metaData.renameTo( newmeta ) ||
! src.renameTo( dest ) ) {
throw new IOException( could not move files for + b +
from tmp to +
dest.getAbsolutePath() );
}
{code}
Let’s then switch to reading. On HDFS, it is expected the client can also read
these unfinished blocks. So when the read calls from client reach datanode, the
datanode will open an input stream on the unfinished block file.
The problem comes in when the file is opened for reading while the datanode
receives last packet from client and try to rename the finished block file.
This operation will succeed on Linux, but not on Windows . The behavior can be
modified on Windows to open the file with FILE_SHARE_DELETE flag on, i.e.
sharing the delete (including renaming) permission with other processes while
opening the file. There is also a Java bug ([id
6357433|http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6357433]) reported a
while back on this. However, since this behavior exists for Java on Windows
since JDK 1.0, the Java developers do not want to break the backward
compatibility on this behavior. Instead, a new file system API is proposed in
JDK 7.
As outlined in the [Java forum|http://www.java.net/node/645421] by the Java
developer (kbr), there are three ways to fix the problem:
# Use different mechanism in the application in dealing with files.
# Create a new implementation of InputStream abstract class using Windows
native code.
# Patch JDK with a private patch that alters FileInputStream behavior.
For the third option, it cannot fix the problem for users using Oracle JDK.
We discussed some options for the first approach. For example one option is to
use two phase renaming, i.e. first hardlink; then remove the old hardlink when
read is finished. This option was thought to be rather pervasive. Another
option discussed is to change the HDFS behavior on Windows by not allowing
client reading unfinished blocks. However this behavior change is thought to be
problematic and may affect other application build on top of HDFS.
For all the reasons discussed above, we will use the second approach to address
the problem.
If there are better options to fix the problem, we would also like to hear
about them.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8565) AuthenticationFilter#doFilter warns unconditionally when using SPNEGO
Eli Collins created HADOOP-8565:
---
Summary: AuthenticationFilter#doFilter warns unconditionally when
using SPNEGO
Key: HADOOP-8565
URL: https://issues.apache.org/jira/browse/HADOOP-8565
Project: Hadoop Common
Issue Type: Improvement
Affects Versions: 2.0.0-alpha, 1.0.3
Reporter: Eli Collins
The following code in AuthenticationFilter#doFilter throws
AuthenticationException (and warns) unconditionally because
KerberosAuthenticator#authenticate returns null if SPNEGO is used.
{code}
token = authHandler.authenticate(httpRequest, httpResponse);
...
if (token != null) { ... } else {
throw new AuthenticationException
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HADOOP-6527) UserGroupInformation::createUserForTesting clobbers already defined group mappings
[
https://issues.apache.org/jira/browse/HADOOP-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE resolved HADOOP-6527.
Resolution: Fixed
Fix Version/s: 1-win
1.1.0
I have committed this. Thanks, Ivan!
UserGroupInformation::createUserForTesting clobbers already defined group
mappings
--
Key: HADOOP-6527
URL: https://issues.apache.org/jira/browse/HADOOP-6527
Project: Hadoop Common
Issue Type: Bug
Components: security
Reporter: Jakob Homan
Assignee: Ivan Mitic
Fix For: 1.1.0, 1-win
Attachments: HADOOP-6527-branch-1-win_UGI_fix(2).patch,
HADOOP-6527-branch-1-win_UGI_fix.patch
In UserGroupInformation::createUserForTesting the follow code creates a new
groups instance, obliterating any groups that have been previously defined in
the static groups field.
{code}if (!(groups instanceof TestingGroups)) {
groups = new TestingGroups();
}
{code}
This becomes a problem in tests that start a Mini{DFS,MR}Cluster and then
create a testing user. The user that started the user (generally the real
user running the test) immediately has their groups wiped out and is
prevented from accessing files/folders/queues they should be able to. Before
the UserGroupInformation.createRemoteUserForTesting, calls to userA.getGroups
may return {a, b, c} and immediately after the new fake user is
created, the same call will return an empty array.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8566) AvroReflectSerializer.accept(Class) throws a NPE if the class has no package (primitive types and arrays)
Alejandro Abdelnur created HADOOP-8566: -- Summary: AvroReflectSerializer.accept(Class) throws a NPE if the class has no package (primitive types and arrays) Key: HADOOP-8566 URL: https://issues.apache.org/jira/browse/HADOOP-8566 Project: Hadoop Common Issue Type: Bug Components: io Affects Versions: 2.0.0-alpha Reporter: Alejandro Abdelnur Fix For: 2.0.1-alpha the accept() method should consider the case where the class getPackage() returns NULL. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HADOOP-8567) Backport conf servlet with dump running configuration to branch 1.x
Junping Du created HADOOP-8567: -- Summary: Backport conf servlet with dump running configuration to branch 1.x Key: HADOOP-8567 URL: https://issues.apache.org/jira/browse/HADOOP-8567 Project: Hadoop Common Issue Type: New Feature Components: conf Affects Versions: 1.0.3 Reporter: Junping Du Assignee: Junping Du Fix For: 0.21.1, 2.0.1-alpha HADOOP-6408 provide conf servlet that can dump running configuration which great helps admin to trouble shooting the configuration issue. However, that patch works on branch after 0.21 only and should be backport to branch 1.x. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
No mapred-site.xml in the hadoop-0.23.3 distribution
Hi, I downloaded the Hadoop-0.23.3 source and tweaked a few classes and when I built the binary distribution and untar'd it .I don't see the mapred-site.xml file in the /etc/hadoop directory. But by the details given on how to run the Hadoop-0.23.3 the mapred-site.xml needs to be configured right? So I was just wondering if we are supposed to create the mapred-site.xml , or it doesn't exist at all? Thanks -- --With Regards Pavan Kulkarni
[jira] [Created] (HADOOP-8568) DNS#reverseDns fails on IPv6 addresses
Eli Collins created HADOOP-8568:
---
Summary: DNS#reverseDns fails on IPv6 addresses
Key: HADOOP-8568
URL: https://issues.apache.org/jira/browse/HADOOP-8568
Project: Hadoop Common
Issue Type: Bug
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Eli Collins
DNS#reverseDns assumes hostIp is a v4 address (4 parts separated by dots),
blows up if given a v6 address:
{noformat}
Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
at org.apache.hadoop.net.DNS.reverseDns(DNS.java:79)
at org.apache.hadoop.net.DNS.getHosts(DNS.java:237)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:340)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:358)
at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:337)
at org.apache.hadoop.hbase.master.HMaster.init(HMaster.java:235)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at
org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:1649)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
