Hi Kevin,
I tried your Latter Code and continuously getting "HTTP/1.1 401
Unauthorized"
Following is my Topology file:
<topology>
<gateway>
<provider>
<role>authentication</role>
<name>ShiroProvider</name>
<enabled>true</enabled>
<param>
<name>sessionTimeout</name>
<value>30</value>
</param>
<param>
<name>main.ldapRealm</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm</value>
</param>
<param>
<name>main.ldapContextFactory</name>
<value>org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.contextFactory</name>
<value>$ldapContextFactory</value>
</param>
<param>
<name>main.ldapRealm.userDnTemplate</name>
<value>cn={0},dc=platalytics,dc=com</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.url</name>
<value>ldap://localhost</value>
</param>
<param>
<name>main.ldapRealm.contextFactory.authenticationMechanism</name>
<value>simple</value>
</param>
<param>
<name>urls./**</name>
<value>authcBasic</value>
</param>
</provider>
<provider>
<role>authorization</role>
<name>AclsAuthz</name>
<enabled>true</enabled>
<param>
<name>knox.acl</name>
<value>admin;*;*</value>
</param>
</provider>
<provider>
<role>identity-assertion</role>
<name>Default</name>
<enabled>true</enabled>
</provider>
<provider>
<role>hostmap</role>
<name>static</name>
<enabled>true</enabled>
<param><name>localhost</name><value>127.0.0.1</value></param>
</provider>
</gateway>
<service>
<role>NAMENODE</role>
<url>hdfs://localhost:8020</url>
</service>
<service>
<role>JOBTRACKER</role>
<url>rpc://localhost:8050</url>
</service>
<service>
<role>RESOURCEMANAGER</role>
<url>http://red3:8088/ws</url>
</service>
<service>
<role>WEBHDFS</role>
<url>http://localhost:50070/webhdfs</url>
</service>
<service>
<role>WEBHCAT</role>
<url>http://webcat-host:50111/templeton</url>
</service>
<service>
<role>WEBHBASE</role>
<url>http://webhbase-host :60080</url>
</service>
<service>
<role>HIVE</role>
<url>http://hive-host :10001/cliservice</url>
</service>
</topology>
Can you please see what i'm missing?
On Wed, Jul 8, 2015 at 6:52 PM, Kevin Minder <[email protected]>
wrote:
> Ok, if you want the simplest thing that will work try the code below.
> This time I’ve tested it. However, keep in mind this code removes much of
> the benefit of SSL between the client and Knox due to the use
> of TrustSelfSignedStrategy and NoopHostnameVerifier. If you are using this
> in production there are a few different routes to go. For example if you
> are using CA signed certs much of the SSL setup code below isn’t even
> required. Can you provide more context about what you are actually trying
> to accomplish?
>
>
> import org.apache.http.HttpEntity;
> import org.apache.http.auth.AuthScope;
> import org.apache.http.auth.UsernamePasswordCredentials;
> import org.apache.http.client.CredentialsProvider;
> import org.apache.http.client.methods.CloseableHttpResponse;
> import org.apache.http.client.methods.HttpGet;
> import org.apache.http.client.protocol.HttpClientContext;
> import org.apache.http.conn.ssl.NoopHostnameVerifier;
> import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
> import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
> import org.apache.http.impl.client.BasicCredentialsProvider;
> import org.apache.http.impl.client.CloseableHttpClient;
> import org.apache.http.impl.client.HttpClients;
> import org.apache.http.ssl.SSLContextBuilder;
> import org.apache.http.util.EntityUtils;
>
> import javax.net.ssl.SSLContext;
>
> public class HttpClientSslNoVerifySslSample {
>
> public static void main( String[] args ) throws Exception {
>
> SSLContext sslContext = SSLContextBuilder.create()
> .loadTrustMaterial( new TrustSelfSignedStrategy() ) // *** Trust
> self signed certs. ***
> .build();
> SSLConnectionSocketFactory sslFactory = new
> SSLConnectionSocketFactory( sslContext );
>
> CloseableHttpClient client = HttpClients.custom()
> .setSSLSocketFactory( sslFactory )
> .setSSLHostnameVerifier( new NoopHostnameVerifier() ) // *** Allow
> all host names. ***
> .build();
>
> HttpClientContext cliContext = HttpClientContext.create();
> CredentialsProvider credentialsProvider = new
> BasicCredentialsProvider();
> credentialsProvider.setCredentials(
> new AuthScope( AuthScope.ANY_HOST, AuthScope.ANY_PORT ),
> new UsernamePasswordCredentials( "guest", "guest-password" ) );
> cliContext.setCredentialsProvider( credentialsProvider );
>
> HttpGet method = new HttpGet( "
> https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY"; );
> CloseableHttpResponse response = client.execute( method, cliContext );
> HttpEntity entity = response.getEntity();
> System.out.println( EntityUtils.toString( entity ) );
>
> response.close();
> client.close();
> }
>
> }
>
>
> From: Hafiz Mujadid <[email protected]>
> Reply-To: "[email protected]" <[email protected]>
> Date: Wednesday, July 8, 2015 at 4:53 AM
>
> To: "[email protected]" <[email protected]>
> Subject: Re: Apache Knox Web API
>
> Hi Kevin!
>
> I tried this code and got following exception
>
> Error: keytool error: java.io.IOException: Keystore was tampered with, or
> password was incorrect
> java.io.IOException: Keystore was tampered with, or password was incorrect
> at sun.security.provider.JavaKeyStore.engineLoad(JavaKeyStore.java:772)
> at sun.security.provider.JavaKeyStore$JKS.engineLoad(JavaKeyStore.java:55)
>
>
> I regenerated password for key store and replaced
> *trustStore.load(stream, "wrong".toCharArray())*
> to
> *trustStore.load(stream, "changeit".toCharArray())*
>
> but still it's not working.
>
> On Wed, Jul 8, 2015 at 1:46 AM, Kevin Minder <[email protected]
> > wrote:
>
>> Take a look at this below. This is a bit of a mod of an existing
>> sample I had laying around so don’t take it as tested.
>>
>> import org.apache.http.HttpEntity;
>> import org.apache.http.auth.AuthScope;
>> import org.apache.http.auth.UsernamePasswordCredentials;
>> import org.apache.http.client.CredentialsProvider;
>> import org.apache.http.client.methods.CloseableHttpResponse;
>> import org.apache.http.client.methods.HttpGet;
>> import org.apache.http.client.protocol.HttpClientContext;
>> import org.apache.http.conn.ssl.AllowAllHostnameVerifier;
>> import org.apache.http.conn.ssl.SSLConnectionSocketFactory;
>> import org.apache.http.conn.ssl.SSLContexts;
>> import org.apache.http.conn.ssl.TrustSelfSignedStrategy;
>> import org.apache.http.impl.client.BasicCredentialsProvider;
>> import org.apache.http.impl.client.CloseableHttpClient;
>> import org.apache.http.impl.client.HttpClients;
>> import org.apache.http.util.EntityUtils;
>>
>> import javax.net.ssl.SSLContext;
>> import java.io.File;
>> import java.io.FileInputStream;
>> import java.security.KeyStore;
>>
>> public class HttpClientSslTest {
>>
>> public static void main( String[] args ) throws Exception {
>>
>> KeyStore trustStore = KeyStore.getInstance( KeyStore.getDefaultType() );
>> FileInputStream stream = new FileInputStream( new File( "gateway.jks" )
>> );
>> trustStore.load( stream, "wrong".toCharArray() );
>> stream.close();
>>
>> SSLContext sslContext = SSLContexts.custom()
>> .loadTrustMaterial( trustStore, *new TrustSelfSignedStrategy()* ) //
>> *** Trust self signed certs. ***
>> .build();
>> SSLConnectionSocketFactory sslFactory = new SSLConnectionSocketFactory(
>> sslContext );
>>
>> CloseableHttpClient client = HttpClients.custom()
>> .setSSLSocketFactory( sslFactory )
>> .setHostnameVerifier( *new AllowAllHostnameVerifier()* ) // ***
>> Trust all host names. ***
>> .build();
>>
>> HttpClientContext cliContext = HttpClientContext.create();
>> CredentialsProvider credentialsProvider = new BasicCredentialsProvider();
>> credentialsProvider.setCredentials(
>> new AuthScope( AuthScope.ANY_HOST, AuthScope.ANY_PORT ),
>> new UsernamePasswordCredentials( "guest", "guest-password" ) );
>> cliContext.setCredentialsProvider( credentialsProvider );
>>
>> HttpGet method = new HttpGet(
>> "https://localhost:8443/gateway/sandbox/webhdfs/v1?op=GETHOMEDIRECTORY"; );
>> CloseableHttpResponse response = client.execute( method, cliContext );
>> HttpEntity entity = response.getEntity();
>> System.out.println( EntityUtils.toString( entity ) );
>>
>> response.close();
>> client.close();
>> }
>>
>> }
>>
>>
>> From: Hafiz Mujadid <[email protected]>
>> Reply-To: "[email protected]" <[email protected]>
>> Date: Tuesday, July 7, 2015 at 4:05 PM
>> To: "[email protected]" <[email protected]>
>> Subject: Re: Apache Knox Web API
>>
>> Hi larry!
>>
>> As suggested by you, I tried to use knox rest api using Apache
>> HttpClient
>>
>> here is my code
>>
>> val provider = new BasicCredentialsProvider()
>> val credentials = new UsernamePasswordCredentials("admin", "12345")
>> provider.setCredentials(AuthScope.ANY, credentials)
>> val client =
>> HttpClientBuilder.create().setDefaultCredentialsProvider(provider) .build()
>> val response = client.execute(new HttpGet("
>> https://localhost:8443/gateway/sample/webhdfs/v1?op=LISTSTATUS";))
>> val statusCode = response.getStatusLine.getStatusCode
>> val input = response.getEntity().getContent()
>> if (statusCode == HttpStatus.SC_OK)
>> println("ok")
>>
>>
>> but I am getting following SSL related exception.
>>
>>
>> Exception in thread "main" javax.net.ssl.SSLHandshakeException:
>> sun.security.validator.ValidatorException: PKIX path building failed:
>> sun.security.provider.certpath.SunCertPathBuilderException: unable to find
>> valid certification path to requested target
>> at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
>> at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1904)
>>
>> Any suggestion?
>>
>>
>> On Mon, Jul 6, 2015 at 10:34 PM, Hafiz Mujadid <[email protected]>
>> wrote:
>>
>>> thanks for your help .:)
>>>
>>> On Mon, Jul 6, 2015 at 10:05 PM, larry mccay <[email protected]>
>>> wrote:
>>>
>>>> As I mentioned, you can dig into the source of the gateway-shell
>>>> classes - which are used when scripting with groovy.
>>>> Here is a link to an hdfs Get request:
>>>> https://github.com/apache/knox/blob/master/gateway-shell/src/main/java/org/apache/hadoop/gateway/shell/hdfs/Get.java#L32
>>>>
>>>> Going to the HttpClient level is like going to bare metal - it
>>>> provides you greatest level of control but you will need to build
>>>> abstractions around its use in order to avoid lots of redundant boilerplate
>>>> code. Which is why we have provided such classes for the scripting.
>>>>
>>>> You can also look at the DefaultDispatch code as an example - it is a
>>>> bit more complicated since it covers more general usecases but you may
>>>> glean some insights from it.
>>>>
>>>> Otherwise, google for examples of "Apache HttpClient REST basic
>>>> authentication" and see what you find.
>>>>
>>>> Hope this is useful for you!
>>>>
>>>>
>>>> On Sun, Jul 5, 2015 at 11:40 AM, Hafiz Mujadid <
>>>> [email protected]> wrote:
>>>>
>>>>> Hi Larry!
>>>>>
>>>>> Can you provide the link to samples using httclient on github etc.?
>>>>>
>>>>> Thanks
>>>>>
>>>>> On Sat, Jul 4, 2015 at 9:40 PM, larry mccay <[email protected]>
>>>>> wrote:
>>>>>
>>>>>> Then you will want to consider the Client library from the first
>>>>>> link.
>>>>>> You can look in the {GATEWAY_HOME}/samples directory for examples of
>>>>>> it's use.
>>>>>> The groovy scripts are a great way to do it or you can use the
>>>>>> underlying java classes that groovy uses.
>>>>>> The latter will require you to dig into the source a bit more to see
>>>>>> how to use them.
>>>>>>
>>>>>> You can also use Apache HttpClient and there are samples of that as
>>>>>> well.
>>>>>>
>>>>>> On Sat, Jul 4, 2015 at 12:04 PM, Aneela Saleem <
>>>>>> [email protected]> wrote:
>>>>>>
>>>>>>> Thanks Larry.
>>>>>>>
>>>>>>> Actually I need some client API like java so that I authenticate
>>>>>>> / authorize my users programmatically through Knox.
>>>>>>>
>>>>>>> On Sat, Jul 4, 2015 at 8:50 PM, larry mccay <[email protected]>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Aneela -
>>>>>>>>
>>>>>>>> I assume that you mean that you would like to add support for a
>>>>>>>> Hadoop API that Knox currently lacks.
>>>>>>>> My suggestion is that you find one that your organization or your
>>>>>>>> personal interests require.
>>>>>>>>
>>>>>>>> There are lots of Jira's filed for bug fixes and other
>>>>>>>> features/enhancements as well.
>>>>>>>>
>>>>>>>> Feel free to start a discussion regarding any contribution that
>>>>>>>> you would like to make.
>>>>>>>>
>>>>>>>> As far as the links that you referenced:
>>>>>>>>
>>>>>>>> 1. The first is a client library for scripting interactions with
>>>>>>>> Hadoop services through Knox - there are some really interesting and
>>>>>>>> powerful capabilities there.
>>>>>>>> 2. The second is actually pointing to a section the dev guide that
>>>>>>>> needs to be completed. We have what we call Gateway Services in the
>>>>>>>> kernel
>>>>>>>> of the Knox server that provide implementations for core server
>>>>>>>> interfaces
>>>>>>>> - crypto, SSL, credential aliasing, etc. I don't think that you want to
>>>>>>>> work in that space. If you want to work on adding new API support for
>>>>>>>> services then you should refer to the Services section -
>>>>>>>> https://knox.apache.org/books/knox-0-6-0/dev-guide.html#Services.
>>>>>>>>
>>>>>>>> Note that the link that I provided above is for the 0.6.0 dev
>>>>>>>> guide. There is a new configuration driven way to add API support to
>>>>>>>> Knox
>>>>>>>> that was added in the 0.6.0 release.
>>>>>>>>
>>>>>>>> Thanks for your interest in contributing to Apache Knox!
>>>>>>>>
>>>>>>>> --larry
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Jul 4, 2015 at 10:56 AM, Aneela Saleem <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hi Everyone,
>>>>>>>>>
>>>>>>>>> I'm going to start development for Hadoop security through
>>>>>>>>> Apache Knox. Can anyone please suggest me some good API for Knox.
>>>>>>>>>
>>>>>>>>> So far i have found following:
>>>>>>>>>
>>>>>>>>> https://cwiki.apache.org/confluence/display/KNOX/Client+Usage
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://knox.apache.org/books/knox-0-5-0/dev-guide.html#Gateway+Services
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Regards: HAFIZ MUJADID
>>>>>
>>>>
>>>>
>>>
>>>
>>> --
>>> Regards: HAFIZ MUJADID
>>>
>>
>>
>>
>> --
>> Regards: HAFIZ MUJADID
>>
>
>
>
> --
> Regards: HAFIZ MUJADID
>
