Re: ldaps and 2 domain controller
> > I tried a load balancer vip before the 2 domain controllers but knox > doesn't accept it. What error did you get? Usually load balancer should work. Kevin Risden Kevin Risden On Tue, Mar 17, 2020 at 6:55 AM Sandeep Moré wrote: > Hello, > So, you want to configure Knox for both the ldap hosts is it? > Perhaps, configuring Knox with multiple LDAP realms [1] might help. > > Best, > Sandeep > > > [1] > https://cwiki.apache.org/confluence/display/KNOX/2017/03/01/Apache+Knox+using+multiple+LDAP+Realms > > > > On Tue, Mar 17, 2020 at 3:47 AM wrote: > >> Hi! >> >> I want to use ldaps to a windows domain. The domain is built with to >> domain controller for high availability reasons and ech host has a >> certificate. The domain 4711.company.com for example doesn't have a >> certificate. >> >> So knox works with one host in the ldap url only. Is there a way to >> configure both hosts ? I tried a load balancer vip before the 2 domain >> controllers but knox doesn't accept it. >> >> >> Best Regards >> >> >> Markus >> >> >> >> Fiducia & GAD IT AG | *www.fiduciagad.de* <http://www.fiduciagad.de/> >> AG Frankfurt a. M. HRB 102381 | Sitz der Gesellschaft: Frankfurt a. M. | >> USt-IdNr. DE 143582320 >> Vorstand: Martin Beyer (Vorstandssprecher), Birgit Frohnhoff, Jörg Staff >> Vorsitzender des Aufsichtsrats: Jürgen Brinkmann >> >>
Re: Is it possible to log Hive queries in Knox?
No not really. In general, Knox deals in HTTP requests and doesn't know what is in those requests. There is no parsing in Knox of the actual request data. A request comes to Knox an then is based through to the backend. In the case of Hive JDBC over HTTP, a single "action" is typically multiple HTTP requests and aren't correlated easily at Knox. Some examples of HTTP requests; * Open connection * Create statement * Execute statement * Get resultset (do this for N number of batches of results) * Close statement * Close connection All of the above are separate HTTP requests that get passed down to HiveServer2. Knox doesn't know what the requests mean other than they need to go to HS2. If you are looking for audit for actions taken, you need to get that from the backend service that knows what action was taken. Kevin Risden On Mon, Feb 3, 2020 at 9:15 AM Yunus Durmuş wrote: > Hi everyone, > > I would like to log every Hive query that runs via Knox. The audit > functionality shows that there is some activity by user X, but it does not > show the action itself. > When a security incident occurs, I should be able to figure out who > accesses what and when. > > cheers > yunus > >
Re: Disabling compression
I was following up on this today and saw that the Hue issue linked was closed. I was able to figure out that this was happening due to consumption of http > post input stream before doPost in Knox. So, Knox itself was sending empty > body. Is there an action item here? Knox uses a replayBuffer to ensure that post messages can typically be replayed. There is also the possibility of using something like "Expect: 100-Continue" to avoid the replay buffer. Kevin Risden On Mon, Dec 30, 2019 at 6:09 PM Aj J wrote: > Thanks for looking into this Kevin. > > For API details and logs, please refer this- > https://github.com/cloudera/hue/issues/1019#issuecomment-569820528 > > I added couple of log statements in the hue notebook APIs and can see the > payload for POST missing. > > > On Mon, Dec 30, 2019 at 1:53 PM Kevin Risden wrote: > >> H this sounds suspicious like something else is going on. Knox and >> Hue have been used pretty heavily recently and I don't think there have >> been issues w/ any specific types of requests. >> >> From the Hue logs I can see that request body for almost all POST >>> requests are missing. >>> >> >> For most POST requests, you typically won't see any of the data in the >> server logs unless you are like debug level. The idea is that the body is >> large and doesn't need to be logged by default. >> >> Is this for a specific API call? Is this for certain sizes of POST >> requests? >> >> What is the issue that was hit to cause you to look into the logs >> originally? Maybe there is something else happening before you got to this >> line of questioning. >> >> It would be good to try to narrow this down a bit further. Hue should be >> capable of handling gzipped requests. >> >> Kevin Risden >> >> >> On Fri, Dec 27, 2019 at 7:54 PM Aj J wrote: >> >>> I'm facing an issue similar to one mentioned here- >>> https://www.mail-archive.com/user@knox.apache.org/msg00542.html >>> >>> I have a kerberized environment with Knox receiving all the requests, >>> doing authN and forwarding it to Hue. >>> >>> From the Hue logs I can see that request body for almost all POST >>> requests are missing. >>> I see that in the discussion on the mailing thread above, the reporter >>> resolved this issue by putting nginx to do decompress requests. >>> >>> I was wondering if knox has made any changes to resolve this. >>> >>> Any workarounds/ suggestions are welcome. >>> I would like to avoid putting another nginx server all together. >>> >>> I'm planning on investigating further on the dispatch side to see what >>> can be done. >>> >>> Thanks, >>> Ajay >>> >>>
Re: Disabling compression
H this sounds suspicious like something else is going on. Knox and Hue have been used pretty heavily recently and I don't think there have been issues w/ any specific types of requests. >From the Hue logs I can see that request body for almost all POST requests > are missing. > For most POST requests, you typically won't see any of the data in the server logs unless you are like debug level. The idea is that the body is large and doesn't need to be logged by default. Is this for a specific API call? Is this for certain sizes of POST requests? What is the issue that was hit to cause you to look into the logs originally? Maybe there is something else happening before you got to this line of questioning. It would be good to try to narrow this down a bit further. Hue should be capable of handling gzipped requests. Kevin Risden On Fri, Dec 27, 2019 at 7:54 PM Aj J wrote: > I'm facing an issue similar to one mentioned here- > https://www.mail-archive.com/user@knox.apache.org/msg00542.html > > I have a kerberized environment with Knox receiving all the requests, > doing authN and forwarding it to Hue. > > From the Hue logs I can see that request body for almost all POST requests > are missing. > I see that in the discussion on the mailing thread above, the reporter > resolved this issue by putting nginx to do decompress requests. > > I was wondering if knox has made any changes to resolve this. > > Any workarounds/ suggestions are welcome. > I would like to avoid putting another nginx server all together. > > I'm planning on investigating further on the dispatch side to see what can > be done. > > Thanks, > Ajay > >
Re: Switching user going from KNOX to WebHDFS
> > If i am to do an hdfs query, all i need to do is to set HADOOP_USER_NAME > to 'hdfs' then everything works nicely. This means that you aren't using Kerberos just regular simple auth for your cluster. This is true until we get to knox. We still communicate with Knox using a > root and an admin password. I believe by default, this user's identity is > used to call webhdfs? > The user identity is asserted by Knox against the backend service. So Knox is configured for authentication that username is asserted to the backend. So however you are doing authentication in Knox needs to be configured. This is usually LDAP out of the box but can be configured with different authentication providers like PAM. Kevin Risden On Mon, Nov 18, 2019 at 2:37 PM jeff saremi wrote: > I'm not sure how to phrase this question and also I don't have any > experience in these two technologies > > Here's the deal: We are switching from running hadoop and related > technologies from under root to a non-root user > > So far we have managed to successfully change our namenodes and datanodes > such that the process is running under a user named 'hdfs'. > > If i am to do an hdfs query, all i need to do is to set HADOOP_USER_NAME > to 'hdfs' then everything works nicely. > > This is true until we get to knox. We still communicate with Knox using a > root and an admin password. I believe by default, this user's identity is > used to call webhdfs? > > We need to change this behavior. Looking for some pointers on what the > changes would be. > > thanks > Jeff >
Re: [DISCUSS] - Integrating Knox with Swagger
I am definitely a fan of avoiding the disconnect we have between documentation and code. The Admin API would be a great place to clean that up and Swagger seems like a reasonable tool to do so. Kevin Risden On Wed, Nov 13, 2019 at 9:59 PM Sandeep Moré wrote: > Agreed +1 ! > > On Wed, Nov 13, 2019 at 4:05 PM larry mccay wrote: > >> + dev@... >> >> Thank you for the idea! >> >> Yes, I am familiar with Swagger and that would be huge for our current >> APIs and others that may come along. >> I think the effort to add a swagger filter or the like will be only one >> part of the larger effort of how it integrates into Knox, the site, the >> Admin UI, the docs, etc. >> >> I think that a KIP detailing the full vision would make a lot of sense. >> >> +1 on moving forward with a KIP and/or JIRA to continue discussion and >> tease out design considerations. >> >> On Wed, Nov 13, 2019 at 3:45 PM Sandor Molnar >> wrote: >> >>> Hi folks, >>> >>> recently I had some contribution that allows end-users managing service >>> definitions without restarting the Knox Gateway. See >>> https://issues.apache.org/jira/browse/KNOX-2053 and >>> https://issues.apache.org/jira/browse/KNOX-2056 for further details. >>> >>> I've been just about creating a new JIRA to document those new API >>> endpoints in Knox user guide but it has come to my mind that we can do it >>> much better by using Swagger. >>> >>> Given the fact, the Admin API does not consist of hundreds of existing >>> endpoints it should not be 'that' huge work. I personally believe the >>> project would gain a lot by using this very useful tool (e.g. Admin API >>> documentation would be generated out-of-the-box, no more documentation >>> JIRAs required. Moreover, the generated documentation would be in sync with >>> the actual implementation). >>> You can check this out here: https://swagger.io/tools/swagger-ui/ (there >>> is a live demo too; it's worth looking at). >>> >>> Any comments, ideas are welcome! >>> >>> Cheers, >>> Sandor >>> >>
Re: [DISCUSS] Planning for Apache Knox 1.4
Based on that we are ~1/2 way through November and Thanksgiving in the US around the corner, I don't see much of the KnoxShell pieces getting integrated before end of November to make an end of November release. Specifically the following section: With the continued increase in cloud based deployments and Knox Gateway use > in securely accessing the exposed resources, we will concentrate on > KnoxShell as a first class environment for this access. This will likely > include an API for discovering metadata about the resources exposed through > Knox, the required authentication mechanisms, resource types and public > certs. It will also include Custom GroovyShell Commands for the KnoxShell > environment to help interact with the remote clusters and resultsets as > local in-memory tables. I will be start a KIP to try and articulate this > vision and related 1.4. usecases as well. > The KIP was just started so would be good to flesh that out more instead of rushing it into 1.4.0. There are a lot of moving pieces in that paragraph and would be good to make sure the Jiras are created and scoped appropriately. In addition to what was mentioned as features, there have been multiple new service definitions (Impala, Kudu, NiFi Registry) added as well as fixes to existing service definitions (Atlas, Livy, Ranger, Spark, YARN). So +1 to an end of November release, but need to make sure not trying to rush in new things just because a release will happen. There will be more releases. Kevin Risden On Fri, Nov 1, 2019 at 11:51 AM Sandeep Moré wrote: > Thanks for starting the planning thread Larry ! > Agree with the theme and the release date for 1.4.0. > > +1 > > Best, > Sandeep > > On Thu, Oct 31, 2019 at 11:53 AM larry mccay wrote: > >> Folks - >> >> Out last release with end of July, I apologize for the delay in starting >> the planning thread for 1.4. >> >> We currently have a backlog of ~65 JIRAs slated for a Fix Version of 1.4. >> >> There has been some work going on within KnoxShell to provide a general >> purpose representation for tabular data. This will be leveraged for >> rendering SQL query results as well as CSV files and simple processing >> within KnoxShell. I will be writing up a KIP to represent the overall >> vision for this work and initial set of usecases. >> >> We also have Cloudera Manager based discovery emerging and we should >> target an initial set of services to enable for CM/CDH and CDP deployments >> where CM is available. >> >> With the continued increase in cloud based deployments and Knox Gateway >> use in securely accessing the exposed resources, we will concentrate on >> KnoxShell as a first class environment for this access. This will likely >> include an API for discovering metadata about the resources exposed through >> Knox, the required authentication mechanisms, resource types and public >> certs. It will also include Custom GroovyShell Commands for the KnoxShell >> environment to help interact with the remote clusters and resultsets as >> local in-memory tables. I will be start a KIP to try and articulate this >> vision and related 1.4. usecases as well. >> >> I propose that the CM based Service Discovery and KnoxShell access to >> remote clusters be the primary themes of the Apache Knox 1.4 release. >> >> I also propose that we target the end of November as the release date for >> 1.4. >> >> Thoughts? >> >> --larry >> >
Re: Infinite login loop with Zeppelin + knox + pac4j on oidc to okta
So this has nothing to do with Knox sadly. I banged my head against a
similar problem a few weeks ago. I had wireshark captures from Knox setting
the cookie and it not being available.
The issue is that *.compute.amazonaws.com does not allow cookies to be set.
Here are some references that explain that browsers treat certain domains
as not allowed to set cookies for.
https://stackoverflow.com/questions/43520667/cookies-are-not-being-set-for-amazonaws-com-in-chrome-57-and-58-browsers
https://en.wikipedia.org/wiki/Public_Suffix_List
https://publicsuffix.org/list/public_suffix_list.dat
So you need to have a custom domain instead of relying on the EC2 hostnames
directly.
Kevin Risden
On Fri, May 24, 2019 at 2:00 AM Lokesh Lingarajan wrote:
> Hello all,
>
> I have been spending several sleepless nights on trying to get around this
> infinite login loop for my zeppelin okta integration.
>
> Following are configurations under knox and zeppelin
>
> *Knox: knoxsso.xml*
>
>
>
>
>
> webappsec
> WebAppSec
> true
> xframe.options.enabledtrue
>
>
> federation
> pac4j
> true
>
> pac4j.callbackUrl
> https://**.
> us-west-2.compute.amazonaws.com:8446/gateway/knoxsso/api/v1/websso
>
>
> pac4j.session.store
> J2ESessionStore
>
>
>clientName
>OidcClient
>
>
> oidc.id
>
>
>
> oidc.secret
> *
>
>
> oidc.preferredJwsAlgorithm
> RS256
>
>
> pac4j.id_attribute
> email
>
>
> oidc.discoveryUri
> https://
> /oauth2/default/.well-known/openid-configuration
>
>
>
> identity-assertion
> Default
> true
>
> principal.mapping
> *=user1;
>
>
> group.principal.mapping
> *=role1
>
>
> input
> (.*)\.(.*)@example\.com
>
>
> output
> {1}_{2}
>
>
>
> identity-assertion
> HadoopGroupProvider
> true
>
> hadoop.security.group.mapping
> org.apache.hadoop.security.ShellBasedUnixGroupsMapping
>
>
>
>
>knoxauth
>
>
> KNOXSSO
>
>knoxsso.cookie.secure.only
>false
>
>
>knoxsso.token.ttl
>360
>
>
> knoxsso.redirect.whitelist.regex
>
>
> ^https?:\/\/(localhost|127\.0\.0\.1|0:0:0:0:0:0:0:1|::1|.us\-west\-2\.compute\.amazonaws\.com):[0-9].*$
>
>
>
>
> *Knox: sandbox.xml*
>
>
>
>
>
> webappsec
> WebAppSec
> true
>
> cors.enabled
> true
>
>
>
> federation
> SSOCookieProvider
> true
>
> sso.authentication.provider.url
> https://**.
> us-west-2.compute.amazonaws.com:8446/gateway/knoxsso/api/v1/websso
>
>
>
>
>
>
> *Zeppelin: shiro.ini*
> [main]
> ## A same for configuring Knox SSO Realm
> knoxJwtRealm = org.apache.zeppelin.realm.jwt.KnoxJwtRealm
> knoxJwtRealm.providerUrl =
> https://ec2-34-221-96-42.us-west-2.compute.amazonaws.com:8446/
> knoxJwtRealm.login = gateway/knoxsso/api/v1/websso
> knoxJwtRealm.logout = gateway/knoxssout/api/v1/webssout
> knoxJwtRealm.logoutAPI = true
> knoxJwtRealm.redirectParam = originalUrl
> knoxJwtRealm.cookieName = hadoop-jwt
> knoxJwtRealm.publicKeyPath = /etc/zeppelin/conf.dist/knox-sso.pem
> #knoxJwtRealm.groupPrincipalMapping = group.principal.mapping
> #knoxJwtRealm.principalMapping = principal.mapping
> authc = org.apache.zeppelin.realm.jwt.KnoxAuthenticationFilter
>
> ##sessionManager =
> org.apache.shiro.web.session.mgt.DefaultWebSessionManager
>
> ### If caching of user is required then uncomment below lines
> #cacheManager = org.apache.shiro.cache.MemoryConstrainedCacheManager
> #securityManager.cacheManager = $cacheManager
>
> ### Enables 'HttpOnly' flag in Zeppelin cookies
> ##cookie = org.apache.shiro.web.servlet.SimpleCookie
> ##cookie.name = JSESSIONID
> #cookie.name = hadoop-jwt
> #cookie.httpOnly = true
> ### Uncomment the below line only when Zeppelin is running over HTTPS
> ##cookie.secure = true
> ##sessionManager.sessionIdCookie = $cookie
>
> ##securityManager.sessionManager = $sessionManager
> # 86,400,000 milliseconds = 24 hour
> #
Re: Unable to POST data to a Target Service which is proxied through KNOX
You haven't shared the exact error message you are getting including the
HTTP status code or the way that you are connecting to the service.
Here is an example without Knox that will cause POST requests to not work.
POST requests typically will not follow a redirect by default. So if you
are connecting to a load balancer in front of Knox, then POST may not work.
Kevin Risden
On Wed, May 1, 2019 at 8:26 AM sk_ac...@yahoo.com
wrote:
>
>
> On Tuesday, April 30, 2019, 7:58:03 PM EDT, sk_ac...@yahoo.com <
> sk_ac...@yahoo.com> wrote:
>
>
> Hi All,
>
>I am using 0.12 version of Knox and configured a service to be proxied
> through knox as listed below:
>
>My Topology is :
>
>
>
>
>
> webappsec
> WebAppSec
> true
>
> cors.enabled
> true
>
>
> cors.supportedMethods
> GET,HEAD,POST,OPTIONS,PUT
>
>
> xframe.options.enabled
> true
>
>
> xframe.options
> SAMEORIGIN
>
>
>
> identity-assertion
> Default
> true
>
>
>
> MYSERVICE
> http://localhost:9090
>
>
>
> and the Service it self is described as
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> classname="org.apache.hadoop.gateway.dispatch.PassAllHeadersDispatch"/>
>
>
>
>
> pattern="*://*:*/**/ myservice/{path=**}?{**}">
>
>
>
>
> pattern="*://*:*/**/myservice/">
>
>
>
> pattern="*://*:*/**/myservice/{**}">
>
>
>
>
>
>
> When i tried access the Service GET Requests are working as expected. POST
> requests are also routed to the service how ever the POST Body is not
> getting sent to the target service. Not sure what's the issue is.
>
> Thanks
> Sateesh
>
>
Re: Topology without authentication
You can configure a topology with anonymous authentication. https://knox.apache.org/books/knox-1-2-0/user-guide.html#Example+Topology authentication Anonymous true identity-assertion Default false This will make Knox not do authentication and will rely on the backend service to do authentication. PS - This assumes that the service definition isn't forcing a specific authentication provider, but most do not specify. Kevin Risden On Thu, Apr 25, 2019 at 1:12 PM Odon Copon wrote: > Hi, > Would be possible to write a basic topology without any kind of > authentication? > I create one and I'm getting 401s when there is no authentication at all? > Is it using some global authentication I'm not aware of? > Thanks! >
Re: Nifi + Knox + Knoxsso
Not sure this reference will help but might give you a push in the right direction: https://risdenk.github.io/2018/03/18/apache-knox-proxying-apache-nifi.html The blog post is focused on setting up NiFi/Knox correctly. The blog post doesn't use KnoxSSO for the authentication but should be possible to replace the LDAP stuff with KnoxSSO. I don't think you need Shiro and SSOCookieProvider. manager.xml has an example of just KnoxSSO without needing the Shiro provider. Kevin Risden On Thu, Apr 4, 2019 at 12:20 PM Thibault VERBEQUE < thibault.verbe...@omnilog.fr> wrote: > Hi all, > > I’m struggling to configure correctly Knox in order to publish a Nifi > cluster. > Currently I’m using the following topologie : > > > > > > > > authentication > > ShiroProvider > > true > > > > sessionTimeout > > 1800 > > > > LDAP_CFG_REDACTED > > > > redirectToUrl > > > /gateway/discovery-preprod/knoxauth/login.html > > > > > > restrictedCookies > > rememberme,WWW-Authenticate > > > > > > urls./** > > authcBasic > > > > > > > > federation > > SSOCookieProvider > > true > > > > sso.authentication.provider.url > > > https://host1.sub.dom.tld:9243/gateway/topologie-name/api/v1/websso > > > > > > > > > webappsec > > WebAppSec > > true > > > > csrf.enabled > > false > > > > > > csrf.customHeader > > X-XSRF-Header > > > > > > csrf.methodsToIgnore > > GET,OPTIONS,HEAD > > > > > > xframe.options.enabled > > true > > > > > > xss.protection.enabled > > false > > > > > > strict.transport.enabled > > false > > > > > > > > identity-assertion > > Default > > true > > > > > > hostmap > > static > > true > > > > host1.dom2.tld2 > > host1.sub.dom.tld > > > > > > > > authorization > > XASecurePDPKnox > > true > > > > > > > > KNOXSSO > > > > knoxsso.cookie.secure.only > > false > > > > > > knoxsso.enable.session > > true > > > > > > knoxsso.token.ttl > > 36 > > > > > > knoxsso.redirect.whitelist.regex > > > ^/.*$;^https?://(.+\.sub\.dom\.tld|.+\.dom2\.tld2)(:[0-9]+)(/|/.*)?$ > > > > > > > > NIFI > > https://host1.dom2.tld:9091 > > https://host2.dom2.tld:9091 > > https://host3.dom2.tld:9091 > > > > useTwoWaySsl > > true > > > > > > > > knoxauth > > > > > > > > Relevant certs are already created and imported (user cert, key and nifi CA > certificate) in gateway.jks. I set up proxy host, path and knox params in > nifi.properties and disabled other users sources (empty > nifi.security.user.login.identity.provider). > > I can successfully authenticate in Nifi with generated certs, so Ranger > policy seems correct. > I observed the following behavior when I enter > https://host1.sub.dom.tld:9243/gateway/topologie-name/nifi-app/nifi : > >- Without modification to service.xml: > - Knox redirects the user to > > https://host1.sub.dom.tld:9243/gateway/topologie-name/knoxauth/login.html > (no request forwarding to the backend) > - If I manually add ?OriginalUrl= >
Re: Connection issues when using Knox + Hive
Ssl should be true? Check gateway.log for any exceptions as well. Kevin Risden On Wed, Apr 3, 2019, 06:45 Odon Copon wrote: > Hi, I'm having issues when trying to connect to Hive through Knox using > beeline. > My current infrastructure is a Hadoop cluster, and a separate instance > running Knox. > > With the following beeline connection I'm able to use Hive from beeline, > without Knox: > > !connect > jdbc:hive2://:10001/default;auth=noSasl;transportMode=http;httpPath=cliservice > org.apache.hive.jdbc.HiveDriver > > This means that Hiveserver2 is using http transport mode, using > cliservice endpoint and port 10001 is working. > > I replicate that on Knox, following these instructions: > https://knox.apache.org/books/knox-1-1-0/user-guide.html#Hive > > And created a topology named "test" that just contains: > > > > authentication > ShiroProvider > true > > > > > > > > HIVE > http://:10001/cliservice > > replayBufferSize > 8 > > > > > But I'm not able to connect running the following: > > !connect > jdbc:hive2://:/default;auth=noSasl;ssl=false;transportMode=http;httpPath=gateway/test/hive > admin admin-secret org.apache.hive.jdbc.HiveDriver > > I get the following error: > > 19/04/03 03:39:48 [main]: WARN jdbc.HiveConnection: Failed to connect to > : > Error: Could not open client transport with JDBC Uri: > jdbc:hive2://:default;auth=noSasl;ssl=false;transportMode=http;httpPath=gateway/test/hive: > Could not establish connection to > jdbc:hive2://:/default;auth=noSasl;ssl=false;transportMode=http;httpPath=gateway/test/hive: > org.apache.http.client.ClientProtocolException (state=08S01,code=0) > > Any idea? Am I missing something? > > Thanks. > >
Re: Differences between conf and topologies
"conf/gateway-site.xml" has configurations that are global to the entire Knox instance. "topologies/abc.xml" are configurations specific to the topology. The topologies can reference some configurations in the global "conf/gateway-site.xml". One of those is the group configurations. This would be useful to say have multiple topologies not have to repeat the same configuration over and over again. On the other hand, you can so choose to duplicate the configuration in each topology. Kevin Risden On Mon, Apr 1, 2019 at 11:13 AM Odon Copon wrote: > Hi, > What are the main differences between "conf/gateway-site.xml", > "topologies/manager.xml" and "topologies/admin.xml"? > I'm still trying to understand how all work together in Knox. > > Per example, I can see there's a section for LDAP configuration on > "topologies/admin.xml" but also as part of "conf/gateway-site.xml". Which > one affects what exactly? > > Thanks >
Re: Add a filter to intercept Knox requests and be able to add a custom logic
Should be able to use the default http client that is included in Knox.
Should be accessible with getHttpClient() from AbstractGatewayDispatch.
Kevin Risden
On Fri, Mar 29, 2019 at 11:22 AM Matteo Alessandroni
wrote:
> Hi,
>
> Phil the "SubjectUtils.getCurrentEffectivePrincipalName()" method did the
> trick, thanks!
>
> Does anybody of you guys know how I could use a CXF REST client from my
> Dispatch class (that extends "DefaultDispatch") to call an external link?
> I tried to make a ".jar" that also includes the following dependencies
>
>
> org.apache.cxf
> cxf-rt-frontend-jaxrs
>
>
> org.apache.cxf
> cxf-rt-rs-client
>
>
> org.apache.cxf
> cxf-rt-transports-http
>
>
> org.apache.cxf
> cxf-rt-transports-http-jetty
>
>
> in order to execute:
>
> MyRequest request = new MyRequest();
> Client client = ClientBuilder.newBuilder().newClient();
> WebTarget target = client.target(MYURL);
> Invocation.Builder builder =
> target.request(MediaType.APPLICATION_JSON_TYPE);
> MyResponse response =
> builder.post(Entity.json(request.toString()), MyResponse.class);
>
> from the "executeRequest()" method:
>
> @Override
> protected void *executeRequest*(
> HttpUriRequest outboundRequest,
> HttpServletRequest inboundRequest,
> HttpServletResponse outboundResponse) throws IOException {
>
> but it does not work and it says:
>
> javax.ws.rs.ProcessingException: org.apache.cxf.BusException: No conduit
> initiator was found for the namespace
> http://cxf.apache.org/transports/http.
> at
> org.apache.cxf.jaxrs.client.AbstractClient.checkClientException(AbstractClient.java:638)
> at
> org.apache.cxf.jaxrs.client.AbstractClient.preProcessResult(AbstractClient.java:614)
> at
> org.apache.cxf.jaxrs.client.WebClient.doResponse(WebClient.java:1115)
> at
> org.apache.cxf.jaxrs.client.WebClient.doChainedInvocation(WebClient.java:1052)
> at
> org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:897)
> at
> org.apache.cxf.jaxrs.client.WebClient.doInvoke(WebClient.java:866)
> at org.apache.cxf.jaxrs.client.WebClient.invoke(WebClient.java:431)
> at
> org.apache.cxf.jaxrs.client.SyncInvokerImpl.method(SyncInvokerImpl.java:150)
> at
> org.apache.cxf.jaxrs.client.SyncInvokerImpl.post(SyncInvokerImpl.java:90)
> at
> org.apache.cxf.jaxrs.client.spec.InvocationBuilderImpl.post(InvocationBuilderImpl.java:158)
>
> is there something built-in I can use? Or maybe I'm doing something wrong
> with the CXF client?
>
> Thanks
>
>
>
> On 18/03/19 09:03, Matteo Alessandroni wrote:
>
> Hi Phil,
>
> thank you for the help!
> I'll check that class and will let you know.
>
> Thanks again!
> Regards,
> Matteo
>
>
>
> On 15/03/19 17:05, Phil Zampino wrote:
>
> I was going to suggest that you might be experiencing a docker issue
> because the ext folder works; You beat me to it.
>
> I'm not sure if this is what you're after, but
> org.apache.knox.gateway.security.SubjectUtils might be of use to you for
> identifying the currently logged-in principal.
>
> On Fri, Mar 15, 2019 at 11:53 AM Matteo Alessandroni
> wrote:
>
>> Ok I think I found the issue:
>> I had to restart the full Docker container to actually restart Knox.
>> So I did that and it seems it's getting my class in "ext/" now.
>>
>> The error I get now is:
>>
>> Caused by: org.apache.knox.gateway.config.ConfigurationException: Failed
>> to find configuration for provider as provider of
>> org.apache.knox.gateway.dispatch.MyCustomDispatch via
>> org.apache.knox.gateway.config.FilterConfigurationAdapter
>>
>> so I guess it's a problem with my class code.
>> Is there a very simple code snippet I could use for my custom dispatch
>> in order to access the "request" object in order to get the name of the
>> user that is currently logged?
>>
>> Regards,
>> Matteo
>>
>>
>> On 15/03/19 15:49, Matteo Alessandroni wrote:
>>
>> Thanks!
>> I checked that too, my ".jar" has the same permissions of the other files
>> in "lib/" folder (e.g. "gateway-shell-1.2.0.jar").
>> I see your point about the script [1], but I also see that the original
>> gateway.sh (for Knox v1.2.0) [2] is not doing anything special for the
>> "ext/" folder, or am I wrong?
>>
>> Regards,
>> Matteo
>>
>>
>> [1]
>> https://github.com/moresandeep/knox-dev-docker/blob/master/build/gateway.sh
&
Re: Hive CLI behind Knox
> > some connections might be not able to connect to HS2? > Not sure what you mean by "some connections". HTTP mode works for JDBC/ODBC. There are 2 modes HTTP and binary. Both are thrift it just is a different protocol. Knox only works with HTTP. Existing binary connections (which are on a different port typically) will not work but there is usually a slightly different way to connect over HTTP. This is probably a better question for the Hive project. > Would a user be able to bypass Knox by using an old Hive CLI (no beeline > based) even if HS2 is using the latest Hive version with Hive CLI wrapping > beeline? > If the user has a Kerberos ticket then they would be able to connect I think. It is more of a question for the Hive project though. Knox can't enforce access outside of it. Kevin Risden On Wed, Mar 20, 2019 at 7:10 AM Odon Copon wrote: > Thanks Kevin. > When you said "Knox with Hive requires HiveServer2 in HTTP mode", does it > mean once this mode is activated some connections might be not able to > connect to HS2? > Would a user be able to bypass Knox by using an old Hive CLI (no beeline > based) even if HS2 is using the latest Hive version with Hive CLI wrapping > beeline? > > Thanks. > > On Wed, 13 Mar 2019 at 17:08, Kevin Risden wrote: > >> I'm not sure which version exactly but this is the umbrella jira tracking >> all the subtasks: >> >> https://issues.apache.org/jira/browse/HIVE-10511 >> >> Kevin Risden >> >> >> On Wed, Mar 13, 2019 at 1:04 PM Odon Copon wrote: >> >>> Amazing, thanks Kevin for the clarification. >>> Do you know exactly which version did the transition to beeline? >>> >>> On Wed, 13 Mar 2019 at 16:53, >>> Kevin Risden >>> wrote: >>> >>>> Knox with Hive requires HiveServer2 in HTTP mode. Hive CLI direct to >>>> metastore is not supported by Knox. Current versions of Hive as far as I >>>> know have Hive CLI wrapping beeline by default for SQL queries. >>>> >>>> Kevin Risden >>>> >>>> >>>> On Wed, Mar 13, 2019 at 12:49 PM Odon Copon >>>> wrote: >>>> >>>>> After reading Knox documentation I'm not totally sure how Knox can sit >>>>> in front of Hive accesses through JDBC/Beeline and Hive CLI. >>>>> From the docs, I see it does for JDBC connections so I assume it does >>>>> for Beeline as well, as both hit HiveServer2, but what about Hive CLI that >>>>> go directly to the Hive Metastore?. >>>>> Thanks. >>>>> >>>>
Re: Hive CLI behind Knox
I'm not sure which version exactly but this is the umbrella jira tracking all the subtasks: https://issues.apache.org/jira/browse/HIVE-10511 Kevin Risden On Wed, Mar 13, 2019 at 1:04 PM Odon Copon wrote: > Amazing, thanks Kevin for the clarification. > Do you know exactly which version did the transition to beeline? > > On Wed, 13 Mar 2019 at 16:53, > Kevin Risden > wrote: > >> Knox with Hive requires HiveServer2 in HTTP mode. Hive CLI direct to >> metastore is not supported by Knox. Current versions of Hive as far as I >> know have Hive CLI wrapping beeline by default for SQL queries. >> >> Kevin Risden >> >> >> On Wed, Mar 13, 2019 at 12:49 PM Odon Copon wrote: >> >>> After reading Knox documentation I'm not totally sure how Knox can sit >>> in front of Hive accesses through JDBC/Beeline and Hive CLI. >>> From the docs, I see it does for JDBC connections so I assume it does >>> for Beeline as well, as both hit HiveServer2, but what about Hive CLI that >>> go directly to the Hive Metastore?. >>> Thanks. >>> >>
Re: Hive CLI behind Knox
Knox with Hive requires HiveServer2 in HTTP mode. Hive CLI direct to metastore is not supported by Knox. Current versions of Hive as far as I know have Hive CLI wrapping beeline by default for SQL queries. Kevin Risden On Wed, Mar 13, 2019 at 12:49 PM Odon Copon wrote: > After reading Knox documentation I'm not totally sure how Knox can sit in > front of Hive accesses through JDBC/Beeline and Hive CLI. > From the docs, I see it does for JDBC connections so I assume it does for > Beeline as well, as both hit HiveServer2, but what about Hive CLI that go > directly to the Hive Metastore?. > Thanks. >
Re: Load balancing of Hiveserver2 through Knox
The main issue with load balancing HS2 is that HS2 is stateful. This means that you need to keep the same user going back to the same HS2 instance. If you connect to HS2 via JDBC, when the next set of rows is requested it must go back to the same HS2 instance. Knox doesn't support load balancing backends currently. Adding an HTTP load balancer behind Knox before HS2 can be done but need to be careful with Kerberos and sticky sessions. Kevin Risden On Tue, Jan 15, 2019 at 11:16 AM David Villarreal wrote: > > Hi Rabii, > > > > There is a lot to think about here. I don’t think every request/connection > would be a good design to check zookeeper every time, but maybe if there is a > way to identify a new client-session we could design it to go check > zookeeper. We would also need to see what impact in performance this could > be. But I do like the concept. Just keep in mind for zookeeper, I don’t > think this is a true loadbalancer in the hive code. I believe it randomly > returns a host:port for a registered hiveserver2 instance. > > > > Best regards, > > > > David > > From: rabii lamriq > Reply-To: "user@knox.apache.org" > Date: Tuesday, January 15, 2019 at 1:01 AM > To: "user@knox.apache.org" > Subject: Load balancing of Hiveserver2 through Knox > > > > Hi > > > > I am using knox to connect to HS2, but Knox ensure only HA and not Load > balancing. > > > > In fact, I noticed that there are a load balancing when I connect to HS2 > using Zookeeper only, but using Knox, knox connect to zookeeper to get an > available instance of HS2, then use this instance for all connection. > > > > My question is : can we make any thing to let knox to connect to zookeeper in > each new connection in order to get a different instance for each new > connection to HS2. > > > > Best > > Rabii
Re: LDAP configuration using knox.
If you change the service.xml or rewrite.xml you also need to touch or modify the topology to create a redeployment. You should see in the gateway.log that the topology was redeployed. If you don't see that, your changes probably didn't take effect. Kevin Risden On Thu, Dec 20, 2018 at 9:20 AM Tomislav Novosel wrote: > > I changed line: > classname="org.apache.knox.gateway.dispatch.PassAllHeadersDispatch"/> > in service.xml > to: > > > and name of authentication provider I changed from anonymous to ShiroProvider. > Nothing happened. Access is open as it was. > > Tom > > On Thu, 20 Dec 2018 at 14:39, larry mccay wrote: >> >> If you followed the proxying article and your service definition is >> indicating the anonymous authentication provider then that is the issue. >> That overrides any provider configured in the topology. >> >> >> On Thu, Dec 20, 2018, 8:27 AM Kevin Risden >> >>> If your service.xml has >>> >>> >> classname="org.apache.knox.gateway.dispatch.PassAllHeadersDispatch"/> >>> >>> Then all the headers including the AUTHORIZATION header are passed >>> through to the backend. This would mean that your backend is >>> responsible for authentication. >>> >>> If you want Knox to do authentication, either remove the dispatch line >>> or change it to: >>> >>> >>> >>> This will then pass through Kerberized credentials to your backend. If >>> you are looking to do SSO with Knox then proxying isn't required. >>> >>> Kevin Risden >>> On Thu, Dec 20, 2018 at 8:22 AM Tomislav Novosel >>> wrote: >>> > >>> > Hi Knox team, >>> > >>> > I'm trying to configure LDAP authentication on Knox. I followed this link >>> > to setup simple web application for testing access. It's a simple >>> > HelloWorld. >>> > >>> > https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox >>> > >>> > After setup, I've succeeded to access app through Knox gateway. >>> > >>> > I want to restrict access on that webapp URL with some username and >>> > password to type, so I tried to configure LDAP using this link: >>> > >>> > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Knox_Gateway_Admin_Guide/content/setting_up_ldap_authentication.html >>> > >>> > After that, access was still the same, without user/pass prompt. IDK if >>> > that is even posible with Knox. >>> > In logs I see that access is granted, response code 200. >>> > >>> > How can I achieve user/pass prompt when accessing webapp URL through Knox >>> > gateway? >>> > >>> > Thank you. >>> > BR, >>> > Tom
Re: LDAP configuration using knox.
If your service.xml has Then all the headers including the AUTHORIZATION header are passed through to the backend. This would mean that your backend is responsible for authentication. If you want Knox to do authentication, either remove the dispatch line or change it to: This will then pass through Kerberized credentials to your backend. If you are looking to do SSO with Knox then proxying isn't required. Kevin Risden On Thu, Dec 20, 2018 at 8:22 AM Tomislav Novosel wrote: > > Hi Knox team, > > I'm trying to configure LDAP authentication on Knox. I followed this link to > setup simple web application for testing access. It's a simple HelloWorld. > > https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox > > After setup, I've succeeded to access app through Knox gateway. > > I want to restrict access on that webapp URL with some username and password > to type, so I tried to configure LDAP using this link: > > https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.3.2/bk_Knox_Gateway_Admin_Guide/content/setting_up_ldap_authentication.html > > After that, access was still the same, without user/pass prompt. IDK if that > is even posible with Knox. > In logs I see that access is granted, response code 200. > > How can I achieve user/pass prompt when accessing webapp URL through Knox > gateway? > > Thank you. > BR, > Tom
[ANNOUNCE] Apache Knox 1.2.0 Release
The Apache Knox team is proud to announce the release of Apache Knox 1.2.0! Apache Knox is a REST API Gateway for providing secure access to the data and processing resources of Hadoop clusters. More details on Apache Knox can be found at: https://knox.apache.org/ The release bits are available at: https://www.apache.org/dyn/closer.cgi/knox/1.2.0 The change list is in the CHANGES file in the above directory. We would like to thank all of contributors who made the release possible and to invite others interested in helping out to engage the community on the dev and users lists! Kevin Risden
Re: [VOTE] Release Apache Knox 1.2.0 RC 4
Billy - Yea we would target 1.3.0 likely unless it is a regression that we can't ship with. Knox 1.2.0 has a lot of fixes for YARN and JobHistoryUI - See https://issues.apache.org/jira/browse/KNOX-1207 If you have fixes for YARNUIV2, can you see if they fix the items laid out here - https://issues.apache.org/jira/browse/KNOX-1589 Kevin Risden On Thu, Dec 13, 2018 at 11:53 AM Watson, Billy wrote: > I have a bunch of fixes for knox yarn, yarnuiv2, jobhistoryui, etc. Is it > too late to get those into 1.2.0 since you’re at RC stages? > > > > *Billy Watson* > > > > > > > *From:* > *Kevin Risden* > > > > > * > Reply-To: > "user@knox.apache.org " > Date: Monday, December 10, 2018 at 3:49 PM To: > "d...@knox.apache.org " >, "user@knox.apache.org " > > Subject: Re: [VOTE] Release > Apache Knox 1.2.0 RC 4 * > > * +1 Tested the following: · src zip - mvn verify -Ppackage,release > · knoxshell - able to connect to Knox and execute a few examples · Ran rc > against https://github.com/risdenk/knox-performance-tests > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Frisdenk%2Fknox-performance-tests=02%7C01%7CBilly.Watson%40disney.com%7C74475999c7364c90edeb08d65ee0f4fe%7C56b731a8a2ac4c32bf6b616810e913c6%7C1%7C0%7C636800717610028222=F0o1oCOldpYUug7riBxSwFJwn3rmFk8mWaoxM3ZFctY%3D=0> > to > test unsecure webhdfs, hbase, hive · Manual testing of UIs with Kerberos > · Zeppelin UI with websocket enabled · Manual testing of Knox against > Kerberized cluster · knoxsso Pac4j backed by Okta · hadoopauth provider > · token service and Bearer tokens · default topology url · topology port > mapping · ambari discovery · hadoop group provider · metrics · ranger > integration - via /ext loading · Knoxsso misconfiguration doesn't result > in stacktrace Kevin Risden On Mon, Dec 10, 2018 at 3:40 PM Kevin Risden > > wrote: Release candidate #4 for > the Apache Knox 1.2.0 release is available at: > https://dist.apache.org/repos/dist/dev/knox/knox-1.2.0/ > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fknox%2Fknox-1.2.0%2F=02%7C01%7CBilly.Watson%40disney.com%7C74475999c7364c90edeb08d65ee0f4fe%7C56b731a8a2ac4c32bf6b616810e913c6%7C1%7C0%7C636800717610028222=ryoJqjoBUis3W0xKVA9V4hEAD2EMriIsjnqjqhsAtDo%3D=0> > The release candidate is a zip archive of the sources in: > https://git-wip-us.apache.org/repos/asf/knox.git > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit-wip-us.apache.org%2Frepos%2Fasf%2Fknox.git=02%7C01%7CBilly.Watson%40disney.com%7C74475999c7364c90edeb08d65ee0f4fe%7C56b731a8a2ac4c32bf6b616810e913c6%7C1%7C0%7C636800717610038227=ti4Yy1n%2BrxZ%2BvpPD1pqcM726f8P3aPTrYWxtnsO1Sm4%3D=0> > Branch v1.2.0 (git checkout -b v1.2.0) Tag is v1.2.0-rc4 (git checkout -b > v1.2.0-rc4) The KEYS file for signature validation is available > at: https://dist.apache.org/repos/dist/release/knox/KEYS > <https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Frelease%2Fknox%2FKEYS=02%7C01%7CBilly.Watson%40disney.com%7C74475999c7364c90edeb08d65ee0f4fe%7C56b731a8a2ac4c32bf6b616810e913c6%7C1%7C0%7C636800717610048236=Tj0YnW%2BLwuygHka0NHSV6w0YmsaJM%2BS0lCrTv7j15RU%3D=0> > Please vote on releasing this package as Apache Knox 1.2.0. The vote is > open for the next 72 hours and passes if a majority of at least three +1 > Apache Knox PMC votes are cast. [ ] +1 Release this package as Apache > Knox 1.2.0 [ ] -1 Do not release this package because... Kevin Risden * >
Re: [VOTE] Release Apache Knox 1.2.0 RC 4
+1 Tested the following: - src zip - mvn verify -Ppackage,release - knoxshell - able to connect to Knox and execute a few examples - Ran rc against https://github.com/risdenk/knox-performance-tests to test unsecure webhdfs, hbase, hive - Manual testing of UIs with Kerberos - Zeppelin UI with websocket enabled - Manual testing of Knox against Kerberized cluster - knoxsso Pac4j backed by Okta - hadoopauth provider - token service and Bearer tokens - default topology url - topology port mapping - ambari discovery - hadoop group provider - metrics - ranger integration - via /ext loading - Knoxsso misconfiguration doesn't result in stacktrace Kevin Risden On Mon, Dec 10, 2018 at 3:40 PM Kevin Risden wrote: > Release candidate #4 for the Apache Knox 1.2.0 release is available at: > > https://dist.apache.org/repos/dist/dev/knox/knox-1.2.0/ > > The release candidate is a zip archive of the sources in: > > https://git-wip-us.apache.org/repos/asf/knox.git > Branch v1.2.0 (git checkout -b v1.2.0) > Tag is v1.2.0-rc4 (git checkout -b v1.2.0-rc4) > > The KEYS file for signature validation is available at: > https://dist.apache.org/repos/dist/release/knox/KEYS > > Please vote on releasing this package as Apache Knox 1.2.0. > The vote is open for the next 72 hours and passes if a majority of at > least three +1 Apache Knox PMC votes are cast. > > [ ] +1 Release this package as Apache Knox 1.2.0 > [ ] -1 Do not release this package because... > > Kevin Risden >
[VOTE] Release Apache Knox 1.2.0 RC 4
Release candidate #4 for the Apache Knox 1.2.0 release is available at: https://dist.apache.org/repos/dist/dev/knox/knox-1.2.0/ The release candidate is a zip archive of the sources in: https://git-wip-us.apache.org/repos/asf/knox.git Branch v1.2.0 (git checkout -b v1.2.0) Tag is v1.2.0-rc4 (git checkout -b v1.2.0-rc4) The KEYS file for signature validation is available at: https://dist.apache.org/repos/dist/release/knox/KEYS Please vote on releasing this package as Apache Knox 1.2.0. The vote is open for the next 72 hours and passes if a majority of at least three +1 Apache Knox PMC votes are cast. [ ] +1 Release this package as Apache Knox 1.2.0 [ ] -1 Do not release this package because... Kevin Risden
[CANCEL][VOTE] Release Apache Knox 1.2.0 RC 3
-1 I found a new issue with misconfigured knoxsso causes a bad stack trace. https://issues.apache.org/jira/browse/KNOX-1651 The reason this comes up is part of KNOX-1152 that was added to 1.2.0. We should fix it. Kevin Risden On Mon, Dec 3, 2018 at 9:51 AM Kevin Risden wrote: > +1 > > Tested the following: > >- src zip - mvn verify -Ppackage,release >- knoxshell - able to connect to Knox and execute a few examples >- Ran rc2 against https://github.com/risdenk/knox-performance-tests to >test unsecure webhdfs, hbase, hive >- Manual testing of UIs with Kerberos >- Zeppelin UI with websocket enabled >- Manual testing of Knox against Kerberized cluster >- knoxsso Pac4j backed by Okta >- hadoopauth provider >- token service and Bearer tokens >- default topology url >- topology port mapping >- ambari discovery >- hadoop group provider >- metrics >- ranger integration - via /ext loading > > Kevin Risden > > > On Fri, Nov 30, 2018 at 11:33 AM Kevin Risden wrote: > >> Release candidate #3 for the Apache Knox 1.2.0 release is available at: >> >> https://dist.apache.org/repos/dist/dev/knox/knox-1.2.0/ >> >> The release candidate is a zip archive of the sources in: >> >> https://git-wip-us.apache.org/repos/asf/knox.git >> Branch v1.2.0 (git checkout -b v1.2.0) >> Tag is v1.2.0-rc3 (git checkout -b v1.2.0-rc3) >> >> The KEYS file for signature validation is available at: >> https://dist.apache.org/repos/dist/release/knox/KEYS >> >> Please vote on releasing this package as Apache Knox 1.2.0. >> The vote is open for the next 72 hours and passes if a majority of at >> least three +1 Apache Knox PMC votes are cast. >> >> [ ] +1 Release this package as Apache Knox 1.2.0 >> [ ] -1 Do not release this package because... >> >> Kevin Risden >> >
Re: [VOTE] Release Apache Knox 1.2.0 RC 3
+1 Tested the following: - src zip - mvn verify -Ppackage,release - knoxshell - able to connect to Knox and execute a few examples - Ran rc2 against https://github.com/risdenk/knox-performance-tests to test unsecure webhdfs, hbase, hive - Manual testing of UIs with Kerberos - Zeppelin UI with websocket enabled - Manual testing of Knox against Kerberized cluster - knoxsso Pac4j backed by Okta - hadoopauth provider - token service and Bearer tokens - default topology url - topology port mapping - ambari discovery - hadoop group provider - metrics - ranger integration - via /ext loading Kevin Risden On Fri, Nov 30, 2018 at 11:33 AM Kevin Risden wrote: > Release candidate #3 for the Apache Knox 1.2.0 release is available at: > > https://dist.apache.org/repos/dist/dev/knox/knox-1.2.0/ > > The release candidate is a zip archive of the sources in: > > https://git-wip-us.apache.org/repos/asf/knox.git > Branch v1.2.0 (git checkout -b v1.2.0) > Tag is v1.2.0-rc3 (git checkout -b v1.2.0-rc3) > > The KEYS file for signature validation is available at: > https://dist.apache.org/repos/dist/release/knox/KEYS > > Please vote on releasing this package as Apache Knox 1.2.0. > The vote is open for the next 72 hours and passes if a majority of at > least three +1 Apache Knox PMC votes are cast. > > [ ] +1 Release this package as Apache Knox 1.2.0 > [ ] -1 Do not release this package because... > > Kevin Risden >
[VOTE] Release Apache Knox 1.2.0 RC 3
Release candidate #3 for the Apache Knox 1.2.0 release is available at: https://dist.apache.org/repos/dist/dev/knox/knox-1.2.0/ The release candidate is a zip archive of the sources in: https://git-wip-us.apache.org/repos/asf/knox.git Branch v1.2.0 (git checkout -b v1.2.0) Tag is v1.2.0-rc3 (git checkout -b v1.2.0-rc3) The KEYS file for signature validation is available at: https://dist.apache.org/repos/dist/release/knox/KEYS Please vote on releasing this package as Apache Knox 1.2.0. The vote is open for the next 72 hours and passes if a majority of at least three +1 Apache Knox PMC votes are cast. [ ] +1 Release this package as Apache Knox 1.2.0 [ ] -1 Do not release this package because... Kevin Risden
[CANCEL][VOTE] Release Apache Knox 1.2.0 RC 2
-1 found https://issues.apache.org/jira/browse/KNOX-1645 All other testing went well minus Zeppelin UI due to KNOX-1645. I'll respin a new rc later today. Tested the following: - src zip - mvn verify -Ppackage,release - knoxshell - able to connect to Knox and execute a few examples - Ran rc2 against https://github.com/risdenk/knox-performance-tests to test unsecure webhdfs, hbase, hive - Manual testing of UIs with Kerberos - Zeppelin UI with websocket enabled - Manual testing of Knox against Kerberized cluster - knoxsso Pac4j backed by Okta - hadoopauth provider - token service and Bearer tokens - default topology url - topology port mapping - ambari discovery - hadoop group provider Kevin Risden On Wed, Nov 28, 2018 at 2:41 PM larry mccay wrote: > All - > > Thanks to Kevin for so much work in cleaning up the backlog and taking on > release manager work for 1.2.0! > > The 1.2.0 release happens to contain many dependency upgrades. > Not the least of which is Jetty itself from 9.2.x to 9.4.x. > > We need to put some key areas through their paces pretty rigorously in > order to tease out any corner case regressions. > > For instance, any Jetty specific features - such as: > * Websocket support > * SSL configuration params > * buffer size tweaks > * comb the gateway config and see which things are configuring aspects of > Jetty where units of measure defaults may change, etc > > * Pac4J was upgraded - we will need KnoxSSO testing of SAML via Okta, > Google Authenticator, etc. > > * Classloading is already known to have changed and has caused some issues > that have already been found. > Give some thought into possible classloading issues: singletons, custom > providers added to the ext directory, etc. > > If anyone can think of other corner cases that may not be immediately > covered by default and common configurations please call them out and/or > list what you have tested for them. > > thanks! > > --larry > > On Wed, Nov 28, 2018 at 2:16 PM Jeffrey Rodriguez > wrote: > > > +1 , based on my review and testing. > > > > Thanks, > > Jeffrey E Rodriguez > > > > On Wed, Nov 28, 2018 at 11:05 AM > Kevin Risden > wrote: > > > > > Release candidate #2 for the Apache Knox 1.2.0 release is available at: > > > > > > https://dist.apache.org/repos/dist/dev/knox/knox-1.2.0/ > > > > > > The release candidate is a zip archive of the sources in: > > > > > > https://git-wip-us.apache.org/repos/asf/knox.git > > > Branch v1.2.0 (git checkout -b v1.2.0) > > > Tag is v1.2.0-rc2 (git checkout -b v1.2.0-rc2) > > > > > > The KEYS file for signature validation is available at: > > > https://dist.apache.org/repos/dist/release/knox/KEYS > > > > > > Please vote on releasing this package as Apache Knox 1.2.0. > > > The vote is open for the next 72 hours and passes if a majority of at > > > least three +1 Apache Knox PMC votes are cast. > > > > > > [ ] +1 Release this package as Apache Knox 1.2.0 > > > [ ] -1 Do not release this package because... > > > > > > Kevin Risden > > > > > >
[VOTE] Release Apache Knox 1.2.0 RC 2
Release candidate #2 for the Apache Knox 1.2.0 release is available at: https://dist.apache.org/repos/dist/dev/knox/knox-1.2.0/ The release candidate is a zip archive of the sources in: https://git-wip-us.apache.org/repos/asf/knox.git Branch v1.2.0 (git checkout -b v1.2.0) Tag is v1.2.0-rc2 (git checkout -b v1.2.0-rc2) The KEYS file for signature validation is available at: https://dist.apache.org/repos/dist/release/knox/KEYS Please vote on releasing this package as Apache Knox 1.2.0. The vote is open for the next 72 hours and passes if a majority of at least three +1 Apache Knox PMC votes are cast. [ ] +1 Release this package as Apache Knox 1.2.0 [ ] -1 Do not release this package because... Kevin Risden
Re: Knox LDAP group filer is not working
The username you used to login doesn't match the regex you gave.
If you are going to only specify the username you need to modify the regex
to only have one capture group.
Kevin Risden
On Mon, Nov 26, 2018 at 9:08 PM Raja Marimuthu <
raja.marimu...@northbaysolutions.com> wrote:
> Kevin,
>
>
>
> Thank you so much. When I have tried with prinicipalrgeex, I get below
> error
>
>
>
>
>
> User DN : CN=Len,OU=US02P01,OU=mmc_users,DC=ds,DC=nb,DC=com
>
> Group DN : CN=m_powerusers,OU=Applications,OU=Groups,DC=ds,DC=nb,DC=com
>
>
>
> I need to provide. Access only to m_powerusers. group
>
>
>
> Setting 1:
>
>value="dc=ds,dc=nb,dc=com"/>
>
>
>
>
>
>value="((objectclass=person)(memberOf=cn={1},OU=Applications,OU=Groups,DC=ds,DC=nb,DC=com)(sAMAccountName={2}))"/>
>
>
>
>
>
>
>
>
>
>
>
> Setting 2 :
>
>
>
>value="dc=ds,dc=nb,dc=com"/>
>
>
>
>
>
>value="((objectclass=person)(memberOf=cn=m_powerusers,OU=Applications,OU=Groups,DC=ds,DC=nb,DC=com)(sAMAccountName={2}))"/>
>
>
>
>
>
>
>
>
>
>
>
>
>
> User DN : CN=Len,OU=US02P01,OU=mmc_users,DC=ds,DC=nb,DC=com
>
> Group DN : CN=m_powerusers,OU=Applications,OU=Groups,DC=ds,DC=nb,DC=com
>
>
>
> Error :
>
>
>
>
>
>
>
> 2018-11-27 02:02:13,678 WARN authc.AbstractAuthenticator
> (AbstractAuthenticator.java:authenticate(216)) - Authentication failed for
> token submission [org.apache.shiro.authc.UsernamePasswordToken - len,
> rememberMe=false (73.230.13.102)]. Possible unexpected error? (Typical or
> expected login exceptions should extend from AuthenticationException).
>
> java.lang.IllegalArgumentException: Principal len does not match
> (.*?)\\(.*?)
>
> at
> org.apache.knox.gateway.shirorealm.KnoxLdapRealm.matchPrincipal(KnoxLdapRealm.java:658)
>
> at
> org.apache.knox.gateway.shirorealm.KnoxLdapRealm.getUserDn(KnoxLdapRealm.java:681)
>
> at
> org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.getUserDn(KnoxLdapRealm.java:98)
>
> at
> org.apache.shiro.realm.ldap.JndiLdapRealm.getLdapPrincipal(JndiLdapRealm.java:342)
>
> at
> org.apache.shiro.realm.ldap.JndiLdapRealm.queryForAuthenticationInfo(JndiLdapRealm.java:371)
>
> at
> org.apache.shiro.realm.ldap.JndiLdapRealm.doGetAuthenticationInfo(JndiLdapRealm.java:295)
>
> at
> org.apache.knox.gateway.shirorealm.KnoxLdapRealm.doGetAuthenticationInfo(KnoxLdapRealm.java:200)
>
> at
> org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm.doGetAuthenticationInfo(KnoxLdapRealm.java:54)
>
> at
> org.apache.shiro.realm.AuthenticatingRealm.getAuthenticationInfo(AuthenticatingRealm.java:568)
>
> at
> org.apache.shiro.authc.pam.ModularRealmAuthenticator.doSingleRealmAuthentication(ModularRealmAuthenticator.java:180)
>
> at
> org.apache.shiro.authc.pam.ModularRealmAuthenticator.doAuthenticate(ModularRealmAuthenticator.java:267)
>
> at
> org.apache.shiro.authc.AbstractAuthenticator.authenticate(AbstractAuthenticator.java:198)
>
> at
> org.apache.shiro.mgt.AuthenticatingSecurityManager.authenticate(AuthenticatingSecurityManager.java:106)
>
> at
> org.apache.shiro.mgt.DefaultSecurityManager.login(DefaultSecurityManager.java:270)
>
> at
> org.apache.shiro.subject.support.DelegatingSubject.login(DelegatingSubject.java:256)
>
> at
> org.apache.shiro.web.filter.authc.AuthenticatingFilter.executeLogin(AuthenticatingFilter.java:53)
>
> at
> org.apache.shiro.web.filter.authc.BasicHttpAuthenticationFilter.onAccessDenied(BasicHttpAuthenticationFilter.java:190)
>
> at
> org.apache.shiro.web.filter.AccessControlFilter.onAccessDenied(AccessControlFilter.java:133)
>
> at
> org.apache.shiro.web.filter.AccessControlFilter.onPreHandle(AccessControlFilter.java:162)
>
> at
> org.apache.shiro.web.filter.PathMatchingFilter.isFilterChainContinued(PathMatchingFilter.java:203)
>
> at
> org.apache.shiro.web.filter.PathMatchingFilter.preHandle(PathMatchingFilter.java:178)
>
> at
> org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:131)
>
> at
> org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125)
>
> at
> org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66)
>
> at
> org.apache.shiro.web.servlet.AbstractShiroFilter.executeChai
Re: Knox LDAP group filer is not working
>From [1], the userSearchFilter needs to have a reference to the user who is logged in. Basically what you are trying to do with the userSearchFilter is only allow the user to login if the user matches the query. When you do the search filter without a reference to the user who is trying to login you are basically just grabbing all users that match the query. This is not what you want. The principalRegex has capture groups that you can use in the userSearchFilter to build out the query to match for the given username and filter. 1. https://knox.apache.org/books/knox-1-1-0/user-guide.html#Advanced+LDAP+configuration+parameters Kevin Risden On Mon, Nov 26, 2018 at 3:53 PM Raja Marimuthu < raja.marimu...@northbaysolutions.com> wrote: > Anyone have experienced this issue ? Using LDAP group filter with AD ? > > > > Logged in user is different, but it’s. taking first user from the group as > computer userDN , and throws. Null pointer . > > > > > > > > > > > > Raja Marimuthu | Solutions Architect (AWS – Big Data) > > NorthBay Solutions > > Direct: 717-808-6966 > > raja.marimu...@northbaysolutions.com > > > www.northbaysolutions.com > > > > > > *From: *Raja Marimuthu > *Date: *Wednesday, November 14, 2018 at 4:37 PM > *To: *"user@knox.apache.org" > *Subject: *Re: Knox LDAP group filer is not working > > > > Kevin, > > > > I have setup AD and configured in gateway xml, but. I am having this issue… > > > > - 2018-11-14 21:08:26,993 DEBUG knox.gateway > (GatewayFilter.java:doFilter(119)) - Received request: GET /ganglia/ > > 2018-11-14 21:08:27,016 DEBUG knox.gateway > (KnoxLdapRealm.java:getUserDn(718)) - Searching from dc=ds,dc=nb,dc=com > where > (&(objectclass=*)(memberOf=cn=marsh-prd-global-bld-powerusers,OU=Applications,OU=Groups,DC=ds,DC=nb,DC=com)) > scope subtree > > 2018-11-14 21:08:27,022 INFO knox.gateway > (KnoxLdapRealm.java:getUserDn(724)) - Computed userDn: > CN=Len,OU=US02P01,OU=mmc_users,DC=ds,DC=nb,DC=com using ldapSearch for > principal: len > > 2018-11-14 21:08:27,045 ERROR knox.gateway > (AbstractGatewayFilter.java:doFilter(66)) - Failed to execute filter: > javax.servlet.ServletException: java.lang.NullPointerException > > javax.servlet.ServletException: java.lang.NullPointerException > > at > org.apache.shiro.web.servlet.AdviceFilter.cleanup(AdviceFilter.java:196) > > at > org.apache.shiro.web.filter.authc.AuthenticatingFilter.cleanup(AuthenticatingFilter.java:155) > > at > org.apache.shiro.web.servlet.AdviceFilter.doFilterInternal(AdviceFilter.java:148) > > at > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) > > at > org.apache.shiro.web.servlet.ProxiedFilterChain.doFilter(ProxiedFilterChain.java:66) > > at > org.apache.shiro.web.servlet.AbstractShiroFilter.executeChain(AbstractShiroFilter.java:449) > > at > org.apache.shiro.web.servlet.AbstractShiroFilter$1.call(AbstractShiroFilter.java:365) > > at > org.apache.shiro.subject.support.SubjectCallable.doCall(SubjectCallable.java:90) > > at > org.apache.shiro.subject.support.SubjectCallable.call(SubjectCallable.java:83) > > at > org.apache.shiro.subject.support.DelegatingSubject.execute(DelegatingSubject.java:383) > > at > org.apache.shiro.web.servlet.AbstractShiroFilter.doFilterInternal(AbstractShiroFilter.java:362) > > at > org.apache.shiro.web.servlet.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:125) > > > > > > Thanks > > Raja > > > > > > > > > *From:Kevin Risden > Reply-To: > "user@knox.apache.org " > Date: Thursday, November 8, 2018 at 5:41 PM To: > "user@knox.apache.org " > Subject: Re: Knox LDAP group filer is not working* > > > > If you are using the demo LDAP server then memberOf isn't available. It is > an LDAP extension that only exists in AD. Apache DS LDAP doesn't support > virtual attributes. If you want to emulate it you would need to update the > user object with the attributes. > > > Kevin Risden > > > > > > On Thu, Nov 8, 2018 at 5:38 PM Raja Marimuthu < > raja.marimu...@northbaysolutions.com> wrote: > > Kevin, > > > > I have tried. But its. Not working, > > > > Here’s my gateway xml LDAP config > > > > > > > > > > > > > > value="org.apache.hadoop.gateway.shirorealm.KnoxLdapRealm" /> > > value="org.apache.hadoop.gateway.shirorealm.KnoxLdapContextFactory" /> >
Re: Secure Solr
Last I checked Knox and Kerberized Solr worked ok. https://issues.apache.org/jira/browse/KNOX-841 https://github.com/risdenk/knox_solr_testing Is there a link to the documentation you are referring to? I think originally there was issues with pass all headers being used. This is no longer the case as of KNOX-841. Kevin Risden On Mon, Nov 26, 2018 at 3:50 PM Lars Francke wrote: > Hi, > > the documentation says that a Secure Solr is not supported. I did look at > the initial Jira that added Solr support[1] but that doesn't mention it at > all. > > Does anyone have details? Is the documentation correct? What's missing? Is > there a reason why this is missing/hard to implement etc.? > > Any hints are greatly appreciated. > > Thank you, > Lars > > [1] <https://issues.apache.org/jira/browse/KNOX-528> >
Re: Knox LDAP group filer is not working
Assuming you are referring to something like KNOX-1307 [1]? The user search filter you can create can filter by groups depending on what you are trying to do. memberOf is one way for AD to limit users to only ones in a certain group. 1. https://issues.apache.org/jira/browse/KNOX-1307 Kevin Risden On Wed, Nov 7, 2018 at 4:24 PM Raja Marimuthu < raja.marimu...@northbaysolutions.com> wrote: > Hi, > > We are trying to filter users by specific. LDAP groups, tried several > options provided in the documentation : > > https://knox.apache.org/books/knox-1-1-0/user-guide.html#Advanced+LDAP+Authentication > > User Search by Filter > >- userSearchBase (Required) >- userSearchFilter (Required) >- userSearchScope (Optional) >- principalRegex (Optional) > > > > Group filter is supported ? Do we have any working alternative to > filter. Users by group ? > > Thanks > Raja > > >
Re: Knox - Ganglia url rewrite failing
I'm not sure the log segment you shared is the same as the screenshot. There is probably something that isn't being rewritten. Chrome dev tools should be able to show what failed to be retrieved. If you figure out what part of the rewrite.xml/service.xml needs to be adjusted then commenting on KNOX-1343 or even better supplying an updated patch would be great. Kevin Risden On Wed, Nov 7, 2018 at 2:10 PM Raja Marimuthu < raja.marimu...@northbaysolutions.com> wrote: > Kevin, > > Thank you for your quick response. I have used the patch that was > released part of the issue 1343 > > Raja > > > > On Nov 7, 2018, at 2:03 PM, > Kevin Risden > wrote: > > Ganglia isn't part of an Apache Knox release yet. KNOX-1343 [1] has a > patch for it but it isn't in a release. Did you add Ganglia support > yourself? > > 1. https://issues.apache.org/jira/browse/KNOX-1343 > > Kevin Risden > > > On Wed, Nov 7, 2018 at 1:55 PM Raja Marimuthu < > raja.marimu...@northbaysolutions.com> wrote: > >> Hi >> >> I have issue in Knox - Ganglia integration, URL rewrite is failing for >> specific pages. >> >> >> Please see screenshot below >> http://prntscr.com/lfkzys >> >> >> >> 018-11-07 18:51:56,126 DEBUG knox.gateway >> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response >> status: 200 >> 2018-11-07 18:51:56,127 DEBUG knox.gateway >> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound >> response entity content type: text/css >> 2018-11-07 18:51:56,127 DEBUG knox.gateway >> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound >> response entity content type: text/css >> 2018-11-07 18:51:56,164 DEBUG knox.gateway >> (GatewayFilter.java:doFilter(119)) - Received request: GET >> /ganglia/js/jstree/themes/default/style.min.css >> 2018-11-07 18:51:56,165 DEBUG knox.gateway >> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: >> https://18.215.186.148:8553/gateway/sandbox/ganglia/js/jstree/themes/default/style.min.css, >> direction: IN via implicit rule: GANGLIA/ganglia/inbound/js to URL: >> http://ip-172-31-9-170.ec2.internal/ganglia/js/jstree/themes/default/style.min.css >> 2018-11-07 18:51:56,165 DEBUG knox.gateway >> (UrlRewriteProcessor.java:rewrite(161)) - Rewrote URL: >> https://18.215.186.148:8553/gateway/sandbox/ganglia/?r=hour===load_one=ch==false%5B%5D=, >> direction: IN via implicit rule: GANGLIA/ganglia/inbound/root/query to URL: >> http://ip-172-31-9-170.ec2.internal/ganglia/?cs==hour==ch=%255B%255D==load_one=false >> 2018-11-07 18:51:56,166 DEBUG knox.gateway >> (DefaultDispatch.java:executeOutboundRequest(121)) - Dispatch request: GET >> http://ip-172-31-9-170.ec2.internal/ganglia/js/jstree/themes/default/style.min.css?user.name=guest >> 2018-11-07 18:51:56,166 DEBUG knox.gateway >> (DefaultDispatch.java:executeOutboundRequest(134)) - Dispatch response >> status: 200 >> 2018-11-07 18:51:56,167 DEBUG knox.gateway >> (DefaultDispatch.java:getInboundResponseContentType(211)) - Inbound >> response entity content type: text/css >> 2018-11-07 18:52:06,902 DEBUG knox.gateway >> (GatewayFilter.java:doFilter(119)) - Received request: GET / >> 2018-11-07 18:52:06,903 WARN knox.gateway >> (GatewayFilter.java:doFilter(185)) - Failed to match path / >> >> >> >> >> >> >> >> >> >> >
Re: HS2 binary thru knox
The issue here is that HiveServer2 contains state. This is true for any query regardless of Knox. Any JDBC client connecting to HS2 needs to disconnect and reconnect on failure. There are no workarounds for this. With HS2 and Zookeeper, the client must disconnect on failure and reconnect at which point the client will ask ZK for a new server. Knox works the same way in that if there is a failure, the client must disconnect and reconnect. If the client doesn't disconnect/reconnect then there is a chance to get weird errors. Work is not preserved moving between HS2 instances in many cases. Kevin Risden On Tue, Oct 16, 2018 at 4:29 AM rabii lamriq wrote: > Hi Larry > > Thank you for your quick reply, my goal is to create two 4 instances (2 > http and 2 binary) and configure HA + load balancing. > I tested HA of Hiveserver2 through zookeeper, it's work well, but the ZK > doesn't support automatic failover. However, Knox support HA with failover > but without load balancing. > > Is there any solution of configuration to make hybrid solution using both > Knox and ZK, > > also, how to test failover via Knox, I am using DBvisualizer as client to > connect to Hive using Knox. > > Regards > Rabii > > Le lun. 15 oct. 2018 à 14:51, larry mccay a écrit : > >> Hi Rabii - >> >> Proxying HS2 through Knox requires HTTP mode. >> Binary is not supported at all. >> >> thanks, >> >> --larry >> >> On Mon, Oct 15, 2018 at 3:15 AM rabii lamriq wrote: >> >>> Hi >>> >>> Is binary mode of Hiveserver2 supported by knox? If yes, how to >>> configure HA between 4 HS2 instances (2 in http and 2 in binary mode) >>> >>> Best >>> Rabii >>> >>
Re: Knox Securing Hadoop App UI's
It is not a matter of just merging them. The tickets have details about what the current status is. For each of them testing and providing feedback on the ticket would be helpful. Kevin Risden On Thu, Oct 11, 2018 at 3:22 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Ganglia UI - KNOX-1343 > Presto UI - KNOX-1163 > > can these two Jira's get merged? Knox community? > > On Thu, Oct 11, 2018 at 3:18 PM sri hari kali charan Tummala < > kali.tumm...@gmail.com> wrote: > >> thanks Kevin , Hive Server UI is something which needs to be added to the >> list. >> >> >> Thanks >> Sri >> >> On Thu, Oct 11, 2018 at 3:10 PM >> Kevin Risden >> wrote: >> >>> Of the 3 you mentioned, all 3 are not currently supported. Two have been >>> looked at but not yet merged. >>> >>> Ganglia UI - KNOX-1343 >>> Presto UI - KNOX-1163 >>> Hive Server 2 UI - No references in Knox Jira yet. >>> >>> Some references to get started: >>> * >>> https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox >>> * >>> https://cwiki.apache.org/confluence/display/KNOX/2015/12/17/Adding+a+service+to+Apache+Knox >>> * >>> https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox >>> >>> If you need more help implementing rules for a new UI, this mailing list >>> is a great place to ask. >>> >>> Kevin Risden >>> >>> >>> On Thu, Oct 11, 2018 at 3:03 PM sri hari kali charan Tummala < >>> kali.tumm...@gmail.com> wrote: >>> >>>> Hi All, >>>> >>>> can know support or secure below following web UI's ? >>>> >>>> Gaglia UI >>>> Presto UI >>>> Hive Server 2 UI >>>> >>>> are there steps how to achieve this? >>>> >>>> -- >>>> Thanks & Regards >>>> Sri Tummala >>>> >>>> >> >> -- >> Thanks & Regards >> Sri Tummala >> >> > > -- > Thanks & Regards > Sri Tummala > >
Re: Knox Securing Hadoop App UI's
Of the 3 you mentioned, all 3 are not currently supported. Two have been looked at but not yet merged. Ganglia UI - KNOX-1343 Presto UI - KNOX-1163 Hive Server 2 UI - No references in Knox Jira yet. Some references to get started: * https://cwiki.apache.org/confluence/display/KNOX/Proxying+a+UI+using+Knox * https://cwiki.apache.org/confluence/display/KNOX/2015/12/17/Adding+a+service+to+Apache+Knox * https://cwiki.apache.org/confluence/display/KNOX/2017/08/14/Understanding+Rewrite+Rules+for+Apache+Knox If you need more help implementing rules for a new UI, this mailing list is a great place to ask. Kevin Risden On Thu, Oct 11, 2018 at 3:03 PM sri hari kali charan Tummala < kali.tumm...@gmail.com> wrote: > Hi All, > > can know support or secure below following web UI's ? > > Gaglia UI > Presto UI > Hive Server 2 UI > > are there steps how to achieve this? > > -- > Thanks & Regards > Sri Tummala > >
Re: WebHDFS performance issue in Knox
For reference I summarized from this thread and put the results here: KNOX-1221 The write specific performance improvement is here: KNOX-1521 Kevin Risden On Wed, Oct 10, 2018 at 3:48 PM Kevin Risden wrote: > I tried disabling GCM ciphers based on the following information: > * https://www.wowza.com/docs/how-to-improve-ssl-performance-with-java-8 > * > https://stackoverflow.com/questions/25992131/slow-aes-gcm-encryption-and-decryption-with-java-8u20 > > The results for the read were: > * knox ssl no GCM - 1,073,741,824 125MB/s in 8.7s > * knox ssl - 1,073,741,824 54.3MB/s in 20s > > This is a little more than a 2x speedup. There is also information in the > links above that there should be more performance improvements with JDK 9+. > > For the write side slow down, I found an issue with how Knox is handing > the streaming data on writes only. I am looking into fixing this to get the > write performance for HDFS improved. > > Kevin Risden > > > On Wed, Oct 10, 2018 at 1:20 PM David Villarreal < > dvillarr...@hortonworks.com> wrote: > >> I believe Curl has an option of what cipher to use.. You may also be >> able to force it at the server jvm level using >> /jre/lib/security/java.security >> >> >> >> >> >> *From: *Sandeep Moré >> *Reply-To: *"user@knox.apache.org" >> *Date: *Tuesday, October 9, 2018 at 6:39 PM >> *To: *"user@knox.apache.org" >> *Subject: *Re: WebHDFS performance issue in Knox >> >> >> >> I think this would be a good test, worth a try, not sure how we can force >> a certain cipher to be used perhaps a permutation combination of >> >> ssl.include.ciphers, ssl.exclude.ciphers. >> >> >> >> Best, >> >> Sandeep >> >> >> >> >> >> On Tue, Oct 9, 2018 at 5:29 PM David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >> Hi Kevin, >> >> >> >> In my humble opinion, this has to do with cpu processing encryption in >> general based on which cipher being used. Couldn’t the same type of >> principals/improvements (hdfs encryption improvements) be done here for >> let’s say for AES cipher suites? If the main bottleneck here is CPU >> couldn’t you enhance encryption though hardware acceleration and you may >> see better performance numbers? >> >> >> >> https://calomel.org/aesni_ssl_performance.html >> >> >> >> Try forcing a less secure cipher to be used in your environment. Do you >> then see better numbers? >> >> >> >> dav >> >> >> >> >> >> >> >> >> >> >> *From:Kevin Risden > Reply-To: >> "user@knox.apache.org " > > Date: Tuesday, October 9, 2018 at 1:05 PM To: >> "user@knox.apache.org " > > Subject: Re: WebHDFS performance issue in Knox* >> >> >> >> @David - Not sure what you mean since this is SSL/TLS and not related to >> RPC encryption like the two JIRAs that you linked. >> >> @Guang - NP just took some time to sit down and look at it. >> >> >> >> Some preliminary investigation shows this may be the JDK implementation >> of TLS/SSL that is slowing down the read path. I need to dig into it >> further but found a few references showing that Java slowness for TLS/SSL >> affects Jetty. >> >>- >>https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions >>- >>https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements >>- https://webtide.com/conscrypting-native-ssl-for-jetty/ >> >> Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to >> enable conscrypting ( >> https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). >> With that I was able to get read performance on par with non ssl and native >> webhdfs. The write side of the equation still has some performance >> differences that need to be looked at further. >> >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 2:01 PM Guang Yang wrote: >> >> Thanks Kevin conducting such experiment! This is exactly what I saw >> before. It doesn't look right the download speed is 10x slower when >> enabling SSL. >> >> >> >> On Tue, Oct 9, 2018 at 10:40 AM David Villarreal < >> dvillarr...@hortonworks.com> wrote: >> >> I bring this up because HDFS encryption saw an increase in performance. >> >> https://issues.a
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Moving this back to the user list since it was sent without the list:
Question: Is this error coming from the spark history server or the knox
server?
Answer: It is coming from the spark history UI. Here is what I see in the
log for spark history ui.
18/10/10 12:30:29 WARN HttpParser: Header is too large >8192
18/10/10 12:30:29 WARN HttpParser: badMessage: 413 for
HttpChannelOverHttp@2756e900{r=1,c=false,a=IDLE,uri=/?doAs=}
With the above information there is probably nothing that can be done from
Knox. The bad news is that there is no configuration in Spark for this yet.
See the following:
* https://issues.apache.org/jira/browse/SPARK-15090
*
https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/ui/JettyUtils.scala#L337
* Should see something about:
* connector.setRequestHeaderSize(requestHeaderSize);
* connector.setResponseHeaderSize(responseHeaderSize);
Without a configuration to change, the Spark history UI is going to default
to the 8192 header size. A related change was done for most projects to
make that header size configurable.
Kevin Risden
On Tue, Oct 9, 2018 at 8:45 PM Theyaa Matti wrote:
> Hi David,
> Thank you for the quick response. I do have that property set
> with the recommended value in Knox, but the issue still persists only with
> the Spark History UI. The issue appears only when I enable ssl for knox. If
> I turn ssl off in Knox the issue disappears, and if I enable it, it shows
> up right away.
>
> Are there any equivalent properties for Spark History UI? since I
> suspect the issue is with the Spark History UI and not with Knox since all
> the other UIs work fine with Knox.
>
> Best,
>
> Theyaa.
>
>
>
> On Tue, Oct 9, 2018 at 7:11 PM David Villarreal <
> dvillarr...@hortonworks.com> wrote:
>
>> Hi Theyaa,
>>
>> Change the size of gateway.httpserver.requestHeaderBuffer property. I
>> think the default is 8912 (8k) change to 16384. See if that helps.
>>
>> For the second problem Request is a replay (34))] this message is often
>> seen when the timing of one of the servers is off. Make sure you use NTPD
>> on all servers and they are all in sync. If everything is in sync you can
>> work around this issue by turning off krb5 replay cache. With the following
>> parameter
>> -Dsun.security.krb5.rcache=none
>>
>> dav
>>
>>
>> On 10/9/18, 9:01 AM, "Theyaa Matti" wrote:
>>
>> Hi,
>>I am getting this error message "WARN HttpParser: Header is too
>> large
>> >8192" when trying to access the spark history ui through knox. Any
>> idea
>> please?
>>
>> Also when trying to load the executors page, I get : GSS initiate
>> failed
>> [Caused by GSSException: Failure unspecified at GSS-API level
>> (Mechanism level:
>> Request is a replay (34))]
>>
>> when knox is requesting executorspage-template.html
>>
>> appreciate any help here.
>>
>>
>>
Re: WebHDFS performance issue in Knox
@David - Not sure what you mean since this is SSL/TLS and not related to RPC encryption like the two JIRAs that you linked. @Guang - NP just took some time to sit down and look at it. Some preliminary investigation shows this may be the JDK implementation of TLS/SSL that is slowing down the read path. I need to dig into it further but found a few references showing that Java slowness for TLS/SSL affects Jetty. - https://nbsoftsolutions.com/blog/the-cost-of-tls-in-java-and-solutions - https://nbsoftsolutions.com/blog/dropwizard-1-3-upcoming-tls-improvements - https://webtide.com/conscrypting-native-ssl-for-jetty/ Locally testing off a Jetty 9.4 branch (for KNOX-1516), I was able to enable conscrypting ( https://www.eclipse.org/jetty/documentation/9.4.x/configuring-ssl.html#conscrypt). With that I was able to get read performance on par with non ssl and native webhdfs. The write side of the equation still has some performance differences that need to be looked at further. Kevin Risden On Tue, Oct 9, 2018 at 2:01 PM Guang Yang wrote: > Thanks Kevin conducting such experiment! This is exactly what I saw > before. It doesn't look right the download speed is 10x slower when > enabling SSL. > > On Tue, Oct 9, 2018 at 10:40 AM David Villarreal < > dvillarr...@hortonworks.com> wrote: > >> I bring this up because HDFS encryption saw an increase in performance. >> >> https://issues.apache.org/jira/browse/HDFS-6606 >> >> >> >> https://issues.apache.org/jira/browse/HADOOP-10768 >> >> >> >> Maybe Knox can make some enhancements in this area? >> >> >> >> *From: *David Villarreal >> *Date: *Tuesday, October 9, 2018 at 10:34 AM >> *To: *"user@knox.apache.org" >> *Subject: *Re: WebHDFS performance issue in Knox >> >> >> >> Hi Kevin, >> >> Now increase your CPU processing power and show me the numbers. >> >> >> >> Do we support AES-NI optimization with extended CPU instruction set for >> AES hardware acceleration? >> >> libcrypto.so library that supports hardware acceleration, such as >> OpenSSL 1.0.1e. (Many OS versions have an older version of the library that >> does not support AES-NI.) >> >> >> >> >> >> >> >> >> >> >> *From:Kevin Risden > Reply-To: >> "user@knox.apache.org " > > Date: Tuesday, October 9, 2018 at 10:26 AM To: >> "user@knox.apache.org " > > Subject: Re: WebHDFS performance issue in Knox* >> >> >> >> Writes look to have performance impact as well: >> >>- directly to webhdfs - ~2.6 seconds >>- knox no ssl - ~29 seconds >>- knox ssl - ~49.6 seconds >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 12:39 PM Kevin Risden wrote: >> >> If I run two downloads concurrently: >> >> >> >> 1,073,741,824 46.1MB/s in 22s >> >> 1,073,741,824 51.3MB/s in 22s >> >> >> >> So it isn't a limitation of the Knox gateway itself in total bandwidth >> but a per connection limitation somehow. >> >> >> Kevin Risden >> >> >> >> >> >> On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden wrote: >> >> So I was able to reproduce a slowdown with SSL with a pseudo distributed >> HDFS setup on a single node with Knox running on the same node. This was >> setup in Virtualbox on my laptop. >> >> >> >> Rough timings with wget for a 1GB random file: >> >>- directly to webhdfs - 1,073,741,824 252MB/s in 3.8s >>- knox no ssl - 1,073,741,824 264MB/s in 3.6s >>- knox ssl - 1,073,741,824 54.3MB/s in 20s >> >> There is a significant decrease with Knox SSL for some reason. >> >> >> >> Kevin Risden >> >> >> >> >> >> On Sun, Sep 23, 2018 at 8:53 PM larry mccay wrote: >> >> SSL handshake will likely happen at least twice. >> >> Once for the request through Knox to the NN then the redirect from the NN >> to the DN goes all the way back to the client. >> >> So they have to follow the redirect and do the handshake to the DN. >> >> >> >> >> >> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden wrote: >> >> So I found this in the Knox issues list in JIRA: >> >> >> >> https://issues.apache.org/jira/browse/KNOX-1221 >> >> >> >> It sounds familiar in terms of a slowdown when going through Knox. >> >> >> Kevin Risden >> >> >>
Re: WebHDFS performance issue in Knox
If I run two downloads concurrently: 1,073,741,824 46.1MB/s in 22s 1,073,741,824 51.3MB/s in 22s So it isn't a limitation of the Knox gateway itself in total bandwidth but a per connection limitation somehow. Kevin Risden On Tue, Oct 9, 2018 at 12:24 PM Kevin Risden wrote: > So I was able to reproduce a slowdown with SSL with a pseudo distributed > HDFS setup on a single node with Knox running on the same node. This was > setup in Virtualbox on my laptop. > > Rough timings with wget for a 1GB random file: > >- directly to webhdfs - 1,073,741,824 252MB/s in 3.8s >- knox no ssl - 1,073,741,824 264MB/s in 3.6s >- knox ssl - 1,073,741,824 54.3MB/s in 20s > > There is a significant decrease with Knox SSL for some reason. > > Kevin Risden > > > On Sun, Sep 23, 2018 at 8:53 PM larry mccay wrote: > >> SSL handshake will likely happen at least twice. >> Once for the request through Knox to the NN then the redirect from the NN >> to the DN goes all the way back to the client. >> So they have to follow the redirect and do the handshake to the DN. >> >> >> On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden wrote: >> >>> So I found this in the Knox issues list in JIRA: >>> >>> https://issues.apache.org/jira/browse/KNOX-1221 >>> >>> It sounds familiar in terms of a slowdown when going through Knox. >>> >>> Kevin Risden >>> >>> >>> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden >>> wrote: >>> >>>> Hmmm yea curl for a single file should do the handshake once. >>>> >>>> What are the system performance statistics during the SSL vs non SSL >>>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here >>>> if using that. Otherwise watching top may be helpful. It would be help to >>>> determine if the Knox is working harder during the SSL transfer. >>>> >>>> Kevin Risden >>>> >>>> >>>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: >>>> >>>>> I'm just using curl to download a single large file. So I suspect SSL >>>>> handshake just happens once? >>>>> >>>>> On Tue, Sep 11, 2018 at 12:02 PM >>>>> Kevin Risden >>>>> wrote: >>>>> >>>>>> What client are you using to connect Knox? Is this for a single file >>>>>> or a bunch of files? >>>>>> >>>>>> The SSL handshake can be slow if the client doesn't keep the >>>>>> connection open. >>>>>> >>>>>> Kevin Risden >>>>>> >>>>>> On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: >>>>>> >>>>>>> Thanks Larry. But the only difference is this part in my >>>>>>> gateway-site.xml. >>>>>>> >>>>>>> ** >>>>>>> *ssl.enabled* >>>>>>> *false* >>>>>>> *Indicates whether SSL is >>>>>>> enabled.* >>>>>>> ** >>>>>>> >>>>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay >>>>>>> wrote: >>>>>>> >>>>>>>> I really don't think that kind of difference should be expected >>>>>>>> from merely SSL overhead. >>>>>>>> I don't however have any metrics to contradict it either since I do >>>>>>>> not run Knox without SSL. >>>>>>>> >>>>>>>> Given the above, I am struggling coming up with a meaningful >>>>>>>> response to this. :( >>>>>>>> I don't think you should see a 10 fold increase in speed by >>>>>>>> disabling SSL though. >>>>>>>> >>>>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>>>>>>> >>>>>>>>> Any idea guys? >>>>>>>>> >>>>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>>>>>>> >>>>>>>>>> Thanks guys! The issue seems exactly what David pointed out, >>>>>>>>>> which is because of encrypted over SSL. >>>>>>>>>> >>>>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>>>>>>>
Re: Spark History UI Error WARN HttpParser: Header is too large >8192
Please don't send to two different mailing lists at the same time. Removed the dev list. It sounds like you are describing the same issue as KNOX-624 [1]. The two configurations are: gateway.httpserver.requestHeaderBuffer gateway.httpserver.responseHeaderBuffer 1. https://issues.apache.org/jira/browse/KNOX-624 Kevin Risden On Tue, Oct 9, 2018 at 12:01 PM Theyaa Matti wrote: > Hi, >I am getting this error message "WARN HttpParser: Header is too large > >8192" when trying to access the spark history ui through knox. Any idea > please? > > Also when trying to load the executors page, I get : GSS initiate failed > [Caused by GSSException: Failure unspecified at GSS-API level (Mechanism > level: > Request is a replay (34))] > > when knox is requesting executorspage-template.html > > appreciate any help here. >
Re: WebHDFS performance issue in Knox
So I was able to reproduce a slowdown with SSL with a pseudo distributed HDFS setup on a single node with Knox running on the same node. This was setup in Virtualbox on my laptop. Rough timings with wget for a 1GB random file: - directly to webhdfs - 1,073,741,824 252MB/s in 3.8s - knox no ssl - 1,073,741,824 264MB/s in 3.6s - knox ssl - 1,073,741,824 54.3MB/s in 20s There is a significant decrease with Knox SSL for some reason. Kevin Risden On Sun, Sep 23, 2018 at 8:53 PM larry mccay wrote: > SSL handshake will likely happen at least twice. > Once for the request through Knox to the NN then the redirect from the NN > to the DN goes all the way back to the client. > So they have to follow the redirect and do the handshake to the DN. > > > On Sun, Sep 23, 2018 at 8:30 PM Kevin Risden wrote: > >> So I found this in the Knox issues list in JIRA: >> >> https://issues.apache.org/jira/browse/KNOX-1221 >> >> It sounds familiar in terms of a slowdown when going through Knox. >> >> Kevin Risden >> >> >> On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden wrote: >> >>> Hmmm yea curl for a single file should do the handshake once. >>> >>> What are the system performance statistics during the SSL vs non SSL >>> testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here >>> if using that. Otherwise watching top may be helpful. It would be help to >>> determine if the Knox is working harder during the SSL transfer. >>> >>> Kevin Risden >>> >>> >>> On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: >>> >>>> I'm just using curl to download a single large file. So I suspect SSL >>>> handshake just happens once? >>>> >>>> On Tue, Sep 11, 2018 at 12:02 PM >>>> Kevin Risden >>>> wrote: >>>> >>>>> What client are you using to connect Knox? Is this for a single file >>>>> or a bunch of files? >>>>> >>>>> The SSL handshake can be slow if the client doesn't keep the >>>>> connection open. >>>>> >>>>> Kevin Risden >>>>> >>>>> On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: >>>>> >>>>>> Thanks Larry. But the only difference is this part in my >>>>>> gateway-site.xml. >>>>>> >>>>>> ** >>>>>> *ssl.enabled* >>>>>> *false* >>>>>> *Indicates whether SSL is enabled.* >>>>>> ** >>>>>> >>>>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay >>>>>> wrote: >>>>>> >>>>>>> I really don't think that kind of difference should be expected from >>>>>>> merely SSL overhead. >>>>>>> I don't however have any metrics to contradict it either since I do >>>>>>> not run Knox without SSL. >>>>>>> >>>>>>> Given the above, I am struggling coming up with a meaningful >>>>>>> response to this. :( >>>>>>> I don't think you should see a 10 fold increase in speed by >>>>>>> disabling SSL though. >>>>>>> >>>>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>>>>>> >>>>>>>> Any idea guys? >>>>>>>> >>>>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>>>>>> >>>>>>>>> Thanks guys! The issue seems exactly what David pointed out, which >>>>>>>>> is because of encrypted over SSL. >>>>>>>>> >>>>>>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>>>>>> Namenode directly. And with disabling SSL, the speed can reach to >>>>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but >>>>>>>>> it does surprised me with such a difference. Is it normal from your >>>>>>>>> perspective? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Guang >>>>>>>>> >>>>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>>
Re: WebHDFS performance issue in Knox
So I found this in the Knox issues list in JIRA: https://issues.apache.org/jira/browse/KNOX-1221 It sounds familiar in terms of a slowdown when going through Knox. Kevin Risden On Sat, Sep 15, 2018 at 10:17 PM Kevin Risden wrote: > Hmmm yea curl for a single file should do the handshake once. > > What are the system performance statistics during the SSL vs non SSL > testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here > if using that. Otherwise watching top may be helpful. It would be help to > determine if the Knox is working harder during the SSL transfer. > > Kevin Risden > > > On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: > >> I'm just using curl to download a single large file. So I suspect SSL >> handshake just happens once? >> >> On Tue, Sep 11, 2018 at 12:02 PM >> Kevin Risden >> wrote: >> >>> What client are you using to connect Knox? Is this for a single file or >>> a bunch of files? >>> >>> The SSL handshake can be slow if the client doesn't keep the connection >>> open. >>> >>> Kevin Risden >>> >>> On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: >>> >>>> Thanks Larry. But the only difference is this part in my >>>> gateway-site.xml. >>>> >>>> ** >>>> *ssl.enabled* >>>> *false* >>>> *Indicates whether SSL is enabled.* >>>> ** >>>> >>>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay >>>> wrote: >>>> >>>>> I really don't think that kind of difference should be expected from >>>>> merely SSL overhead. >>>>> I don't however have any metrics to contradict it either since I do >>>>> not run Knox without SSL. >>>>> >>>>> Given the above, I am struggling coming up with a meaningful response >>>>> to this. :( >>>>> I don't think you should see a 10 fold increase in speed by disabling >>>>> SSL though. >>>>> >>>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>>>> >>>>>> Any idea guys? >>>>>> >>>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>>>> >>>>>>> Thanks guys! The issue seems exactly what David pointed out, which >>>>>>> is because of encrypted over SSL. >>>>>>> >>>>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>>>> Namenode directly. And with disabling SSL, the speed can reach to >>>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>>>>> significantly to *~40M/s*. I know it's because of encrypted, but it >>>>>>> does surprised me with such a difference. Is it normal from your >>>>>>> perspective? >>>>>>> >>>>>>> Thanks, >>>>>>> Guang >>>>>>> >>>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>>>>> dvillarr...@hortonworks.com> wrote: >>>>>>> >>>>>>>> Hi Guang, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Keep in mind the data is being encrypted over SSL. If you disable >>>>>>>> SSL you will most likely see a very significant boost in throughput. >>>>>>>> Some >>>>>>>> people have used more powerful computers to make encryption quicker. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> David >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *From: *Sean Roberts >>>>>>>> *Reply-To: *"user@knox.apache.org" >>>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM >>>>>>>> *To: *"user@knox.apache.org" >>>>>>>> *Subject: *Re: WebHDFS performance issue in Knox >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Guang – This is somewhat to be expected. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> When you talk to WebHDFS directly, the client can distribute the >>>>>>>> request across many data nodes. Also, you are getting data directly >>>>>>>> from >>>>>>>> the source. >>>>>>>> >>>>>>>> With Knox, all traffic goes through the single Knox host. Knox is >>>>>>>> responsible for fetching from the datanodes and consolidating to send >>>>>>>> to >>>>>>>> you. This means overhead as it’s acting as a middle man, and lower >>>>>>>> network >>>>>>>> capacity since only 1 host is serving data to you. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Also, if running on a cloud provider, the Knox host may be a >>>>>>>> smaller instance size with lower network capacity. >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Sean Roberts >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> *From: *Guang Yang >>>>>>>> *Reply-To: *"user@knox.apache.org" >>>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46 >>>>>>>> *To: *"user@knox.apache.org" >>>>>>>> *Subject: *WebHDFS performance issue in Knox >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a >>>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s. >>>>>>>> However, if we download directly from datanode, the speed is about >>>>>>>> 40M/s at >>>>>>>> least. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Are you guys aware of this problem? Any suggestion? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thanks, >>>>>>>> >>>>>>>> Guang >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>
Re: x-ndjson via Knox
https://github.com/elastic/elasticsearch/issues/25718
Elasticsearch endpoints should take both application/json and
application/x-ndjson. There might be something else going on here. That is
why I asked for the Elasticsearch logs.
It might not be necessary to rewrite the content type.
Kevin Risden
On Fri, Sep 21, 2018 at 1:17 PM rabii lamriq wrote:
> Hi all,
>
> My problem is how to change the content-type=application/json to
> application/x-ndjson by rewriting the header in the knox.
>
> Because grafana send their query with content-type=application/json and
> knox return status 400,
>
> Hower when i reproduce the problem by curl comande, I noticed that if i
> replace json by x-ndjson through knox it wirk well.
>
> Sence i can't change the content-type in grafana, is this possible by
> rewriting the header in knox to replace json by x-ndjson.
>
> Regards
> Rabii
>
> Le ven. 21 sept. 2018 à 18:05,
> Kevin Risden
> a écrit :
>
>> Do you have the error from Elasticsearch logs? It might explain if Knox
>> or Grafana is sending a bad request.
>>
>> Kevin Risden
>>
>>
>> On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré
>> wrote:
>>
>>> I don't see any issues from Knox side, what error do you see in Grafana
>>> ? I think this could be because Grafana is expecting certain content-type.
>>>
>>> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq wrote:
>>>
>>>> Hi Sandeep
>>>>
>>>> thanks for your feedback, below is the deboug log for this curl
>>>>
>>>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>>>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL:
>>>> https://:8443/gateway/default/es5//_msearch,
>>>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>>>> :9200//_msearch
>>>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>>>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>>>> POST http://:9200//_msearch?doAs=
>>>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>>>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>>>> status: 400
>>>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>>>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>>>> character set UTF-8 for entity of type application/json
>>>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>>>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>>>> response entity content type: application/json; charset=UTF-8
>>>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>>>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>>>> /es5//_msearch
>>>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>>>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>>>
>>>> Regards
>>>> Rabii
>>>>
>>>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré a
>>>> écrit :
>>>>
>>>>> Hello Rab,
>>>>>
>>>>> Interesting, do you have the debug logs for this ? they will tell us
>>>>> exactly what is happening.
>>>>>
>>>>> Best,
>>>>> Sandeep
>>>>>
>>>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> From Grafana, I Post multisearch query to Elasticsearch using the
>>>>>> bulk API with the newline delimited JSON (NDJSON) format.
>>>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>>>> like this:
>>>>>>
>>>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>>>> http://host_ES:900/index/_msearch; --data-binary ’
>>>>>> { first json doc}
>>>>>> {second json doc}
>>>>>> ’
>>>>>> when bypassing Knox, The query work well either we made json or
>>>>>> x-ndjson,
>>>>>> However, through Knox gateway, if the content-type=application/json,
>>>>>> the query retun this error of 400:
>>>>>>
>>>>>> HTTP/1.1 400 Bad Request
>>>>>> content-type: application/json
>>>>>>
>>>>>> But when we replace json by x-ndjson like mentioned in the bulk API,
>>>>>> it work well.
>>>>>>
>>>>>> My problem is that Grafana send their query with the
>>>>>> content-type=application/json and not x-ndjson, for this, it is possible
>>>>>> to
>>>>>> make any thing on Knox to solve this problem?
>>>>>>
>>>>>> Regards
>>>>>> Rab
>>>>>>
>>>>>
Re: x-ndjson via Knox
Do you have the error from Elasticsearch logs? It might explain if Knox or
Grafana is sending a bad request.
Kevin Risden
On Fri, Sep 21, 2018 at 12:03 PM Sandeep Moré wrote:
> I don't see any issues from Knox side, what error do you see in Grafana ?
> I think this could be because Grafana is expecting certain content-type.
>
> On Fri, Sep 21, 2018 at 11:37 AM rabii lamriq wrote:
>
>> Hi Sandeep
>>
>> thanks for your feedback, below is the deboug log for this curl
>>
>> 2018-09-18 17:23:23,154 DEBUG hadoop.gateway
>> (UrlRewriteProcessor.java:rewrite(166)) - Rewrote URL:
>> https://:8443/gateway/default/es5//_msearch,
>> direction: IN via explicit rule: ES5/inbound/query to URL: http://
>> :9200//_msearch
>> 2018-09-18 17:23:23,156 DEBUG hadoop.gateway
>> (ElasticsearchDispatch.java:executeOutboundRequest(79)) - Dispatch request:
>> POST http://:9200//_msearch?doAs=
>> 2018-09-18 17:23:23,193 DEBUG hadoop.gateway
>> (ElasticsearchDispatch.java:executeOutboundRequest(95)) - Dispatch response
>> status: 400
>> 2018-09-18 17:23:23,194 DEBUG hadoop.gateway
>> (DefaultDispatch.java:getInboundResponseContentType(202)) - Using explicit
>> character set UTF-8 for entity of type application/json
>> 2018-09-18 17:23:23,195 DEBUG hadoop.gateway
>> (DefaultDispatch.java:getInboundResponseContentType(210)) - Inbound
>> response entity content type: application/json; charset=UTF-8
>> 2018-09-18 17:23:54,519 DEBUG hadoop.gateway
>> (GatewayFilter.java:doFilter(116)) - Received request: POST
>> /es5//_msearch
>> 2018-09-18 17:23:54,521 INFO hadoop.gateway
>> (AclsAuthorizationFilter.java:doFilter(85)) - Access Granted: true
>>
>> Regards
>> Rabii
>>
>> Le ven. 21 sept. 2018 à 17:17, Sandeep Moré a
>> écrit :
>>
>>> Hello Rab,
>>>
>>> Interesting, do you have the debug logs for this ? they will tell us
>>> exactly what is happening.
>>>
>>> Best,
>>> Sandeep
>>>
>>> On Fri, Sep 21, 2018 at 9:56 AM rabii lamriq wrote:
>>>
>>>> Hi All,
>>>>
>>>> From Grafana, I Post multisearch query to Elasticsearch using the bulk
>>>> API with the newline delimited JSON (NDJSON) format.
>>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/search-multi-search.html
>>>> like this:
>>>>
>>>> curl -u myuser -i -H “Content-Type: application/json” -XPOST "
>>>> http://host_ES:900/index/_msearch; --data-binary ’
>>>> { first json doc}
>>>> {second json doc}
>>>> ’
>>>> when bypassing Knox, The query work well either we made json or
>>>> x-ndjson,
>>>> However, through Knox gateway, if the content-type=application/json,
>>>> the query retun this error of 400:
>>>>
>>>> HTTP/1.1 400 Bad Request
>>>> content-type: application/json
>>>>
>>>> But when we replace json by x-ndjson like mentioned in the bulk API, it
>>>> work well.
>>>>
>>>> My problem is that Grafana send their query with the
>>>> content-type=application/json and not x-ndjson, for this, it is possible to
>>>> make any thing on Knox to solve this problem?
>>>>
>>>> Regards
>>>> Rab
>>>>
>>>
Re: WebHDFS performance issue in Knox
Hmmm yea curl for a single file should do the handshake once. What are the system performance statistics during the SSL vs non SSL testing? CPU/memory/disk/etc? Ambari metrics with Grafana would help here if using that. Otherwise watching top may be helpful. It would be help to determine if the Knox is working harder during the SSL transfer. Kevin Risden On Wed, Sep 12, 2018 at 2:52 PM Guang Yang wrote: > I'm just using curl to download a single large file. So I suspect SSL > handshake just happens once? > > On Tue, Sep 11, 2018 at 12:02 PM > Kevin Risden > wrote: > >> What client are you using to connect Knox? Is this for a single file or a >> bunch of files? >> >> The SSL handshake can be slow if the client doesn't keep the connection >> open. >> >> Kevin Risden >> >> On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: >> >>> Thanks Larry. But the only difference is this part in my >>> gateway-site.xml. >>> >>> ** >>> *ssl.enabled* >>> *false* >>> *Indicates whether SSL is enabled.* >>> ** >>> >>> On Tue, Sep 11, 2018 at 11:42 AM, larry mccay wrote: >>> >>>> I really don't think that kind of difference should be expected from >>>> merely SSL overhead. >>>> I don't however have any metrics to contradict it either since I do not >>>> run Knox without SSL. >>>> >>>> Given the above, I am struggling coming up with a meaningful response >>>> to this. :( >>>> I don't think you should see a 10 fold increase in speed by disabling >>>> SSL though. >>>> >>>> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >>>> >>>>> Any idea guys? >>>>> >>>>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>>>> >>>>>> Thanks guys! The issue seems exactly what David pointed out, which is >>>>>> because of encrypted over SSL. >>>>>> >>>>>> Without Knox, the download speed can reach to *400M/s* if I call >>>>>> Namenode directly. And with disabling SSL, the speed can reach to >>>>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>>>> significantly to *~40M/s*. I know it's because of encrypted, but it >>>>>> does surprised me with such a difference. Is it normal from your >>>>>> perspective? >>>>>> >>>>>> Thanks, >>>>>> Guang >>>>>> >>>>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>>>> dvillarr...@hortonworks.com> wrote: >>>>>> >>>>>>> Hi Guang, >>>>>>> >>>>>>> >>>>>>> >>>>>>> Keep in mind the data is being encrypted over SSL. If you disable >>>>>>> SSL you will most likely see a very significant boost in throughput. >>>>>>> Some >>>>>>> people have used more powerful computers to make encryption quicker. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> >>>>>>> >>>>>>> David >>>>>>> >>>>>>> >>>>>>> >>>>>>> *From: *Sean Roberts >>>>>>> *Reply-To: *"user@knox.apache.org" >>>>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM >>>>>>> *To: *"user@knox.apache.org" >>>>>>> *Subject: *Re: WebHDFS performance issue in Knox >>>>>>> >>>>>>> >>>>>>> >>>>>>> Guang – This is somewhat to be expected. >>>>>>> >>>>>>> >>>>>>> >>>>>>> When you talk to WebHDFS directly, the client can distribute the >>>>>>> request across many data nodes. Also, you are getting data directly from >>>>>>> the source. >>>>>>> >>>>>>> With Knox, all traffic goes through the single Knox host. Knox is >>>>>>> responsible for fetching from the datanodes and consolidating to send to >>>>>>> you. This means overhead as it’s acting as a middle man, and lower >>>>>>> network >>>>>>> capacity since only 1 host is serving data to you. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Also, if running on a cloud provider, the Knox host may be a smaller >>>>>>> instance size with lower network capacity. >>>>>>> >>>>>>> -- >>>>>>> >>>>>>> Sean Roberts >>>>>>> >>>>>>> >>>>>>> >>>>>>> *From: *Guang Yang >>>>>>> *Reply-To: *"user@knox.apache.org" >>>>>>> *Date: *Tuesday, 4 September 2018 at 07:46 >>>>>>> *To: *"user@knox.apache.org" >>>>>>> *Subject: *WebHDFS performance issue in Knox >>>>>>> >>>>>>> >>>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> >>>>>>> >>>>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a >>>>>>> file through WebHDFS in Knox, the download speed is just about 11M/s. >>>>>>> However, if we download directly from datanode, the speed is about >>>>>>> 40M/s at >>>>>>> least. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Are you guys aware of this problem? Any suggestion? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thanks, >>>>>>> >>>>>>> Guang >>>>>>> >>>>>> >>>>>> >>>>> >>>
Re: WebHDFS performance issue in Knox
What client are you using to connect Knox? Is this for a single file or a bunch of files? The SSL handshake can be slow if the client doesn't keep the connection open. Kevin Risden On Tue, Sep 11, 2018, 14:51 Guang Yang wrote: > Thanks Larry. But the only difference is this part in my gateway-site.xml. > > ** > *ssl.enabled* > *false* > *Indicates whether SSL is enabled.* > ** > > On Tue, Sep 11, 2018 at 11:42 AM, larry mccay wrote: > >> I really don't think that kind of difference should be expected from >> merely SSL overhead. >> I don't however have any metrics to contradict it either since I do not >> run Knox without SSL. >> >> Given the above, I am struggling coming up with a meaningful response to >> this. :( >> I don't think you should see a 10 fold increase in speed by disabling SSL >> though. >> >> On Tue, Sep 11, 2018 at 2:35 PM Guang Yang wrote: >> >>> Any idea guys? >>> >>> On Mon, Sep 10, 2018 at 3:07 PM, Guang Yang wrote: >>> >>>> Thanks guys! The issue seems exactly what David pointed out, which is >>>> because of encrypted over SSL. >>>> >>>> Without Knox, the download speed can reach to *400M/s* if I call >>>> Namenode directly. And with disabling SSL, the speed can reach to >>>> *~400M/s* as well through Knox. But with SSL, the speed drops >>>> significantly to *~40M/s*. I know it's because of encrypted, but it >>>> does surprised me with such a difference. Is it normal from your >>>> perspective? >>>> >>>> Thanks, >>>> Guang >>>> >>>> On Tue, Sep 4, 2018 at 11:07 AM, David Villarreal < >>>> dvillarr...@hortonworks.com> wrote: >>>> >>>>> Hi Guang, >>>>> >>>>> >>>>> >>>>> Keep in mind the data is being encrypted over SSL. If you disable SSL >>>>> you will most likely see a very significant boost in throughput. Some >>>>> people have used more powerful computers to make encryption quicker. >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> >>>>> >>>>> David >>>>> >>>>> >>>>> >>>>> *From: *Sean Roberts >>>>> *Reply-To: *"user@knox.apache.org" >>>>> *Date: *Tuesday, September 4, 2018 at 1:53 AM >>>>> *To: *"user@knox.apache.org" >>>>> *Subject: *Re: WebHDFS performance issue in Knox >>>>> >>>>> >>>>> >>>>> Guang – This is somewhat to be expected. >>>>> >>>>> >>>>> >>>>> When you talk to WebHDFS directly, the client can distribute the >>>>> request across many data nodes. Also, you are getting data directly from >>>>> the source. >>>>> >>>>> With Knox, all traffic goes through the single Knox host. Knox is >>>>> responsible for fetching from the datanodes and consolidating to send to >>>>> you. This means overhead as it’s acting as a middle man, and lower network >>>>> capacity since only 1 host is serving data to you. >>>>> >>>>> >>>>> >>>>> Also, if running on a cloud provider, the Knox host may be a smaller >>>>> instance size with lower network capacity. >>>>> >>>>> -- >>>>> >>>>> Sean Roberts >>>>> >>>>> >>>>> >>>>> *From: *Guang Yang >>>>> *Reply-To: *"user@knox.apache.org" >>>>> *Date: *Tuesday, 4 September 2018 at 07:46 >>>>> *To: *"user@knox.apache.org" >>>>> *Subject: *WebHDFS performance issue in Knox >>>>> >>>>> >>>>> >>>>> Hi, >>>>> >>>>> >>>>> >>>>> We're using Knox 1.1.0 to proxy WebHDFS request. If we download a file >>>>> through WebHDFS in Knox, the download speed is just about 11M/s. However, >>>>> if we download directly from datanode, the speed is about 40M/s at least. >>>>> >>>>> >>>>> >>>>> Are you guys aware of this problem? Any suggestion? >>>>> >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> Guang >>>>> >>>> >>>> >>> >
Re: Official Apache Knox Docker Image?
No worries Ryan. Created https://issues.apache.org/jira/browse/KNOX-1285 to track this further. Kevin Risden On Mon, Apr 23, 2018 at 7:31 AM, Ryan H <ryan.howell.developm...@gmail.com> wrote: > Hi Kevin, > > I ended up not getting too far with it as I had to put it down to get > another piece of the project out the door with priority. I won't be able to > pick back up with Knox for probably a month or a little longer. I never > opened up the JIRA either (sorry!). The plan is to build out the Knox piece > after the other portion gets "released" as Knox really was a last minute > idea and I didn't want to include it in willy-nilly without having it in a > stable state. The plan is to stay active with Knox, it is just in the > garage for a little bit until we get our initial release out the door. > Anything I can do in the meantime to help (very limited time at this point)? > > Cheers, > > Ryan > > On Sun, Apr 22, 2018 at 10:14 PM, Kevin Risden <kris...@apache.org> wrote: > >> Ryan - Any luck with Knox and Docker? Would be a good JIRA. >> >> Kevin Risden >> >> On Sun, Mar 25, 2018 at 9:19 PM, Ryan H <ryan.howell.development@gmail >> .com> wrote: >> >>> Awesome, thanks for the links. Next time I have some extra free time I >>> will take a look at the process for feature requests and see about opening >>> one for an official image. It'd definitely be a nice-to-have. >>> >>> Cheers, >>> >>> Ryan >>> >>> On Sun, Mar 25, 2018 at 7:57 PM, Kevin Risden <kris...@apache.org> >>> wrote: >>> >>>> I have a few examples but none that I would use on a regular basis >>>> outside of testing. >>>> >>>> * https://github.com/risdenk/knox_solr_testing >>>> * https://github.com/risdenk/knox_nifi_testing >>>> * https://github.com/risdenk/knox_livy_testing >>>> >>>> Maybe they will help get you over whatever hump you are stuck on. >>>> >>>> I haven't seen any Knox JIRAs about a docker image, but it definitely >>>> could be useful. >>>> >>>> Kevin Risden >>>> >>>> On Sun, Mar 25, 2018 at 6:26 PM, Ryan H <ryan.howell.development@gmail >>>> .com> wrote: >>>> >>>>> Hi All, >>>>> >>>>> Are there any plans for an official Knox Docker image? Just curious >>>>> about it. I've searched around a bit to see what was out there and there >>>>> isn't much in way of Knox images. I've been wrestling with getting one >>>>> built, but I'm no Docker pro and can't seem to get one going. Just >>>>> wondering if there were plans to include one as a part of the build or not >>>>> at some point. >>>>> >>>>> Cheers, >>>>> >>>>> Ryan >>>>> >>>> >>>> >>> >> >
Re: Official Apache Knox Docker Image?
Ryan - Any luck with Knox and Docker? Would be a good JIRA. Kevin Risden On Sun, Mar 25, 2018 at 9:19 PM, Ryan H <ryan.howell.developm...@gmail.com> wrote: > Awesome, thanks for the links. Next time I have some extra free time I > will take a look at the process for feature requests and see about opening > one for an official image. It'd definitely be a nice-to-have. > > Cheers, > > Ryan > > On Sun, Mar 25, 2018 at 7:57 PM, Kevin Risden <kris...@apache.org> wrote: > >> I have a few examples but none that I would use on a regular basis >> outside of testing. >> >> * https://github.com/risdenk/knox_solr_testing >> * https://github.com/risdenk/knox_nifi_testing >> * https://github.com/risdenk/knox_livy_testing >> >> Maybe they will help get you over whatever hump you are stuck on. >> >> I haven't seen any Knox JIRAs about a docker image, but it definitely >> could be useful. >> >> Kevin Risden >> >> On Sun, Mar 25, 2018 at 6:26 PM, Ryan H <ryan.howell.development@gmail >> .com> wrote: >> >>> Hi All, >>> >>> Are there any plans for an official Knox Docker image? Just curious >>> about it. I've searched around a bit to see what was out there and there >>> isn't much in way of Knox images. I've been wrestling with getting one >>> built, but I'm no Docker pro and can't seem to get one going. Just >>> wondering if there were plans to include one as a part of the build or not >>> at some point. >>> >>> Cheers, >>> >>> Ryan >>> >> >> >
Re: turn off debug logging from org.apache.http.wire
Are you using Ambari or deploying Knox manually?
If you using Ambari, then Ambari will force overwrite the log4j configs
during a restart. You must update the log4j settings in Ambari. Another
option if using Ambari is to find the debug setting and set it to false (I
don't have a cluster in front of me so can't look up the setting).
Kevin Risden
On Sun, Apr 15, 2018 at 10:56 AM, Wei Han <wei...@uber.com> wrote:
> Interesting. Thanks Larry. I'll dig more on my side.
>
> On Sun, Apr 15, 2018 at 4:54 AM, larry mccay <lmc...@apache.org> wrote:
>
>> No, I cannot reproduce it.
>> If you are modifying the correct gateway-log4j.properties and restarting
>> the server you should not see that.
>>
>> In fact, turning on DEBUG for wire via:
>> log4j.logger.org.apache.http.wire=DEBUG
>>
>> Doesn't result in output in gateway.out for me but instead gateway.log
>> and turning it on and off certainly works for me.
>>
>> You may have enabled TRACE logging if you are seeing body content - those
>> settings are like the following:
>>
>> #log4j.logger.org.apache.knox.gateway.access=TRACE,httpaccess
>> #log4j.additivity.org.apache.knox.gateway.access=false
>>
>> #log4j.logger.org.apache.knox.gateway.http=TRACE,httpserver
>> #log4j.additivity.org.apache.knox.gateway.http=false
>> ##log4j.logger.org.apache.knox.gateway.http.request.headers=OFF
>> ##log4j.logger.org.apache.knox.gateway.http.response.headers=OFF
>> ##log4j.logger.org.apache.knox.gateway.http.request.body=OFF
>> ##log4j.logger.org.apache.knox.gateway.http.response.body=OFF
>>
>> I suggest you back up to the gateway-log4j.properties from the original
>> install and remove any other log4j config that you may have elsewhere.
>>
>> On Sun, Apr 15, 2018 at 1:58 AM, Wei Han <wei...@uber.com> wrote:
>>
>>> Hi Larry - Thanks a lot for getting back to me.
>>>
>>> Yes I made sure all DEBUG level is turned off in my gateway-log4j.properties
>>> file, but that doesn't seem to be working. I also tried to explicitly
>>> set log4j.logger.httpclient.wire.header to WARN (as suggested in post
>>> <https://stackoverflow.com/questions/4915414/disable-httpclient-logging>),
>>> but that also didn't help.
>>>
>>> Actually it's very easy to reproduce this(at least on my side). If you
>>> call knox with a webhdfs request (like
>>> webhdfs/v1/tmp/weihan/small.txt?op=OPEN),
>>> you should be able to see a bunch of below logs in gateway.out. In fact it
>>> outputs the actual content on the wire(security hole?)
>>>
>>> 06:52:49.751 [qtp1473205473-61] DEBUG org.apache.http.wire -
>>> http-outgoing-2 << "[0x0][0x0
>>>
>>> Let me know if you're able to repro this.
>>>
>>> Thanks.
>>>
>>> On Sat, Apr 14, 2018 at 7:11 AM, larry mccay <lmc...@apache.org> wrote:
>>>
>>>> Hi Wei -
>>>>
>>>> If you look at your gateway-log4j.properties file, you should see
>>>> something like the following near the top:
>>>>
>>>> app.log.dir=${launcher.dir}/../logs
>>>> app.log.file=${launcher.name}.log
>>>> app.audit.file=${launcher.name}-audit.log
>>>>
>>>> log4j.rootLogger=ERROR, drfa
>>>>
>>>> log4j.logger.org.apache.knox.gateway=INFO
>>>> #log4j.logger.org.apache.knox.gateway=DEBUG
>>>>
>>>> #log4j.logger.org.eclipse.jetty=DEBUG
>>>> #log4j.logger.org.apache.shiro=DEBUG
>>>> #log4j.logger.org.apache.http=DEBUG
>>>> #log4j.logger.org.apache.http.client=DEBUG
>>>> #log4j.logger.org.apache.http.headers=DEBUG
>>>> #log4j.logger.org.apache.http.wire=DEBUG
>>>>
>>>> Note that all of the DEBUG settings are commented out.
>>>> Also note that the rootLogger is set to ERROR and not DEBUG.
>>>>
>>>> Can you compare and share with us what yours are set to?
>>>>
>>>> thanks,
>>>>
>>>> --larry
>>>>
>>>> On Sat, Apr 14, 2018 at 2:56 AM, Wei Han <wei...@uber.com> wrote:
>>>>
>>>>> Hi Knox experts -
>>>>>
>>>>> Has anyone successfully turn off the debug loggings
>>>>> from org.apache.http.wire, like below?
>>>>>
>>>>> 06:52:49.751 [qtp1473205473-61] DEBUG org.apache.http.wire -
>>>>> http-outgoing-2 << "[0x0][0x0
>>>>>
>>>>> I've tried almost everything in this stackoverflow post
>>>>> <https://stackoverflow.com/questions/4915414/disable-httpclient-logging>,
>>>>> but my gateway.out is still filled with all the debug logs from
>>>>> httpclient.
>>>>>
>>>>> Any tip would be appreciated.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: Official Apache Knox Docker Image?
I have a few examples but none that I would use on a regular basis outside of testing. * https://github.com/risdenk/knox_solr_testing * https://github.com/risdenk/knox_nifi_testing * https://github.com/risdenk/knox_livy_testing Maybe they will help get you over whatever hump you are stuck on. I haven't seen any Knox JIRAs about a docker image, but it definitely could be useful. Kevin Risden On Sun, Mar 25, 2018 at 6:26 PM, Ryan H <ryan.howell.developm...@gmail.com> wrote: > Hi All, > > Are there any plans for an official Knox Docker image? Just curious about > it. I've searched around a bit to see what was out there and there isn't > much in way of Knox images. I've been wrestling with getting one built, but > I'm no Docker pro and can't seem to get one going. Just wondering if there > were plans to include one as a part of the build or not at some point. > > Cheers, > > Ryan >
Re: Knox SSL with NiFi SSLPeerUnverifiedException
Digging into the NiFi TLS toolkit and SubjectAlternativeName for IP addresses. I checked the NiFi TLS toolkit code for SubjectAlternativeName and found this: https://github.com/apache/nifi/blob/master/nifi-toolkit/nifi-toolkit-tls/src/main/java/org/apache/nifi/toolkit/tls/util/TlsHelper.java#L226 namesList.add(new GeneralName(GeneralName.dNSName, cn)); Since the type is "GeneralName.dNSName" that means that it is assuming a DNSName and not an IP address. If the NiFi TLS toolkit supported IP addresses for SubjectAlternativeName then it would have to be "GeneralName.iPAddress" http://javadox.com/org.bouncycastle/bcprov-jdk15on/1.51/org/bouncycastle/asn1/x509/GeneralName.html#iPAddress Again the error message is confusing but think this could be the cause of the issue since the subject alternative name is being treated as a DNSName and not an IP address like you think it is. Kevin Risden On Tue, Mar 13, 2018 at 8:46 PM, Kevin Risden <kris...@apache.org> wrote: > Extracting my reply from the other thread here since this has a clearly > identified error message: > > *2 way SSL on* > - SSLPeerUnverifiedException: Certificate for doesn't match any > of the subject alternative names: [NIFI-IP] > > This error is most likely occurring because you are using IP addresses. > SSL certificate CN=ip_address I'm not sure is 100% valid. The debug output > you have above has one problem though where SubjectAlternativeName has > DNSName=ip_address. Instead this should have IP=ip_address. I don't know if > the nifi SSL toolkit can generate a SubjectAlternativeName with > IP=ip_address. It might be assuming you provide a hostname. The error > output is confusing but the first NIFI-IP is the certificate common name > and the second is most likely a DNSName and NOT an ip address like you > think it is. > > *SSL - Hostnames vs IPs* > This is super general but I have run into issues with hostnames vs IPs for > SSL before. SSL was built for hostnames. CN=HOSTNAME is the original way to > configure an SSL certificate. If you put the FQDN in the CN then you MUST > use the FQDN when trying to connect to that server. You can't just use the > short hostname since it won't match. SubjectAlternativeName is an extension > of SSL certificates that allow for easier configuration of certificates > instead of always using a hostname in the CN. A single certificate could > now be used for multiple hostnames and in some cases IPs. > SubjectAlternativeName defaults to DNSName when you add to it. I have not > seen this work with pure IP addresses. If you want to use IP addresses > inside SubjectAlternativeName, you must have the prefix "IP:" for example > IP:127.0.0.1. This differentiates it from DNSName. > > > > Kevin Risden > > On Sat, Mar 10, 2018 at 11:17 AM, Ryan H <ryan.howell.development@ > gmail.com> wrote: > >> Hi All, >> >> I am starting a new thread on this as it is a different error on the last >> thread I was on and hopefully to catch the attention of additional persons >> that may have expertise with this (sorry for the many emails on this). >> >> I have a NiFi secure cluster that I am using Apache Knox to proxy all >> traffic to. This set up has been tested first as an insecure cluster, which >> works fine. Switching to the secure cluster though has been a nightmare >> with an error that I cannot get past. >> >> *The original error was:* >> javax.net.ssl.SSLHandshakeException: >> sun.security.validator.ValidatorException: >> PKIX path building failed: >> sun.security.provider.certpath.SunCertPathBuilderException: >> unable to find valid certification path to requested target >> >> Adding the keystore.jks and truststore.jks obtained from the NiFi TLS >> Toolkit to the Knox gateway.jks keystore got rid of the above error, but >> then resulted in a new one. >> >> *This is the new error that I am now stuck at:* >> javax.net.ssl.SSLPeerUnverifiedException: Certificate for >> doesn't match any of the subject alternative names: [ >> NIFI-IP-ADDR] >> >> The error seems to be misleading as the NIFI-IP-ADDR is in fact in the >> SANS within the certificate, as well as the error itself. >> >> On the Knox side, "twoWaySsl" is enabled both in the sandbox.xml as well >> as the service.xml (defined in the NIFI service). On the NiFi side, the >> Knox identity has an entry in the authorizers.xml file as a User Identity >> (I did try adding it as a Node Identity as well just to try something new). >> >> My last thoughts on this were that it potentially had something to do >> with either the version of Java or openssl that I am running, but >> unfortu
Re: Knox SSL with NiFi SSLPeerUnverifiedException
Extracting my reply from the other thread here since this has a clearly identified error message: *2 way SSL on* - SSLPeerUnverifiedException: Certificate for doesn't match any of the subject alternative names: [NIFI-IP] This error is most likely occurring because you are using IP addresses. SSL certificate CN=ip_address I'm not sure is 100% valid. The debug output you have above has one problem though where SubjectAlternativeName has DNSName=ip_address. Instead this should have IP=ip_address. I don't know if the nifi SSL toolkit can generate a SubjectAlternativeName with IP=ip_address. It might be assuming you provide a hostname. The error output is confusing but the first NIFI-IP is the certificate common name and the second is most likely a DNSName and NOT an ip address like you think it is. *SSL - Hostnames vs IPs* This is super general but I have run into issues with hostnames vs IPs for SSL before. SSL was built for hostnames. CN=HOSTNAME is the original way to configure an SSL certificate. If you put the FQDN in the CN then you MUST use the FQDN when trying to connect to that server. You can't just use the short hostname since it won't match. SubjectAlternativeName is an extension of SSL certificates that allow for easier configuration of certificates instead of always using a hostname in the CN. A single certificate could now be used for multiple hostnames and in some cases IPs. SubjectAlternativeName defaults to DNSName when you add to it. I have not seen this work with pure IP addresses. If you want to use IP addresses inside SubjectAlternativeName, you must have the prefix "IP:" for example IP:127.0.0.1. This differentiates it from DNSName. Kevin Risden On Sat, Mar 10, 2018 at 11:17 AM, Ryan H <ryan.howell.developm...@gmail.com> wrote: > Hi All, > > I am starting a new thread on this as it is a different error on the last > thread I was on and hopefully to catch the attention of additional persons > that may have expertise with this (sorry for the many emails on this). > > I have a NiFi secure cluster that I am using Apache Knox to proxy all > traffic to. This set up has been tested first as an insecure cluster, which > works fine. Switching to the secure cluster though has been a nightmare > with an error that I cannot get past. > > *The original error was:* > javax.net.ssl.SSLHandshakeException: > sun.security.validator.ValidatorException: > PKIX path building failed: > sun.security.provider.certpath.SunCertPathBuilderException: > unable to find valid certification path to requested target > > Adding the keystore.jks and truststore.jks obtained from the NiFi TLS > Toolkit to the Knox gateway.jks keystore got rid of the above error, but > then resulted in a new one. > > *This is the new error that I am now stuck at:* > javax.net.ssl.SSLPeerUnverifiedException: Certificate for > doesn't match any of the subject alternative names: [NIFI-IP-ADDR] > > The error seems to be misleading as the NIFI-IP-ADDR is in fact in the > SANS within the certificate, as well as the error itself. > > On the Knox side, "twoWaySsl" is enabled both in the sandbox.xml as well > as the service.xml (defined in the NIFI service). On the NiFi side, the > Knox identity has an entry in the authorizers.xml file as a User Identity > (I did try adding it as a Node Identity as well just to try something new). > > My last thoughts on this were that it potentially had something to do with > either the version of Java or openssl that I am running, but unfortunately > its not much help as I am not sure what would work vs what wouldn't work. > The other thought would be that this may potentially be an issue with the > TLS Toolkit, but it works just fine for the cluster nodes. > > Can anyone verify that this setup works? Has anyone set this up with Knox > proxying NiFi over SSL using the self-signed certs via TLS Toolkit (if so, > can you share details on your set up)? I have to missing something here, > but out of ideas on what... > > NiFi Version: 1.6.0 > NiFi TLS Toolkit Version: 1.6.0 > Knox Version: 1.1.0 > > Java Version: > openjdk version "1.8.0_141" > OpenJDK Runtime Environment (build 1.8.0_141-8u141-b15-3~14.04-b15) > OpenJDK 64-Bit Server VM (build 25.141-b15, mixed mode) > > OS: > Distributor ID: Ubuntu > Description: Ubuntu 14.04.5 LTS > Release: 14.04 > Codename: trusty > > OpenSSL Version: > OpenSSL 1.0.1f 6 Jan 2014 > >
.Net WebHDFS Client (with and without Knox)
TL;DR .Net WebHDFS client that works with and without Apache Knox. * Nuget: https://www.nuget.org/packages/WebHDFS.Client/ * Source & Example: https://github.com/risdenk/webhdfs-dotnet https://risdenk.github.io/2017/12/19/webhdfs-dotnet-client.html I wrote the above blog to explain a new .Net WebHDFS client that I wrote. It was primarily built to work with Knox but should work without Knox as well. I'm hoping this is helpful to others since I couldn't find an existing .Net library out there that supported Knox (basic auth for now). I'm definitely open to improvements/suggestions/etc as well. Kevin Risden
Re: Knox Audit Logging - duplicate correlation ids
Responding to myself to add some more detail after digging deeper: More evidence from duplicate correlation ids for the following cases: - this affects different services (same correlation id, different services being called) - multiple successful individual requests with same correlation id I'll try anonymize them and share to this thread in the next few days. We had a few theories that we are slowly working through - Correlation id generation is returning duplicates? - Status: Basically disproven - https://github.com/apache/knox/blob/master/gateway-server/src/main/java/org/apache/hadoop/gateway/filter/CorrelationHandler.java#L37 - Wrote a small Java program to test this. - UUID.randomUUID().toString in multiple threads - No duplicates returned for 100+ threads and 1 ids generated quickly - Pretty sure the random number generator is ok - Requests from same IP address = same correlation id? - Status: Disproven - This doesn't make sense for the different services - Different users would be hitting different services (hive vs hbase) - If it only occurs with authentication failures could be the system recycling the ID? - Status: Not explored yet - Active conversations require a unique ID, so it might be that auth fail 401 = conversation over and UUID recycling. - Not sure how to test this right now - CorrelationContext or CorrelationService or CorrelationHandler is getting confused? - The filter is adding the same id multiple times? - Asynchronous calls are being handled differently? As of right now we don't have a smoking gun as to what is happening, but I would guess there is something in Knox causing the duplicates to happen. Any ideas on where to look next would be appreciated. Kevin Risden On Tue, Oct 17, 2017 at 4:35 PM, Kevin Risden <kris...@apache.org> wrote: > From the Knox User list thread: "Multiple topology audit logging", it came > to my attention that Knox seems to be logging duplicate correlation ids. > Separating out the topic specifically here to dig a bit deeper. > > While looking at our Knox audit logs (Knox 0.9 on HDP 2.5) the > "correlation id" doesn't seem to be unique across requests. Is this to be > expected? Here is a snippet (anonymized): > > grep 7557c91b-2a48-4e09-aefc-44e9892372da /var/knox/gateway-audit.log >> > > >> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >> 892372da|audit|WEBHBASEaccess|uri|/gateway/HADOOPTEST >> /hbase/hbase/NAMESPACE1:TABLE1/ID1//|unavailable|Request method: GET >> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >> 892372da|audit|WEBHBASE|USER1|||authentication|uri|/gateway/ >> HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success| >> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >> 892372da|audit|WEBHBASE|USER1|||authentication|uri|/gateway/ >> HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|Groups: >> [] >> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >> 892372da|audit|WEBHBASE|USER1|||dispatch|uri|http://WEBHBASE >> .example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1& >> row=ID2%2Fd%3Araw|unavailable|Request >> <http://webhbase.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw%7Cunavailable%7CRequest> >> method: >> GET >> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >> 892372da|audit|WEBHBASE|USER1|||dispatch|uri|http://WEBHBASE >> .example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1& >> row=ID2%2Fd%3Araw|success|Response >> <http://webhbase.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw%7Csuccess%7CResponse> >> status: >> 200 >> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >> 892372da|audit|WEBHBASE|USER1|||access|uri|/gateway/HADOOPPR >> OD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|Response >> status: 200 >> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >> 892372da|audit|WEBHBASEauthentication|principal|USER2|failure|LDAP >> authentication failed. >> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >> 892372da|audit|WEBHBASEaccess|uri|/gateway/HADOOPTEST >> /hbase/hbase/NAMESPACE1:TABLE2/ID1//|success|Response status: 401 > > > The things to highlight here for the same correlation id: > * different topologies are being used > * different uris are being used > * different users are being used > > Some of the things that we have configured that could impact results: > * authentication caching > * multiple Knox servers > * load balancer in front of Knox > > I would be surprised if those things would make one Knox node (it c
Knox Audit Logging - duplicate correlation ids
>From the Knox User list thread: "Multiple topology audit logging", it came to my attention that Knox seems to be logging duplicate correlation ids. Separating out the topic specifically here to dig a bit deeper. While looking at our Knox audit logs (Knox 0.9 on HDP 2.5) the "correlation id" doesn't seem to be unique across requests. Is this to be expected? Here is a snippet (anonymized): grep 7557c91b-2a48-4e09-aefc-44e9892372da /var/knox/gateway-audit.log > > 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE||| > |access|uri|/gateway/HADOOPTEST/hbase/hbase/NAMESPACE1:TABLE1/ID1//|unavailable|Request > method: GET > 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE| > USER1|||authentication|uri|/gateway/HADOOPPROD/hbase/ > NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success| > 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE| > USER1|||authentication|uri|/gateway/HADOOPPROD/hbase/ > NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|Groups: [] > 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE| > USER1|||dispatch|uri|http://WEBHBASE.example.com:8084/ > NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw| > unavailable|Request > <http://webhbase.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw%7Cunavailable%7CRequest> > method: > GET > 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE| > USER1|||dispatch|uri|http://WEBHBASE.example.com:8084/ > NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw|success|Response > <http://webhbase.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw%7Csuccess%7CResponse> > status: > 200 > 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE| > USER1|||access|uri|/gateway/HADOOPPROD/hbase/NAMESPACE2: > TABLE2/multiget?row=ID2%2fd%3araw&|success|Response status: 200 > 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE||| > |authentication|principal|USER2|failure|LDAP authentication failed. > 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE||| > |access|uri|/gateway/HADOOPTEST/hbase/hbase/NAMESPACE1:TABLE2/ID1//|success|Response > status: 401 The things to highlight here for the same correlation id: * different topologies are being used * different uris are being used * different users are being used Some of the things that we have configured that could impact results: * authentication caching * multiple Knox servers * load balancer in front of Knox I would be surprised if those things would make one Knox node (it could affect our other Knox nodes haven't looked yet) have duplicate correlation ids. >From Larry Mccay: > Interesting... > > I have not seen this before at all. > I don't think there is anyway for those things to affect it either - > especially the HA related things. > Those entries are all a single request that will not go to different hosts > - unless there is are redirects involved and then it should become a > separate correlation id. > > We should dig into the creation of the id to see what could cause it. > >From myself: > Yea that is what I was afraid of :) I looked a bit further and it isn't > every request that has duplicates. The ones I have found so far are from > LDAP authentication failures. There could very well be others since I've > only been looking at authentication failures. > I'll see if my team can dig some more information up. I'm hoping to get a > few more people involved with Apache Knox. It will probably take a few days > to dig further. Kevin Risden
Re: Multiple topology audit logging
Yea that is what I was afraid of :) I looked a bit further and it isn't every request that has duplicates. The ones I have found so far are from LDAP authentication failures. There could very well be others since I've only been looking at authentication failures. I'll see if my team can dig some more information up. I'm hoping to get a few more people involved with Apache Knox. It will probably take a few days to dig further. On Tue, Oct 10, 2017 at 3:57 PM, larry mccay <lmc...@apache.org> wrote: > Interesting... > > I have not seen this before at all. > I don't think there is anyway for those things to affect it either - > especially the HA related things. > Those entries are all a single request that will not go to different hosts > - unless there is are redirects involved and then it should become a > separate correlation id. > > We should dig into the creation of the id to see what could cause it. > > > On Tue, Oct 10, 2017 at 4:38 PM, Kevin Risden <kris...@apache.org> wrote: > >> Thanks Larry the correlation id looks like it will work for our case. >> >> However, I found an interesting anomaly while looking at our logs (Knox >> 0.9 on HDP 2.5 btw). The "correlation id" doesn't seem to be unique across >> requests. Is this to be expected? Here is a snippet (anonymized): >> >> grep 7557c91b-2a48-4e09-aefc-44e9892372da /var/knox/gateway-audit.log >>> >> >> >>> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >>> 892372da|audit|WEBHBASEaccess|uri|/gateway/HADOOPTEST >>> /hbase/hbase/NAMESPACE1:TABLE1/ID1//|unavailable|Request method: GET >>> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >>> 892372da|audit|WEBHBASE|USER1|||authentication|uri|/gateway/ >>> HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success| >>> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >>> 892372da|audit|WEBHBASE|USER1|||authentication|uri|/gateway/ >>> HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|Groups: >>> [] >>> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >>> 892372da|audit|WEBHBASE|USER1|||dispatch|uri|http://WEBHBASE >>> .example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1& >>> row=ID2%2Fd%3Araw|unavailable|Request >>> <http://WEBHBASE.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw%7Cunavailable%7CRequest> >>> method: GET >>> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >>> 892372da|audit|WEBHBASE|USER1|||dispatch|uri|http://WEBHBASE >>> .example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1& >>> row=ID2%2Fd%3Araw|success|Response >>> <http://WEBHBASE.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw%7Csuccess%7CResponse> >>> status: 200 >>> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >>> 892372da|audit|WEBHBASE|USER1|||access|uri|/gateway/HADOOPPR >>> OD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|Response >>> status: 200 >>> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >>> 892372da|audit|WEBHBASEauthentication|principal|USER2|failure|LDAP >>> authentication failed. >>> 17/10/10 12:50:09 ||7557c91b-2a48-4e09-aefc-44e9 >>> 892372da|audit|WEBHBASEaccess|uri|/gateway/HADOOPTEST >>> /hbase/hbase/NAMESPACE1:TABLE2/ID1//|success|Response status: 401 >> >> >> The things to highlight here for the same correlation id: >> * different topologies are being used >> * different uris are being used >> * different users are being used >> >> >> Some of the things that we have configured that could impact results: >> * authentication caching >> * multiple Knox servers >> * load balancer in front of Knox >> >> I would be surprised if those things would make one Knox node (it could >> affect our other Knox nodes haven't looked yet) have duplicate correlation >> ids. >> >> On Tue, Oct 10, 2017 at 9:54 AM, larry mccay <larry.mc...@gmail.com> >> wrote: >> >>> If there is no access entry then that definitely needs to be fixed. >>> Please file a JIRA for 0.14.0 if that is the case. >>> >>> On Tue, Oct 10, 2017 at 10:47 AM, Kevin Risden <kris...@apache.org> >>> wrote: >>> >>>> Thanks Larry for the quick response. I'll have to go back and double >>>> check if the access and uri are in there. If so I must have missed it the >>>> first time through. Correlating across the GUID should work for our quick >>>> debugging. >>>> >&
Re: Multiple topology audit logging
Thanks Larry the correlation id looks like it will work for our case. However, I found an interesting anomaly while looking at our logs (Knox 0.9 on HDP 2.5 btw). The "correlation id" doesn't seem to be unique across requests. Is this to be expected? Here is a snippet (anonymized): grep 7557c91b-2a48-4e09-aefc-44e9892372da /var/knox/gateway-audit.log > > 17/10/10 12:50:09 > ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASEaccess|uri|/gateway/HADOOPTEST/hbase/hbase/NAMESPACE1:TABLE1/ID1//|unavailable|Request > method: GET > 17/10/10 12:50:09 > ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||authentication|uri|/gateway/HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success| > 17/10/10 12:50:09 > ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||authentication|uri|/gateway/HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|Groups: > [] > 17/10/10 12:50:09 > ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||dispatch|uri| > http://WEBHBASE.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw|unavailable|Request > method: GET > 17/10/10 12:50:09 > ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||dispatch|uri| > http://WEBHBASE.example.com:8084/NAMESPACE2:TABLE2/multiget?doAs=USER1=ID2%2Fd%3Araw|success|Response > status: 200 > 17/10/10 12:50:09 > ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASE|USER1|||access|uri|/gateway/HADOOPPROD/hbase/NAMESPACE2:TABLE2/multiget?row=ID2%2fd%3araw&|success|Response > status: 200 > 17/10/10 12:50:09 > ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASEauthentication|principal|USER2|failure|LDAP > authentication failed. > 17/10/10 12:50:09 > ||7557c91b-2a48-4e09-aefc-44e9892372da|audit|WEBHBASEaccess|uri|/gateway/HADOOPTEST/hbase/hbase/NAMESPACE1:TABLE2/ID1//|success|Response > status: 401 The things to highlight here for the same correlation id: * different topologies are being used * different uris are being used * different users are being used Some of the things that we have configured that could impact results: * authentication caching * multiple Knox servers * load balancer in front of Knox I would be surprised if those things would make one Knox node (it could affect our other Knox nodes haven't looked yet) have duplicate correlation ids. On Tue, Oct 10, 2017 at 9:54 AM, larry mccay <larry.mc...@gmail.com> wrote: > If there is no access entry then that definitely needs to be fixed. > Please file a JIRA for 0.14.0 if that is the case. > > On Tue, Oct 10, 2017 at 10:47 AM, Kevin Risden <kris...@apache.org> wrote: > >> Thanks Larry for the quick response. I'll have to go back and double >> check if the access and uri are in there. If so I must have missed it the >> first time through. Correlating across the GUID should work for our quick >> debugging. >> >> On Tue, Oct 10, 2017 at 9:23 AM, larry mccay <lmc...@apache.org> wrote: >> >>> Hi Kevin - >>> >>> I would think that you can use the correlation id (GUID) to find the url >>> used to access the failed authentication. >>> For instance, consider the following: >>> >>> *17/10/10 10:19:20 >>> ||a7bae089-e52d-403e-a623-fddde5c15c67|audit|0:0:0:0:0:0:0:1|AMBARIaccess|uri|/gateway/mgt/ambari/api/v1/clusters/ljm/requests?to=end_size=10=Requests&_=15076451603 >>> <(507)%20645-1603>83|unavailable|Request method: GET* >>> *17/10/10 10:19:20 >>> ||a7bae089-e52d-403e-a623-fddde5c15c67|audit|0:0:0:0:0:0:0:1|AMBARI|anonymous|||authentication|uri|/gateway/mgt/ambari/api/v1/clusters/ljm/requests?to=end_size=10=Requests&_=1507645160383|success|* >>> *17/10/10 10:19:20 ||a7bae089-e52d-403e-a623-*fddd >>> e5c15c67|audit|0:0:0:0:0:0:0:1|AMBARI|anonymous|||dispatch|uri| >>> http://c6401.ambari.apache.org:8080/api/v1/clusters/ljm/r >>> equests?to=end=Requests_size=10&_=1507645160383| >>> unavailable|Request method: GET >>> 17/10/10 10:19:20 ||a7bae089-e52d-403e-a623-fddd >>> e5c15c67|audit|0:0:0:0:0:0:0:1|AMBARI|anonymous|||dispatch|uri| >>> http://c6401.ambari.apache.org:8080/api/v1/clusters/ljm/r >>> equests?to=end=Requests_size=10&_=1507645160383| >>> success|Response status: 200 >>> 17/10/10 10:19:20 ||a7bae089-e52d-403e-a623-fddd >>> e5c15c67|audit|0:0:0:0:0:0:0:1|AMBARI|anonymous|||access|uri >>> |/gateway/mgt/ambari/api/v1/clusters/ljm/requests?to=end >>> e_size=10=Requests&_=1507645160383|success|Response status: 200 >>> >>> All of the above are related to the same request. >>> The "access" entry prior to the "authentication&
Re: Multiple topology audit logging
Thanks Larry for the quick response. I'll have to go back and double check if the access and uri are in there. If so I must have missed it the first time through. Correlating across the GUID should work for our quick debugging. On Tue, Oct 10, 2017 at 9:23 AM, larry mccay <lmc...@apache.org> wrote: > Hi Kevin - > > I would think that you can use the correlation id (GUID) to find the url > used to access the failed authentication. > For instance, consider the following: > > *17/10/10 10:19:20 > ||a7bae089-e52d-403e-a623-fddde5c15c67|audit|0:0:0:0:0:0:0:1|AMBARIaccess|uri|/gateway/mgt/ambari/api/v1/clusters/ljm/requests?to=end_size=10=Requests&_=1507645160383|unavailable|Request > method: GET* > *17/10/10 10:19:20 > ||a7bae089-e52d-403e-a623-fddde5c15c67|audit|0:0:0:0:0:0:0:1|AMBARI|anonymous|||authentication|uri|/gateway/mgt/ambari/api/v1/clusters/ljm/requests?to=end_size=10=Requests&_=1507645160383|success|* > *17/10/10 10:19:20 ||a7bae089-e52d-403e-a623-*fddd > e5c15c67|audit|0:0:0:0:0:0:0:1|AMBARI|anonymous|||dispatch|uri| > http://c6401.ambari.apache.org:8080/api/v1/clusters/ljm/ > requests?to=end=Requests_size=10&_= > 1507645160383|unavailable|Request method: GET > 17/10/10 10:19:20 ||a7bae089-e52d-403e-a623-fddd > e5c15c67|audit|0:0:0:0:0:0:0:1|AMBARI|anonymous|||dispatch|uri| > http://c6401.ambari.apache.org:8080/api/v1/cluster > s/ljm/requests?to=end=Requests_size=10&_= > 1507645160383|success|Response status: 200 > 17/10/10 10:19:20 ||a7bae089-e52d-403e-a623-fddd > e5c15c67|audit|0:0:0:0:0:0:0:1|AMBARI|anonymous|||access| > uri|/gateway/mgt/ambari/api/v1/clusters/ljm/requests?to= > end_size=10=Requests&_=1507645160383|success|Response status: > 200 > > All of the above are related to the same request. > The "access" entry prior to the "authentication" entry contains the URI > being request with the topology in it. > > Does this not meet the needs? > > thanks, > > --larry > > On Tue, Oct 10, 2017 at 9:59 AM, Kevin Risden <kris...@apache.org> wrote: > >> Background >> >> Currently we have multiple topologies in a single Knox server. Each >> topology points to a different Hadoop environment. For this example, assume >> the topologies are named DEV, TEST, and PROD. >> >> We had a group who hits Knox forget to change their LDAP password so >> there were a bunch of messages like this in the audit logs: >> >> 17/09/12 15:05:08 ||GUID|audit|WEBHBASEauthe >>> ntication|principal|USERNAME|failure|LDAP authentication failed. >> >> >> We contacted the group and they wanted to know which topology the >> requests were hitting so they could fix their password. Regardless of if >> they should have different users per environment or not, we had no way to >> easily tell the group which topology they were hitting. The LDAP >> authentication failure log didn't say which topology it was hitting. >> >> It would be great if the audit log message was something like this: >> >> >>> 17/09/12 15:05:08 ||GUID|audit|DEV|WEBHBASEa >>> uthentication|principal|USERNAME|failure|LDAP authentication failed. >>> >> >> In this case, the topology was added in the audit line maybe near the >> service name. We think having the topology name on the line somewhere would >> be useful for debugging purposes. >> >> Question >> >> Is it possible to configure Knox to log which topology each line in the >> audit log came from? >> >> I was looking at https://github.com/apache/k >> nox/blob/master/gateway-util-common/src/main/java/org/apache >> /hadoop/gateway/audit/log4j/layout/AuditLayout.java and I'm not sure if >> its possible to easily add the topology there or if it is even the right >> place? >> >> >> >
Multiple topology audit logging
Background Currently we have multiple topologies in a single Knox server. Each topology points to a different Hadoop environment. For this example, assume the topologies are named DEV, TEST, and PROD. We had a group who hits Knox forget to change their LDAP password so there were a bunch of messages like this in the audit logs: 17/09/12 15:05:08 > ||GUID|audit|WEBHBASEauthentication|principal|USERNAME|failure|LDAP > authentication failed. We contacted the group and they wanted to know which topology the requests were hitting so they could fix their password. Regardless of if they should have different users per environment or not, we had no way to easily tell the group which topology they were hitting. The LDAP authentication failure log didn't say which topology it was hitting. It would be great if the audit log message was something like this: > 17/09/12 15:05:08 > ||GUID|audit|DEV|WEBHBASEauthentication|principal|USERNAME|failure|LDAP > authentication failed. > In this case, the topology was added in the audit line maybe near the service name. We think having the topology name on the line somewhere would be useful for debugging purposes. Question Is it possible to configure Knox to log which topology each line in the audit log came from? I was looking at https://github.com/apache/knox/blob/master/gateway-util-common/src/main/java/org/apache/hadoop/gateway/audit/log4j/layout/AuditLayout.java and I'm not sure if its possible to easily add the topology there or if it is even the right place?
Re: WebHBase URL Encoding Issue
I know the last email was long so wanted to boil down the question: Does the change from KNOX-690 make sense since it changes the URL meaning? Maybe there is a way to do template matching without modifying the URL that gets passed to the backend? Kevin Risden On Wed, May 24, 2017 at 5:21 PM, Kevin Risden <compuwizard...@gmail.com> wrote: > Sorry for the long email but hopefully it provides enough detail to > understand the problem and if there is anything we can do to work around it > differently. > > > *Problem* > > > > With Knox 0.6.x (HDP 2.3), the Knox WebHBase call returns results > correctly. With Knox 0.9.x (HDP 2.5), the Knox WebHBase call returns a 404 > not found. If we hit WebHBase directly then there is no issue. > > > > An example call: > > > > curl -i -k -u USER 'https://HOST:8443/gateway/TOPOLOGY/hbase/ns:table/% > 2frkpart1%2frkpart2' > > > > *Analysis* > > > > Looked closer at gateway-audit and noticed the dispatch urls were being > encoded differently between the two versions. > > > > Works – Knox 0.6.x (HDP 2.3) > > > > 17/05/23 16:54:13 ||7c4131fc-8638-4a1a-9228-d9a67a312a40|audit|WEBHBASE| > USER|||dispatch|uri|http://HOST:8084/ns:table/%2frkpart1% > 2frkpart2?doAs=USER|success|Response > <http://HOST:8084/ns:table/%2frkpart1%2frkpart2?doAs=USER%7Csuccess%7CResponse> > status: 200 > > > > Doesn’t work – Knox 0.9.x (HDP 2.5) > > > > 17/05/23 17:23:13 ||4244f242-6694-40bb-914d-8dc7e222f074|audit|WEBHBASE| > USER|||dispatch|uri|http://HOST:8084/ns%3Atable/rkpart1/ > rkpart2?doAs=USER|success|Response > <http://HOST:8084/ns%3Atable/rkpart1/rkpart2?doAs=USER%7Csuccess%7CResponse> > status: 404 > > > > The 404 is coming from WebHBase directly not being able to find the split > row key with the extra slash. The difference is that the %2f which is a / > is being decoded and then removed instead of being left as a %2f in the > URL. This changes the meaning of the url and causes issues for WebHBase on > the backend. > > > > At first the culprit seemed like https://issues.apache.org/ > jira/browse/KNOX-709, but this wasn’t the case. Looks like KNOX-709 may > have been caused by KNOX-690. > > > > I pulled down a few Knox versions 0.8.0 and 0.9.0 and found that it did > not affect 0.8.0. I pulled down the code from https://github.com/ > hortonworks/knox-release/tree/HDP-2.5.3.77-tag and did a git bisect to > find the offending commit using this test case: https://gist.github.com/ > risdenk/afecc66d6fc0c9d665abd1ae5466f341. The commit is > https://git-wip-us.apache.org/repos/asf?p=knox.git;h=c28224c and related > JIRA is https://issues.apache.org/jira/browse/KNOX-690. > > > > *Resolution* > > > > I rebuilt Knox from https://github.com/hortonworks/knox-release/tree/ > HDP-2.5.3.77-tag with the commit c28224c for that reverted. The adjusted > code is here: https://github.com/risdenk/knox-release/tree/hdp25_ > revert_KNOX-690. The change is only a single commit > https://github.com/risdenk/knox-release/commit/ > dc452126de99f6f1d15938f7294e95e3b7c89328 > > > > I rebuilt Knox with mvn -DskipTests package and copied the two affected > jars (gateway-provider-rewrite and gateway-util-urltemplate) to > /usr/hdp/current/knox-server/lib/ > > > > I moved the two old jars to /root. The affected jars were > > · gateway-provider-rewrite-0.9.0.2.5.3.0-37.jar > > · gateway-util-urltemplate-0.9.0.2.5.3.0-37.jar > > > > I then restarted Knox on hdpr05en02. This made the following curl call > work: > > > > curl -i -k -u USER 'https://KNOXHOST:8443/gateway/TOPOLOGY/hbase/ns: > table/%2frkpart1%2frkpart2' > > > > *Conclusion* > > > I'm not convinced that KNOX-690 is a good idea but it basically made it so > url encoded paths were checked by the templates/parser. URL encoding should > be left alone in many cases. Reverting the change from KNOX-690 shouldn't > affect us much more other than upgrades to HDP could break this. I think we > should really avoid using url encodable characters in the rowkey especially > for webhbase. / is a bad character to try to pass through webservices. > Since having the customer change rowkey design will be painful, we will be > using a workaround in the short term. > > Kevin Risden >
Re: Encoding/escaping whitespace in WebHDFS requests
Thanks Larry yea I had stumbled upon KNOX-709. Way more detail is in thread "WebHBase URL Encoding Issue". Didn't want to hijack this thread if the WebHBase issue isn't related. Kevin Risden On Wed, May 24, 2017 at 6:08 PM, larry mccay <lmc...@apache.org> wrote: > Hi Kevin - > > You may see some change related to https://issues.apache.org/ > jira/browse/KNOX-709. > > thanks, > > --larry > > On Wed, May 24, 2017 at 6:24 PM, Kevin Risden <compuwizard...@gmail.com> > wrote: > >> Just saw this as I was submitting a potentially related WebHBase url >> encoding email to the knox-user list. Curious if they are related. >> >> Alex - out of curiousity did you use Knox with HDP 2.4 or prior and not >> see this issue? >> >> Kevin Risden >> >> On Wed, May 24, 2017 at 4:08 PM, larry mccay <larry.mc...@gmail.com> >> wrote: >> >>> Thank you, Alex. >>> >>> Please file a JIRA for this with the above details. >>> I will try and reproduce and investigate and see if we can't get it >>> fixed or a workaround for the 0.13.0 release. >>> This is planned for the end of next week. >>> >>> On Wed, May 24, 2017 at 3:18 PM, Willmer, Alex (UK Defence) < >>> alex.will...@cgi.com> wrote: >>> >>>> Hi Larry, >>>> >>>> The same file does work directly from WebHDFS (see below). Looking more >>>> closely at the logs I sent previously, it looks like Knox (or something in >>>> the chain I'm unaware of) is decoding the %20 encoded spaces, then >>>> reencoding them as + encoded, i.e. >>>> >>>> 17/05/24 15:51:05 ||88ce58ea-d7c5-46cd-a87a-c2f9 >>>> 6b38130e|audit|WEBHDFSaccess|uri|/gateway//webh >>>> dfs/v1/docs/filename with spaces.pdf?op=OPEN|unavailable|Request >>>> method: GET >>>> .. >>>> 17/05/24 15:51:05 ||88ce58ea-d7c5-46cd-a87a-c2f9 >>>> 6b38130e|audit|WEBHDFSdispatch|uri|http://>>> enode>.:50070/webhdfs/v1/docs/filename+with+spaces.pdf >>>> ?op=OPEN=|success|Response status: 404 >>>> >>>> With thanks, Alex >>>> >>>> >>>> Direct WebHDFS request (hostnames redacted) >>>> >>>> # curl -si -u: "http://:50070/webhd >>>> fs/v1/docs/filename%20with%20spaces.pdf?op=OPEN" --negotiate -L | head >>>> -n40 >>>> HTTP/1.1 401 Authentication required >>>> Cache-Control: must-revalidate,no-cache,no-store >>>> Date: Wed, 24 May 2017 19:01:41 GMT >>>> Pragma: no-cache >>>> Date: Wed, 24 May 2017 19:01:41 GMT >>>> Pragma: no-cache >>>> X-FRAME-OPTIONS: SAMEORIGIN >>>> WWW-Authenticate: Negotiate >>>> Set-Cookie: hadoop.auth=; Path=/; HttpOnly >>>> Content-Type: text/html; charset=iso-8859-1 >>>> Content-Length: 1533 >>>> Server: Jetty(6.1.26.hwx) >>>> >>>> HTTP/1.1 307 TEMPORARY_REDIRECT >>>> Cache-Control: no-cache >>>> Expires: Wed, 24 May 2017 19:01:42 GMT >>>> Date: Wed, 24 May 2017 19:01:42 GMT >>>> Pragma: no-cache >>>> Expires: Wed, 24 May 2017 19:01:42 GMT >>>> Date: Wed, 24 May 2017 19:01:42 GMT >>>> Pragma: no-cache >>>> X-FRAME-OPTIONS: SAMEORIGIN >>>> WWW-Authenticate: Negotiate YGkGCSqGSIb3EgECAgIAb1owWKADAg >>>> EFoQMCAQ+iTDBKoAMCARKiQwRBQM/auuLcl2xey6wMp6EjCPJFSqK3snscxM >>>> zW7RvfgxOo7182GzD5N9jf+OWGr+tjpvlRX0c/7iTBfYKSetf4ekU= >>>> Set-Cookie: hadoop.auth="u=admin=admin@C >>>> YSAFA=kerberos=1495688502002=b7p35TgaxItAUTkKJuSXuynoq9E="; >>>> Path=/; HttpOnly >>>> Content-Type: application/octet-stream >>>> Location: http://:1022/webhdfs/v1/docs/filename%20with%20sp >>>> aces.pdf?op=OPEN=HgAFYWRtaW4FYWRtaW4AigFcO9YJ8ooB >>>> XF_ijfJFAxSBYFUnsXY3up11ZNIi4hIi__5RvRJXRUJIREZTIGRlbGVnYXRp >>>> b24PMTcyLjE4LjAuOTo4MDIw=:8020=0 >>>> Content-Length: 0 >>>> Server: Jetty(6.1.26.hwx) >>>> >>>> HTTP/1.1 200 OK >>>> Access-Control-Allow-Methods: GET >>>> Access-Control-Allow-Origin: * >>>> Content-Type: application/octet-stream >>>> Connection: close >>>> Content-Length: 13365618 >>>> >>>> %1.6 >>>> <>stream >>>> ... >>>> >>>> >>>> -- >>>> *From:* larry mccay [lm
