Hi Thanks , will take a look. Our usecase was to write a continuous stream of
data through KNOX/Webhdfs , where cluster is kerberoized also we have internal
authtoken implementation to reach knox.
As of now we solved by using CREATE op by adding heartbeat interval of <30sec
and write to file for an hour. Let see how it goes with its scale .
Sandeep ,
Will try to use patch(KNOX-829) having APPEND into KNOX code and see how it
goes. Which version you tried and saw errors.
\Rajesh
On Wednesday 17 January 2018, 5:44:33 AM GMT-8, Sandeep More
<[email protected]> wrote:
Thanks Sourav,
This looks interesting, I'll try it out. Thanks for bringing it up on the
mailing list !
Best,Sandeep
On Tue, Jan 16, 2018 at 10:49 PM, Sourav Mazumder <[email protected]>
wrote:
You may try out this implementation in Apache Bahir. -
https://github.com/apache/ bahir/pull/28.
Here we wrote a custom HDFS file system implementation extending the existing
Webhdfs implementation to take care of Knox and SSL related requirements.
This works fine while accessing data from Spark. Should work for other clients
too.
Would be happy to help/extend this solution in case you have any concrete
requirement. The latest version has support for csv, parquet, json types.
Regards,
Sourav
On Wed, Jan 17, 2018 at 7:54 AM, Rajesh Chandramohan <[email protected]>
wrote:
Hi ,
Looks like its related to this KNOX Jira https://issues.apache.org
/jira/browse/KNOX-829
Thanks\Rajesh
On Tuesday 16 January 2018, 4:27:41 PM GMT-8, Rajesh Chandramohan
<[email protected]> wrote:
Hi ,
Why we don't have append operation implemented here https://github.com/apache
/knox/tree/v0.14.0-release/ gateway-shell/src/main/java/
org/apache/hadoop/gateway/ shell/hdfs
That would be useful if we like to have streaming usecase with KNOX right ? Si
nobody bothered to expand for that case or some other reason for not having
append operation implemented in gateway-shell
\Rajesh
On Tuesday 16 January 2018, 2:11:05 PM GMT-8, Rajesh Chandramohan
<[email protected]> wrote:
Hi KNOX-team,
Is anybody implemented to write Streaming data to HDFS via Knox webhdfs ? Is
that possible where HDFS cluster is kerberized .Will Authentication/Delegation
token will be refreshed etc .. Any thoughts ?
\Rajesh
