Hi Srinath/Anjana,
Sorry for the delay. Yes, we're changing from Hadoop as part of the
improvements, where we will have the same underlying implementation for the
config but with two DatasourceReader implementations, one each for HBase
and HDFS respectively.
We are also modifying the format of the expected configuration to better
match the properties specified in other Carbon datasource configurations,
so the updated config would look something like:
<datasource>
<name>WSO2_ANALYTICS_FS_DB_HDFS</name>
<description>The datasource used for analytics file system</description>
<jndiConfig>
<name>jdbc/WSO2HDFSDB</name>
</jndiConfig>
<definition type="HDFS">
<configuration>
<property name="fs.default.name" value="hdfs://localhost:9000"
/>
<property name="dfs.data.dir" value="/dfs/data" />
<property name="fs.hdfs.impl" value
="org.apache.hadoop.hdfs.DistributedFileSystem" />
<property name="fs.file.impl" value
="org.apache.hadoop.fs.LocalFileSystem" />
</configuration>
</definition>
</datasource>
Thanks,
On 14 May 2015 at 00:52, Anjana Fernando <[email protected]> wrote:
> Hi Srinath,
>
> Yeah, I'd a chat with Gokul yesterday, we are changing this to HDFS and
> also having another HBase one as well, I think he has already done the
> changes. @Gokul, please send the updated information.
>
> Cheers,
> Anjana.
>
> On Thu, May 14, 2015 at 1:10 PM, Srinath Perera <[email protected]> wrote:
>
>> Can we call type HDFS instead of Hadoop? ( if we can change that without
>> much trouble)
>>
>> On Tue, May 12, 2015 at 8:38 PM, Gokul Balakrishnan <[email protected]>
>> wrote:
>>
>>> Hi all,
>>>
>>> As part of the HBase analytics datasource implementation for DAS 3.0, we
>>> have come up with $subject which is envisioned to offer a standardised way
>>> to specify connectivity parameters for a remote Hadoop-based instance in a
>>> Carbon datasource configuration.
>>>
>>> The datasource reader will expect the configuration to be specified in a
>>> similar format which is used for standard Apache Commons Configuration [1],
>>> as used by both HDFS and HBase. An example datasource definition would look
>>> like:
>>>
>>> <datasource>
>>> <name>WSO2_ANALYTICS_FS_DB_HDFS</name>
>>> <description>The datasource used for analytics file system</
>>> description>
>>> <jndiConfig>
>>> <name>jdbc/WSO2HDFSDB</name>
>>> </jndiConfig>
>>> <definition type="HADOOP">
>>> <configuration>
>>> <property>
>>> <name>fs.default.name</name>
>>> <value>hdfs://localhost:9000</value>
>>> </property>
>>> <property>
>>> <name>dfs.data.dir</name>
>>> <value>/dfs/data</value>
>>> </property>
>>> <property>
>>> <name>fs.hdfs.impl</name>
>>> <value>org.apache.hadoop.hdfs.DistributedFileSystem</
>>> value>
>>> </property>
>>> <property>
>>> <name>fs.file.impl</name>
>>> <value>org.apache.hadoop.fs.LocalFileSystem</value>
>>> </property>
>>> </configuration>
>>> </definition>
>>> </datasource>
>>>
>>> The definition type for the above is set as "HADOOP". The datasource
>>> reader implementation is currently hosted at [2], and would be merged with
>>> the carbon-data git repo once reviewed.
>>>
>>> Appreciate your thought and suggestions.
>>>
>>> Thanks,
>>> Gokul.
>>>
>>> [1] http://commons.apache.org/proper/commons-configuration/
>>>
>>> [2]
>>> https://github.com/gokulbs/carbon-data/tree/master/components/data-sources/org.wso2.carbon.datasource.reader.hadoop
>>>
>>> --
>>> Balakrishnan Gokulakrishnan
>>> Senior Software Engineer,
>>> WSO2, Inc. http://wso2.com
>>> Mob: +94 77 593 5789 | +1 650 272 9927
>>>
>>> _______________________________________________
>>> Architecture mailing list
>>> [email protected]
>>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>>
>>>
>>
>>
>> --
>> ============================
>> Srinath Perera, Ph.D.
>> http://people.apache.org/~hemapani/
>> http://srinathsview.blogspot.com/
>>
>> _______________________________________________
>> Architecture mailing list
>> [email protected]
>> https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
>>
>>
>
>
> --
> *Anjana Fernando*
> Senior Technical Lead
> WSO2 Inc. | http://wso2.com
> lean . enterprise . middleware
>
--
Balakrishnan Gokulakrishnan
Senior Software Engineer,
WSO2, Inc. http://wso2.com
Mob: +94 77 593 5789 | +1 650 272 9927
_______________________________________________
Architecture mailing list
[email protected]
https://mail.wso2.org/cgi-bin/mailman/listinfo/architecture
