Hi all,
Since this thread is too long with lots of suggestions. Let me summarize
all the things that has been discussed in here.
Issue:-
We are trying to give better user experience by handling the port offset so
user don't have to do anything when deploying a toolbox in the default BAM
setup after changing the port offset in carbon.xml.
Suggesions and Approaches:-
1. Current implementation
Introduced a parameter ('carbon.port.offset') to hive script and by using
that we can control the port offset depending on whether Cassandra is
embedded or external.
disadvantages :- we have to change the hive script
('carbon.port.offset'=false) if the deployment is using external cluster.
change is not scriptable
2. Change the Cassandra datasource url, so that we can identify the
embedded and external Cassandra deployment and assign the Cassandra port
value accordingly.
disadvantages :- we have to change the current Cassandra datasource url
pattern (jdbc:cassandra://Ip:port/keyspace)and there will be two urls for
embedded one and external one.
3. Hive will take the decision whether it is embedded or external by
checking the availability of Cassandra port on the local server.
disadvantage :- In the external Cassandra deployment, there can be a port
with (<defaultCassandraPort>+<portOffset>) in the local machine. In this
case there can be an issue.
4. Port offset doesn't handle by Hive. :) User will change it via
master-datasource.xml
disadvantage :- If a user change the port offset and he is using embedded
Cassandra then every time he will have to change the Cassandra port value
in master-datasource.xml.
Thanks,
KasunW.
On Sat, Apr 27, 2013 at 12:01 AM, Kasun Weranga <[email protected]> wrote:
>
>
>
> On Fri, Apr 26, 2013 at 9:42 PM, Sinthuja Ragendran <[email protected]>wrote:
>
>>
>>
>>
>> On Fri, Apr 26, 2013 at 8:43 PM, Kasun Weranga <[email protected]> wrote:
>>
>>>
>>>
>>> On Friday, April 26, 2013, Sinthuja Ragendran wrote:
>>>
>>>>
>>>> On 26 Apr 2013 19:10, "Kasun Weranga" <[email protected]> wrote:
>>>> >
>>>> > I was thinking about removing this carbon.port.offset property from
>>>> hive script and handle this situation by hive itself. What we can do is we
>>>> can check the availability of cassandra port (<defaultCassandraPort> +<port
>>>> offset>) and if it is available we can assign that port as the cassandra
>>>> port.
>>>>
>>>> What do you mean by availability of cassandra port? Are you suggesting
>>>> to check whether the cassandra port is open?
>>>>
>>> Yes.
>>>
>>
>> Let say we have two BAM instances in the same node. One server as
>> analyzer instance and other one as cassandra instance. And from analyzer
>> instance I need to connect to the cassandra server instance. Since the
>> cassandra port will also be open in the analyzer instance AFAIU in your
>> suggested approach, the hive scripts in analyzer node will connect to the
>> embedded cassandra server of the analyzer instance isn't?
>>
>
> I ignore this case, since if he is using external Cassandra we don't
> recommend to start the Cassandra server in analyzer node. Anyway there is a
> issue if this connect to some other port in the same machine there can be a
> rare possibility for that.
>
> Anyway now this thread seems filled with lots of suggestions :) and it is
> difficult for others to follow.
>
>
> And how can we change it to connect to the cassandra instance from
>> analyzer instance in your suggested approach? Can you please clear me on
>> this on your approach.
>>
>> Thanks,
>> Sinthuja.
>>
>>
>>
>> Thanks,
>>>> Sinthuja.
>>>>
>>>> If it is not available then we can use the port value define in the
>>>> Cassandra data source.
>>>> > This will handle both embedded and external cassandra usecase without
>>>> changing the hive script and existing Cassandra datasource url pattern. I
>>>> think this is much cleaner way to handle this situation. Isn't it?
>>>> >
>>>> > On Friday, April 26, 2013, Supun Malinga wrote:
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> Couldn't read all the replies thoroughly.
>>>> >> Do you suggest that we can define the port in
>>>> master-datasources.xml?.
>>>> >>
>>>> >> About user mistakes I believe proper documenting how to handle a
>>>> portoff set scenario would be a better option.
>>>> >>
>>>> >> thanks,
>>>> >>
>>>> >> On Fri, Apr 26, 2013 at 4:06 PM, Chamara Ariyarathne <
>>>> [email protected]> wrote:
>>>> >>>
>>>> >>>
>>>> >>>
>>>> >>> On Fri, Apr 26, 2013 at 4:04 PM, Sinthuja Ragendran <
>>>> [email protected]> wrote:
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> On Fri, Apr 26, 2013 at 3:50 PM, Kasun Weranga <[email protected]>
>>>> wrote:
>>>> >>>>>
>>>> >>>>> We introduce this carbon.port.offset property to prevent issues
>>>> due to port offset for users who are completely new to WSO2 BAM product. In
>>>> this case they don't need to change the hive scripts after they change the
>>>> port offset because they are using embedded Cassandra and port values
>>>> change with the port offset value.
>>>> >>>>>
>>>> >>>>> Currently we are facing this issue in API-M since they start BAM
>>>> in different offsets. That is the main requirement for adding this
>>>> functionality.
>>>> >>>>>
>>>> >>>>> But we need to disable port offset changes for user who are using
>>>> external Cassandra. These users are advance users and I think it is ok for
>>>> telling them to change the hive script to disable it.
>>>> (carbon.port.offset='false')
>>>> >>>>
>>>> >>>>
>>>> >>>> This property is going to be in each cassandra external tables
>>>> defined, which is same as 'cassandra.port' property. Therefore there may be
>>>> many places required to change if we connecting to external cassandra.
>>>> >>>>
>>>> >>>> I would prefer this change to be in one single place. IMHO it's
>>>> better to remove the port from the url or having some property along with
>>>> the url such as
>>>> 'jdbc:cassandra://localhost:9160/EVENT_KS;APPLY_PORT_OFFSET=TRUE' would be
>>>> better.
>>>> >>>>
>>>> >>> APPLY_PORT_OFFSET=TRUE in a database connection url?
>>>> >>>
>>>> >>> I don't think that is good!
>>>> >>>
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>> Sinthuja.
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>>
>>>> >>>>> I am not sure changing the cassandra datasource url according our
>>>> requirement is the right thing to do. If we do that we need to change the
>>>> datasource implementation too and also there will be two way of defining
>>>> Cassandra url.
>>>> >>>>>
>>>> >>>>> I feel it is ok for advance BAM users to edit this value in the
>>>> hive script.
>>>> >>>>>
>>>> >>>>> Thanks,
>>>> >>>>> KasunW.
>>>> >>>>>
>>>> >>>>>
>>>> >>>>>
>>>> >>>>> On Fri, Apr 26, 2013 at 3:30 PM, Sinthuja Ragendran <
>>>> [email protected]> wrote:
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> On Fri, Apr 26, 2013 at 3:18 PM, Ayashkantha Ramasinghe <
>>>> [email protected]> wrote:
>>>> >>>>>>>
>>>> >>>>>>> Hi Sinthuja,
>>>> >>>>>>>
>>>> >>>>>>> After Kasun apply new patch, that problem will be solved.
>>>> Because, after reading master-datasource.xml the cassandra handler is
>>>> adding the port offset internally.
>>>> >>>>>>
>>>> >>>>>>
>>>> >>>>>> Here this property is added in the hive script within
>>>> SERDEPROPERTIES of cassandra external table, isn't it?
>>>> >>>>>>
>>>> >> --
>>>> >> Supun Malinga,
>>>> >>
>>>> >> Software Engineer,
>>>> >> WSO2 Inc.
>>>> >> http://wso2.com
>>>> >> <http://wso2.org>
>>>>
>>>
>>>
>>> --
>>> *Kasun Weranga*
>>> **
>>> Member, Management Committee - Data Technologies
>>> Software Engineer
>>> *WSO2, Inc.
>>> *lean.enterprise.middleware.
>>> mobile : +94 772314602
>>> <http://sanjeewamalalgoda.blogspot.com/>blog :
>>> http://kasunweranga.blogspot.com/
>>>
>>
>>
>>
>> --
>> *Sinthuja Rajendran*
>> Software Engineer <http://wso2.com/>
>> WSO2, Inc.:http://wso2.com
>>
>> Blog: http://sinthu-rajan.blogspot.com/
>> Mobile: +94774273955
>>
>>
>>
>
>
> --
> *Kasun Weranga*
> **
> Member, Management Committee - Data Technologies
> Software Engineer
> *WSO2, Inc.
> *lean.enterprise.middleware.
> mobile : +94 772314602
> <http://sanjeewamalalgoda.blogspot.com/>blog :
> http://kasunweranga.blogspot.com/
>
--
*Kasun Weranga*
**
Member, Management Committee - Data Technologies
Software Engineer
*WSO2, Inc.
*lean.enterprise.middleware.
mobile : +94 772314602
<http://sanjeewamalalgoda.blogspot.com/>blog :
http://kasunweranga.blogspot.com/
_______________________________________________
Dev mailing list
[email protected]
http://wso2.org/cgi-bin/mailman/listinfo/dev
