[jira] [Updated] (TUBEMQ-320) Request for static web contents would get responses with no content
[ https://issues.apache.org/jira/browse/TUBEMQ-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-320: - Attachment: NormalLook.png BugLook.png > Request for static web contents would get responses with no content > --- > > Key: TUBEMQ-320 > URL: https://issues.apache.org/jira/browse/TUBEMQ-320 > Project: Apache TubeMQ > Issue Type: Bug > Components: Server >Affects Versions: 0.6.0 >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > Fix For: 0.6.0 > > Attachments: BugLook.png, NormalLook.png > > > After clearing the cache, refresh homepage of master would result in loss of > static materials, like styles, scripts and pictures, while all dynamic > elements are working as expected. > The looks would be like in attachments. > This should be due to Servlet binding on Embedded-Jetty, which does too much > to cover Servlet requests while missing the ability to respond static css > contents. > So obviously the fix plan is to recover this ability, restoring original and > expected behavior when browsing webpages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TUBEMQ-320) Request for css files would respond with no content
Jeff Zhou created TUBEMQ-320: Summary: Request for css files would respond with no content Key: TUBEMQ-320 URL: https://issues.apache.org/jira/browse/TUBEMQ-320 Project: Apache TubeMQ Issue Type: Bug Components: Server Affects Versions: 0.6.0 Reporter: Jeff Zhou Assignee: Jeff Zhou Fix For: 0.6.0 After clearing the cache, refresh homepage of master would result in loss of static materials, like styles, scripts and pictures, while all dynamic elements are working as expected. The looks would be like in attachments. This should be due to Servlet binding on Embedded-Jetty, which does too much to cover Servlet requests while missing the ability to respond static css contents. So obviously the fix plan is to recover this ability, restoring original and expected behavior when browsing webpages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TUBEMQ-320) Request for static web contents would get responses with no content
[ https://issues.apache.org/jira/browse/TUBEMQ-320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-320: - Summary: Request for static web contents would get responses with no content (was: Request for css files would respond with no content) > Request for static web contents would get responses with no content > --- > > Key: TUBEMQ-320 > URL: https://issues.apache.org/jira/browse/TUBEMQ-320 > Project: Apache TubeMQ > Issue Type: Bug > Components: Server >Affects Versions: 0.6.0 >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > Fix For: 0.6.0 > > > After clearing the cache, refresh homepage of master would result in loss of > static materials, like styles, scripts and pictures, while all dynamic > elements are working as expected. > The looks would be like in attachments. > This should be due to Servlet binding on Embedded-Jetty, which does too much > to cover Servlet requests while missing the ability to respond static css > contents. > So obviously the fix plan is to recover this ability, restoring original and > expected behavior when browsing webpages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TUBEMQ-309) Add POST support to WebAPI
[ https://issues.apache.org/jira/browse/TUBEMQ-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou resolved TUBEMQ-309. -- Resolution: Implemented > Add POST support to WebAPI > -- > > Key: TUBEMQ-309 > URL: https://issues.apache.org/jira/browse/TUBEMQ-309 > Project: Apache TubeMQ > Issue Type: Sub-task > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > This is a key part of improving TubeMQ about WebAPI, which is built on top of > Jetty 9 architecture. > Attention: > 1. WebAPI could then support BOTH of GET and POST, not either of GET or POST > request, after this sub-task; > 2. WebAPI still work on one-to-one HTTP 1.1 mode as it has been for a long > time after this sub-task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TUBEMQ-308) Upgrade Jetty 6 (mortbay) => Jetty 9 (eclipse)
[ https://issues.apache.org/jira/browse/TUBEMQ-308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou resolved TUBEMQ-308. -- Resolution: Implemented > Upgrade Jetty 6 (mortbay) => Jetty 9 (eclipse) > -- > > Key: TUBEMQ-308 > URL: https://issues.apache.org/jira/browse/TUBEMQ-308 > Project: Apache TubeMQ > Issue Type: Sub-task > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > This sub-task is for component upgrade. > Currently using Jetty 6 is so old, which has stopped maintenance for 10+ > years, and stay in HTTP 1.1, and the mechanism currently using based on > Filters to start processing could be so hard to do feature extension and > supporting POST request. > So this could be the first step polishing TubeMQ WebAPI, to make it stay in > latest. > Attention: no WebAPI feature would be changed after this sub-task, it's a > fundamental upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TUBEMQ-309) Add POST support to WebAPI
[ https://issues.apache.org/jira/browse/TUBEMQ-309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou reassigned TUBEMQ-309: Assignee: Jeff Zhou > Add POST support to WebAPI > -- > > Key: TUBEMQ-309 > URL: https://issues.apache.org/jira/browse/TUBEMQ-309 > Project: Apache TubeMQ > Issue Type: Sub-task > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > > This is a key part of improving TubeMQ about WebAPI, which is built on top of > Jetty 9 architecture. > Attention: > 1. WebAPI could then support BOTH of GET and POST, not either of GET or POST > request, after this sub-task; > 2. WebAPI still work on one-to-one HTTP 1.1 mode as it has been for a long > time after this sub-task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TUBEMQ-310) Upgrade WebAPI protocol: HTTP/1.1 => HTTP/2
Jeff Zhou created TUBEMQ-310: Summary: Upgrade WebAPI protocol: HTTP/1.1 => HTTP/2 Key: TUBEMQ-310 URL: https://issues.apache.org/jira/browse/TUBEMQ-310 Project: Apache TubeMQ Issue Type: Sub-task Components: Server Reporter: Jeff Zhou THIS IS AN OPTIONAL TASK. HTTP/2 could bring more intelligent control on Frames-based data submission (no longer a big form-data or alike), Interactive data exchange (Multiplexing/ServerPush mechanism), API QoS (Stream Prioritization), while current HTTP/1.1 could fulfill basic requirement of WebAPI. So it brings bigger possibility to WebAPI, but it may not be bright spot improvement for now (or maybe some day :) It's nice to discuss about it, and just implement it if this upgrade itself is straight-forward based on Jetty 9. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TUBEMQ-309) Add POST support to WebAPI
Jeff Zhou created TUBEMQ-309: Summary: Add POST support to WebAPI Key: TUBEMQ-309 URL: https://issues.apache.org/jira/browse/TUBEMQ-309 Project: Apache TubeMQ Issue Type: Sub-task Components: Server Reporter: Jeff Zhou This is a key part of improving TubeMQ about WebAPI, which is built on top of Jetty 9 architecture. Attention: 1. WebAPI could then support BOTH of GET and POST, not either of GET or POST request, after this sub-task; 2. WebAPI still work on one-to-one HTTP 1.1 mode as it has been for a long time after this sub-task. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TUBEMQ-307) Introduce POST request support to WebAPI service
[ https://issues.apache.org/jira/browse/TUBEMQ-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-307: - Description: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided TubeMQ is currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is an important requirement as well in addition to implementation of this functionality. Potential extra benefits from this upgrade: 1. HTTP 2 Support: maybe one day some of these features could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major steps: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). was: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided TubeMQ is currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is an important requirement as well in addition to implementation of this functionality. Potential extra benefits from this upgrade: 1. HTTP 2 Support: maybe one day some of these features could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). > Introduce POST request support to WebAPI service > > > Key: TUBEMQ-307 > URL: https://issues.apache.org/jira/browse/TUBEMQ-307 > Project: Apache TubeMQ > Issue Type: Improvement > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > > As for WebAPI service part, currently it
[jira] [Updated] (TUBEMQ-307) Introduce POST request support to WebAPI service
[ https://issues.apache.org/jira/browse/TUBEMQ-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-307: - Description: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided TubeMQ is currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is an important requirement as well in addition to implementation of this functionality. Potential extra benefits from this upgrade: 1. HTTP 2 Support: maybe one day any of these features could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). was: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided TubeMQ is currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is an important requirement as well in addition to implementation of this functionality. Potential extra benefits from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). > Introduce POST request support to WebAPI service > > > Key: TUBEMQ-307 > URL: https://issues.apache.org/jira/browse/TUBEMQ-307 > Project: Apache TubeMQ > Issue Type: Improvement > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > > As for WebAPI service part, currently it
[jira] [Updated] (TUBEMQ-307) Introduce POST request support to WebAPI service
[ https://issues.apache.org/jira/browse/TUBEMQ-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-307: - Description: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided TubeMQ is currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is an important requirement as well in addition to implementation of this functionality. Potential extra benefits from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). was: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided TubeMQ is currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is an important requirement as well in addition to implementation of this functionality. Potential extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). > Introduce POST request support to WebAPI service > > > Key: TUBEMQ-307 > URL: https://issues.apache.org/jira/browse/TUBEMQ-307 > Project: Apache TubeMQ > Issue Type: Improvement > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > > As for WebAPI service part, currently it
[jira] [Updated] (TUBEMQ-307) Introduce POST request support to WebAPI service
[ https://issues.apache.org/jira/browse/TUBEMQ-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-307: - Description: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided TubeMQ is currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is an important requirement as well in addition to implementation of this functionality. Potential extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). was: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided TubeMQ is currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is an important requirement as well in addition to implementation of this functionality. Possible extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). > Introduce POST request support to WebAPI service > > > Key: TUBEMQ-307 > URL: https://issues.apache.org/jira/browse/TUBEMQ-307 > Project: Apache TubeMQ > Issue Type: Improvement > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > > As for WebAPI service part, currently it supports
[jira] [Updated] (TUBEMQ-307) Introduce POST request support to WebAPI service
[ https://issues.apache.org/jira/browse/TUBEMQ-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-307: - Description: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided TubeMQ is currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is an important requirement as well in addition to implementation of this functionality. Possible extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). was: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided the currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is also an important requirement in addition to implementation of this functionality. Possible extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). > Introduce POST request support to WebAPI service > > > Key: TUBEMQ-307 > URL: https://issues.apache.org/jira/browse/TUBEMQ-307 > Project: Apache TubeMQ > Issue Type: Improvement > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > > As for WebAPI service part, currently it supports only GET
[jira] [Updated] (TUBEMQ-307) Introduce POST request support to WebAPI service
[ https://issues.apache.org/jira/browse/TUBEMQ-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-307: - Description: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you could literally post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided the currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is also an important requirement in addition to implementation of this functionality. Possible extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). was: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you literally could post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided the currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is also an important requirement in addition to implementation of this functionality. Possible extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). > Introduce POST request support to WebAPI service > > > Key: TUBEMQ-307 > URL: https://issues.apache.org/jira/browse/TUBEMQ-307 > Project: Apache TubeMQ > Issue Type: Improvement > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > > As for WebAPI service part, currently it supports only GET request,
[jira] [Updated] (TUBEMQ-307) Introduce POST request support to WebAPI service
[ https://issues.apache.org/jira/browse/TUBEMQ-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-307: - Description: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encode them by BASE64 or something like it), however, POST body does NOT have such restriction, so you literally could post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided the currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is also an important requirement in addition to implementation of this functionality. Possible extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). was: As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encoded by BASE64 or something like it), however, POST body does NOT such restriction, so you literally could post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided the currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is also an important requirement in addition to implementation of this functionality. Possible extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). > Introduce POST request support to WebAPI service > > > Key: TUBEMQ-307 > URL: https://issues.apache.org/jira/browse/TUBEMQ-307 > Project: Apache TubeMQ > Issue Type: Improvement > Components: Server >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > > As for WebAPI service part, currently it supports only GET request, and >
[jira] [Created] (TUBEMQ-308) Upgrade Jetty 6 (mortbay) => Jetty 9 (eclipse)
Jeff Zhou created TUBEMQ-308: Summary: Upgrade Jetty 6 (mortbay) => Jetty 9 (eclipse) Key: TUBEMQ-308 URL: https://issues.apache.org/jira/browse/TUBEMQ-308 Project: Apache TubeMQ Issue Type: Sub-task Components: Server Reporter: Jeff Zhou Assignee: Jeff Zhou This sub-task is for component upgrade. Currently using Jetty 6 is so old, which has stopped maintenance for 10+ years, and stay in HTTP 1.1, and the mechanism currently using based on Filters to start processing could be so hard to do feature extension and supporting POST request. So this could be the first step polishing TubeMQ WebAPI, to make it stay in latest. Attention: no WebAPI feature would be changed after this sub-task, it's a fundamental upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TUBEMQ-307) Introduce POST request support to WebAPI service
Jeff Zhou created TUBEMQ-307: Summary: Introduce POST request support to WebAPI service Key: TUBEMQ-307 URL: https://issues.apache.org/jira/browse/TUBEMQ-307 Project: Apache TubeMQ Issue Type: Improvement Components: Server Reporter: Jeff Zhou Assignee: Jeff Zhou As for WebAPI service part, currently it supports only GET request, and supporting POST request could apparently and significantly improve the usability and extend the functionality of WebAPI itself. It could be described as below, some may be so familiar to developers, users, or relating stakeholders: 1. GET has hard limitation of the length of URL (2KB), however, POST using body to carry data section, so it does NOT have such limitation, e.g. you may request an API with even 16MB something if it's really needed; 2. GET requires data to be ASCII characters (or you have to encoded by BASE64 or something like it), however, POST body does NOT such restriction, so you literally could post whatever kind of binary data you want, so it has wider compatibility, and reduces complexity and contributes to performance (no extra encoder needed); 3. GET combines all things in URL, so it could easily captured and cached, while it's visible and less secure to carry sensitive data like identity, however, POST is obviously safer as generally post body won't be cached or captured in analyzer or logger or something else in infrastructure; Provided the currently using Mortbay Jetty (Jetty 6, which has stopped maintenance for 10+ years), for security, performance and extensibility consideration, upgrade current Jetty component to Eclipse Jetty (Jetty 9, which is current mainstream of Jetty, now in hot developing) is also an important requirement in addition to implementation of this functionality. Possible extra benefit from this upgrade: 1. HTTP 2 Support: maybe one day any of these feature could play an important role of WebAPI interaction: Multiplexing / Server Push / Frame-based Compression / Stream Prioritization 2. Stay in latest distribution, with Feature and Security improvements (see Jetty VERSION: change log). So this improvement would consists of 3 major parts: 1. Upgrade Jetty 6 (mortbay) to Jetty 9 (eclipse), with NO WebAPI feature change; 2. Adding POST support to WebAPI, but stay in HTTP 1.1 as before; 3. Upgrade protocol to HTTP 2.0 for WebAPI (with backward compatibility to HTTP 1.1). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (TUBEMQ-215) TubeMQ should support running for windows.
[ https://issues.apache.org/jira/browse/TUBEMQ-215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155299#comment-17155299 ] Jeff Zhou edited comment on TUBEMQ-215 at 7/10/20, 9:33 AM: Hi netroby, As for current mainline of TubeMQ, there should be 3 more command-line script (cmd) for windows on starting and environment settings, the development requirement for TubeMQ should be fulfilled. As for now, you may use widely-spread Service Registers or scheduler to automatically start master and/or broker as scheduled. Should you have any issue on using these cmds, please let us know. All in all, thanks for your use of TubeMQ. Best regards. was (Author: hystericalhell): Hi netroby, As for current mainline of TubeMQ, there should be 3 more command-line script (cmd) for windows on starting and environment settings, the development requirement for TubeMQ should be fulfilled. As for now, you may use widely-spread Server Registers or scheduler to automatically start master and/or broker as scheduled. Should you have any issue on using these cmds, please let us know. All in all, thanks for your use of TubeMQ. Best regards. > TubeMQ should support running for windows. > --- > > Key: TUBEMQ-215 > URL: https://issues.apache.org/jira/browse/TUBEMQ-215 > Project: Apache TubeMQ > Issue Type: Improvement >Reporter: netroby >Priority: Major > > RocketMQ , the other MQ platform , can running with Windows. working as well > as Linux. > TubeMQ written with Java, should working no problem with Windows. > > Let's improve the running for windows? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TUBEMQ-215) TubeMQ should support running for windows.
[ https://issues.apache.org/jira/browse/TUBEMQ-215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17155299#comment-17155299 ] Jeff Zhou commented on TUBEMQ-215: -- Hi netroby, As for current mainline of TubeMQ, there should be 3 more command-line script (cmd) for windows on starting and environment settings, the development requirement for TubeMQ should be fulfilled. As for now, you may use widely-spread Server Registers or scheduler to automatically start master and/or broker as scheduled. Should you have any issue on using these cmds, please let us know. All in all, thanks for your use of TubeMQ. Best regards. > TubeMQ should support running for windows. > --- > > Key: TUBEMQ-215 > URL: https://issues.apache.org/jira/browse/TUBEMQ-215 > Project: Apache TubeMQ > Issue Type: Improvement >Reporter: netroby >Priority: Major > > RocketMQ , the other MQ platform , can running with Windows. working as well > as Linux. > TubeMQ written with Java, should working no problem with Windows. > > Let's improve the running for windows? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (TUBEMQ-265) Unexpected broker disappearance in broker list after updating default broker metadata
[ https://issues.apache.org/jira/browse/TUBEMQ-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou closed TUBEMQ-265. > Unexpected broker disappearance in broker list after updating default broker > metadata > - > > Key: TUBEMQ-265 > URL: https://issues.apache.org/jira/browse/TUBEMQ-265 > Project: Apache TubeMQ > Issue Type: Bug > Components: Broker, Master >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > It's a bug on 0.5.0 after migration of > [TUBEMQ-126|https://issues.apache.org/jira/projects/TUBEMQ/issues/TUBEMQ-126]: > broker would disappear after updating its default metadata on > unflushDataHold, which resulting an "out-of-sync" issue on metadata > management. > Reproduce steps: > # Bring a broker online; > # Use webapi to submit an update about type of > "admin_update_broker_configure", setting a new unflushDataHold; > # Monitor log, it would successfully fire this event and proceed the update; > # Refresh the broker list on master, you will find it's disappeared; > # Retry to add the broker with the same BrokerID and IP Address, it would > raise an error about duplicated configuration found, but the broker is still > unseen in the broker list. > Bug is relating to BDB and Master initialization and meta-replication > routines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (TUBEMQ-126) Increase the unflushed data bytes control
[ https://issues.apache.org/jira/browse/TUBEMQ-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou closed TUBEMQ-126. > Increase the unflushed data bytes control > - > > Key: TUBEMQ-126 > URL: https://issues.apache.org/jira/browse/TUBEMQ-126 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Labels: pull-request-available, suspended > Attachments: screenshot-1.png > > Time Spent: 3h > Remaining Estimate: 0h > > Currently, there are unflushThreshold indicating the number of unwashed disks > and unflushInterval indicating the maximum interval of unwashed disks. > Through analysis, data flushing check should increase the unflushed data > bytes control: > !screenshot-1.png! > The unflushed indicators are setted to threshold to monitor and make the > possibility of data loss of within the controllable range, but the time > interval and the number of messages are not enough, for example, the same > 1000 messages, in the case of a 100-byte message and a 512K-byte message, we > not only need to check the number of entries, but also need to check the > accumulated total data size of the unwashed disk > For this problem, the new version plans to adjust to increase the > unflushDataSize indicator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TUBEMQ-265) Unexpected broker disappearance in broker list after updating default broker metadata
[ https://issues.apache.org/jira/browse/TUBEMQ-265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou resolved TUBEMQ-265. -- Resolution: Fixed Tested and Verified OK by PR#180. > Unexpected broker disappearance in broker list after updating default broker > metadata > - > > Key: TUBEMQ-265 > URL: https://issues.apache.org/jira/browse/TUBEMQ-265 > Project: Apache TubeMQ > Issue Type: Bug > Components: Broker, Master >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 20m > Remaining Estimate: 0h > > It's a bug on 0.5.0 after migration of > [TUBEMQ-126|https://issues.apache.org/jira/projects/TUBEMQ/issues/TUBEMQ-126]: > broker would disappear after updating its default metadata on > unflushDataHold, which resulting an "out-of-sync" issue on metadata > management. > Reproduce steps: > # Bring a broker online; > # Use webapi to submit an update about type of > "admin_update_broker_configure", setting a new unflushDataHold; > # Monitor log, it would successfully fire this event and proceed the update; > # Refresh the broker list on master, you will find it's disappeared; > # Retry to add the broker with the same BrokerID and IP Address, it would > raise an error about duplicated configuration found, but the broker is still > unseen in the broker list. > Bug is relating to BDB and Master initialization and meta-replication > routines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TUBEMQ-265) Unexpected broker disappearance in broker list after updating default broker metadata
Jeff Zhou created TUBEMQ-265: Summary: Unexpected broker disappearance in broker list after updating default broker metadata Key: TUBEMQ-265 URL: https://issues.apache.org/jira/browse/TUBEMQ-265 Project: Apache TubeMQ Issue Type: Bug Components: Broker, Master Reporter: Jeff Zhou Fix For: 0.5.0 It's a bug on 0.5.0 after migration of [TUBEMQ-126|https://issues.apache.org/jira/projects/TUBEMQ/issues/TUBEMQ-126]: broker would disappear after updating its default metadata on unflushDataHold, which resulting an "out-of-sync" issue on metadata management. Reproduce steps: # Bring a broker online; # Use webapi to submit an update about type of "admin_update_broker_configure", setting a new unflushDataHold; # Monitor log, it would successfully fire this event and proceed the update; # Refresh the broker list on master, you will find it's disappeared; # Retry to add the broker with the same BrokerID and IP Address, it would raise an error about duplicated configuration found, but the broker is still unseen in the broker list. Bug is relating to BDB and Master initialization and meta-replication routines. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (TUBEMQ-226) Add Windows startup scripts
[ https://issues.apache.org/jira/browse/TUBEMQ-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou closed TUBEMQ-226. Finished, nothing else for this request. > Add Windows startup scripts > --- > > Key: TUBEMQ-226 > URL: https://issues.apache.org/jira/browse/TUBEMQ-226 > Project: Apache TubeMQ > Issue Type: Improvement >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: High > Labels: pull-request-available, ready-to-commit > Fix For: 0.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently there's no starting up on Windows platform, one has to set much to > start relating servers, while manually control each options and > specifications. Moreover, it could be difficult to maintain a stable runtime > configurations. > So here it's to add scripts to: > # Setting up the environment on Windows, which is also imported as parameter > settings; > # Startup Master node and Broker node, in the same way as it does in Linux > systems by shell scripts. > At last, it's best to be composed in plain command script (cmd), which could > be widely accepted on various versions of Windows, meanwhile, it's not so > complex to introduce Powershell. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (TUBEMQ-226) Add Windows startup scripts
[ https://issues.apache.org/jira/browse/TUBEMQ-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou resolved TUBEMQ-226. -- Resolution: Fixed PR has been merged. > Add Windows startup scripts > --- > > Key: TUBEMQ-226 > URL: https://issues.apache.org/jira/browse/TUBEMQ-226 > Project: Apache TubeMQ > Issue Type: Improvement >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: High > Labels: pull-request-available, ready-to-commit > Fix For: 0.5.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Currently there's no starting up on Windows platform, one has to set much to > start relating servers, while manually control each options and > specifications. Moreover, it could be difficult to maintain a stable runtime > configurations. > So here it's to add scripts to: > # Setting up the environment on Windows, which is also imported as parameter > settings; > # Startup Master node and Broker node, in the same way as it does in Linux > systems by shell scripts. > At last, it's best to be composed in plain command script (cmd), which could > be widely accepted on various versions of Windows, meanwhile, it's not so > complex to introduce Powershell. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TUBEMQ-126) Increase the unflushed data bytes control
[ https://issues.apache.org/jira/browse/TUBEMQ-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-126: - Labels: pull-request-available suspended (was: pull-request-available) > Increase the unflushed data bytes control > - > > Key: TUBEMQ-126 > URL: https://issues.apache.org/jira/browse/TUBEMQ-126 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Labels: pull-request-available, suspended > Attachments: screenshot-1.png > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there are unflushThreshold indicating the number of unwashed disks > and unflushInterval indicating the maximum interval of unwashed disks. > Through analysis, data flushing check should increase the unflushed data > bytes control: > !screenshot-1.png! > The unflushed indicators are setted to threshold to monitor and make the > possibility of data loss of within the controllable range, but the time > interval and the number of messages are not enough, for example, the same > 1000 messages, in the case of a 100-byte message and a 512K-byte message, we > not only need to check the number of entries, but also need to check the > accumulated total data size of the unwashed disk > For this problem, the new version plans to adjust to increase the > unflushDataSize indicator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TUBEMQ-126) Increase the unflushed data bytes control
[ https://issues.apache.org/jira/browse/TUBEMQ-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135522#comment-17135522 ] Jeff Zhou commented on TUBEMQ-126: -- OK, marked TUBEMQ-126 suspended. Whenever there's revised plan, please inform. > Increase the unflushed data bytes control > - > > Key: TUBEMQ-126 > URL: https://issues.apache.org/jira/browse/TUBEMQ-126 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Labels: pull-request-available > Attachments: screenshot-1.png > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there are unflushThreshold indicating the number of unwashed disks > and unflushInterval indicating the maximum interval of unwashed disks. > Through analysis, data flushing check should increase the unflushed data > bytes control: > !screenshot-1.png! > The unflushed indicators are setted to threshold to monitor and make the > possibility of data loss of within the controllable range, but the time > interval and the number of messages are not enough, for example, the same > 1000 messages, in the case of a 100-byte message and a 512K-byte message, we > not only need to check the number of entries, but also need to check the > accumulated total data size of the unwashed disk > For this problem, the new version plans to adjust to increase the > unflushDataSize indicator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TUBEMQ-124) Structured index storage
[ https://issues.apache.org/jira/browse/TUBEMQ-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135518#comment-17135518 ] Jeff Zhou commented on TUBEMQ-124: -- Yes, that's correct about overhead of index, currently the phase is on WHICH level of index we COULD put to best exploit the negative-prediction efficiency, and then the use case about whether "interest clients" are sparse enough. As the index affects on such a bottom layer of system, ultimately it's better to be controlled by something like "Coefficient of Sparse" inside of SegmentList I think, though it's always a good start to make it a system low-level option, since there's also chance to aggregate them to be a use-case option. Thanks for your advice. > Structured index storage > > > Key: TUBEMQ-124 > URL: https://issues.apache.org/jira/browse/TUBEMQ-124 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Labels: pull-request-available > Attachments: s15917036711158.png, screenshot-1.png > > Time Spent: 20m > Remaining Estimate: 0h > > 1. Structured index storage: optimize the current index storage, for example, > increase the structured index storage, which can be quickly retrieved through > the index when in use to quickly locate the data; the increase in the index > structure may make the write request slower, At the same time, it takes more > time to check and restore the index when the system restarts > -- > To solve this problem, I plan to implement it like this: > !screenshot-1.png! > The first add 2 bytes of version information at the end of the segment file, > then, divide the datas to bucket in index segment file, and use the Bloom > filter algorithm to save the position of the filter item for each data in the > bucket. After this improvement, there are level 2 indexes in the index > segment file, the Bloom filter bitmap is the first level, and the index > bucket with message index information is the second level. > When filtering consumption, the system first searches whether the filter item > exists in the corresponding data bucket from the first level. If it does not > exist, it continues to search for the existence of the next data bucket until > the index segment file is completed and the filter is switched to the next > index segment file; if the filter item is in a data bucket, the data in the > corresponding data bucket will be read according to the current index file > retrieval method. > Implementation effect estimation: The results of using the Bloom filter > algorithm to locate the results are not guaranteed to be unique, but they > should be improved compared to the current item-by-item inspection, at least > in the worst case, the filtering effect is consistent; and it will be a very > good help if the sparse and non-colliding index item collection. The impact > is that we need additional index storage space, and index file recovery > requires special attention. > If the design needs to be implemented, I think the following points need to > be considered: > 1. Due to the addition of a bitmap index, the checkpoint file needs to be > added to the index store, so, when the system is restarted we can know the > starting checkpoint of the index file; > 2. Due to the change in file structure, before releasing the version of this > feature, we need to first release a historical version compatible with this > feature to solve the system rollback problem after this feature version is > upgraded abnormally. I think that this is a one-time operation, the price is > worth it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TUBEMQ-126) Increase the unflushed data bytes control
[ https://issues.apache.org/jira/browse/TUBEMQ-126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17135498#comment-17135498 ] Jeff Zhou commented on TUBEMQ-126: -- I see recently the PR of TUBEMQ-126 has been ceased, is there any modification on the design? Through TUBEMQ-123, it seems FileStorage is the next downstream layer before actual flush, so is there architectural change on the plan of TUBEMQ-110? > Increase the unflushed data bytes control > - > > Key: TUBEMQ-126 > URL: https://issues.apache.org/jira/browse/TUBEMQ-126 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Labels: pull-request-available > Attachments: screenshot-1.png > > Time Spent: 10m > Remaining Estimate: 0h > > Currently, there are unflushThreshold indicating the number of unwashed disks > and unflushInterval indicating the maximum interval of unwashed disks. > Through analysis, data flushing check should increase the unflushed data > bytes control: > !screenshot-1.png! > The unflushed indicators are setted to threshold to monitor and make the > possibility of data loss of within the controllable range, but the time > interval and the number of messages are not enough, for example, the same > 1000 messages, in the case of a 100-byte message and a 512K-byte message, we > not only need to check the number of entries, but also need to check the > accumulated total data size of the unwashed disk > For this problem, the new version plans to adjust to increase the > unflushDataSize indicator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TUBEMQ-233) Performance Improvement on FileSegmentList
[ https://issues.apache.org/jira/browse/TUBEMQ-233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-233: - Description: Current implementation of FileSegmentList is based on *native compact array* (Object[]), so: 1. it lacks flexibility on add/remove; 2. it would suffer a great performance impact while applying CAS performing an operation; 3. it may cause phantom read as multiple temporary copy of candidates may fail to announce its update due to unintended currentView change; 4. less space efficiency, as there might be multiple temporary copies in system in order to do an update. So this issue is raised to optimize the storage list, and it's to exploit *RoundQueue* structure to achieve a better performance while make use of *weak CAS* to better adapt multi-threaded runtime. was: Current implementation of FileSegmentList is based on native compact array (Object[]), so: 1. it lacks flexibility on add/remove; 2. it would suffer a great performance impact while applying CAS performing an operation; 3. it may cause phantom read as multiple temporary copy of candidates may fail to announce its update due to unintended currentView change; 4. less space efficiency, as there might be multiple temporary copies in system in order to do an update. So this issue is raised to optimize the storage list, and it's to exploit RoundQueue structure to achieve a better performance while make use of weak CAS to better adapt multi-threaded runtime. > Performance Improvement on FileSegmentList > -- > > Key: TUBEMQ-233 > URL: https://issues.apache.org/jira/browse/TUBEMQ-233 > Project: Apache TubeMQ > Issue Type: Improvement > Components: Broker >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: Major > Labels: performance > > Current implementation of FileSegmentList is based on *native compact array* > (Object[]), so: > 1. it lacks flexibility on add/remove; > 2. it would suffer a great performance impact while applying CAS performing > an operation; > 3. it may cause phantom read as multiple temporary copy of candidates may > fail to announce its update due to unintended currentView change; > 4. less space efficiency, as there might be multiple temporary copies in > system in order to do an update. > So this issue is raised to optimize the storage list, and it's to exploit > *RoundQueue* structure to achieve a better performance while make use of > *weak CAS* to better adapt multi-threaded runtime. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TUBEMQ-233) Performance Improvement on FileSegmentList
Jeff Zhou created TUBEMQ-233: Summary: Performance Improvement on FileSegmentList Key: TUBEMQ-233 URL: https://issues.apache.org/jira/browse/TUBEMQ-233 Project: Apache TubeMQ Issue Type: Improvement Components: Broker Reporter: Jeff Zhou Assignee: Jeff Zhou Current implementation of FileSegmentList is based on native compact array (Object[]), so: 1. it lacks flexibility on add/remove; 2. it would suffer a great performance impact while applying CAS performing an operation; 3. it may cause phantom read as multiple temporary copy of candidates may fail to announce its update due to unintended currentView change; 4. less space efficiency, as there might be multiple temporary copies in system in order to do an update. So this issue is raised to optimize the storage list, and it's to exploit RoundQueue structure to achieve a better performance while make use of weak CAS to better adapt multi-threaded runtime. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TUBEMQ-124) Structured index storage
[ https://issues.apache.org/jira/browse/TUBEMQ-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17129198#comment-17129198 ] Jeff Zhou commented on TUBEMQ-124: -- !s15917036711158.png! As shown in arch diagram, I think there could be 3 levels that we could put the Bloom Filter, which does quick negative-prediction on getting rid of diving deeper, like traversing a tree by DFS. Provided the penalty of requiring rebuilding Bloom Filter after a long run with topics in and out, L1 *may* cause severe performance drop by long STP (stop-the-world) by writing lock during rebuild, and L3 *may* contribute little if the scale of IndexEntity is not enormous enough. May I have your input on above cases? Personally I think maybe Bloom Filter only on L2 and L3 is a reasonable choice, and we may increase the scale of FileSegment in order to take advantage of this architecture. > Structured index storage > > > Key: TUBEMQ-124 > URL: https://issues.apache.org/jira/browse/TUBEMQ-124 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Attachments: s15917036711158.png, screenshot-1.png > > > 1. Structured index storage: optimize the current index storage, for example, > increase the structured index storage, which can be quickly retrieved through > the index when in use to quickly locate the data; the increase in the index > structure may make the write request slower, At the same time, it takes more > time to check and restore the index when the system restarts > -- > To solve this problem, I plan to implement it like this: > !screenshot-1.png! > The first add 2 bytes of version information at the end of the segment file, > then, divide the datas to bucket in index segment file, and use the Bloom > filter algorithm to save the position of the filter item for each data in the > bucket. After this improvement, there are level 2 indexes in the index > segment file, the Bloom filter bitmap is the first level, and the index > bucket with message index information is the second level. > When filtering consumption, the system first searches whether the filter item > exists in the corresponding data bucket from the first level. If it does not > exist, it continues to search for the existence of the next data bucket until > the index segment file is completed and the filter is switched to the next > index segment file; if the filter item is in a data bucket, the data in the > corresponding data bucket will be read according to the current index file > retrieval method. > Implementation effect estimation: The results of using the Bloom filter > algorithm to locate the results are not guaranteed to be unique, but they > should be improved compared to the current item-by-item inspection, at least > in the worst case, the filtering effect is consistent; and it will be a very > good help if the sparse and non-colliding index item collection. The impact > is that we need additional index storage space, and index file recovery > requires special attention. > If the design needs to be implemented, I think the following points need to > be considered: > 1. Due to the addition of a bitmap index, the checkpoint file needs to be > added to the index store, so, when the system is restarted we can know the > starting checkpoint of the index file; > 2. Due to the change in file structure, before releasing the version of this > feature, we need to first release a historical version compatible with this > feature to solve the system rollback problem after this feature version is > upgraded abnormally. I think that this is a one-time operation, the price is > worth it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TUBEMQ-124) Structured index storage
[ https://issues.apache.org/jira/browse/TUBEMQ-124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-124: - Attachment: s15917036711158.png > Structured index storage > > > Key: TUBEMQ-124 > URL: https://issues.apache.org/jira/browse/TUBEMQ-124 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Attachments: s15917036711158.png, screenshot-1.png > > > 1. Structured index storage: optimize the current index storage, for example, > increase the structured index storage, which can be quickly retrieved through > the index when in use to quickly locate the data; the increase in the index > structure may make the write request slower, At the same time, it takes more > time to check and restore the index when the system restarts > -- > To solve this problem, I plan to implement it like this: > !screenshot-1.png! > The first add 2 bytes of version information at the end of the segment file, > then, divide the datas to bucket in index segment file, and use the Bloom > filter algorithm to save the position of the filter item for each data in the > bucket. After this improvement, there are level 2 indexes in the index > segment file, the Bloom filter bitmap is the first level, and the index > bucket with message index information is the second level. > When filtering consumption, the system first searches whether the filter item > exists in the corresponding data bucket from the first level. If it does not > exist, it continues to search for the existence of the next data bucket until > the index segment file is completed and the filter is switched to the next > index segment file; if the filter item is in a data bucket, the data in the > corresponding data bucket will be read according to the current index file > retrieval method. > Implementation effect estimation: The results of using the Bloom filter > algorithm to locate the results are not guaranteed to be unique, but they > should be improved compared to the current item-by-item inspection, at least > in the worst case, the filtering effect is consistent; and it will be a very > good help if the sparse and non-colliding index item collection. The impact > is that we need additional index storage space, and index file recovery > requires special attention. > If the design needs to be implemented, I think the following points need to > be considered: > 1. Due to the addition of a bitmap index, the checkpoint file needs to be > added to the index store, so, when the system is restarted we can know the > starting checkpoint of the index file; > 2. Due to the change in file structure, before releasing the version of this > feature, we need to first release a historical version compatible with this > feature to solve the system rollback problem after this feature version is > upgraded abnormally. I think that this is a one-time operation, the price is > worth it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (TUBEMQ-124) Structured index storage
[ https://issues.apache.org/jira/browse/TUBEMQ-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128999#comment-17128999 ] Jeff Zhou edited comment on TUBEMQ-124 at 6/9/20, 8:41 AM: --- I think there are several major issues directly from the initial plan, which contains (but not limited to): 1. Bloom Filter, as a probability prediction set, does *not* support a *remove* operation, so we have to choose an *appropriate* time to rebuild it after a long run with massive topics in and out, which is such an expensive thing just like the GC STP (stop-the-world); 2. Fine-grained control might be introduced to *scatter the impact* while *exploit efficiency* provided by negative-prediction. Like we could do something like Java 8 revised ConcurrentHashMap, we could setup Bloom Filter on *Segment* (even available for memory prediction), and apply something like URA (unbalanced resource allocation) to better concentrate messages of the same topic into the same segment. Please review the prototype about the overall architecture of how Bloom Filter (my opinion) could behave, and let's discuss something in details about above factors, if any or more. Do you have any input or idea about above topics? was (Author: hystericalhell): I think there are several major issues directly from the initial plan, which contains (but not limited to): 1. Bloom Filter, as a probability prediction set, does **not** support a **remove** operation, so we have to choose an *appropriate* time to rebuild it after a long run with massive topics in and out, which is such an expensive thing just like the GC STP (stop-the-world); 2. Fine-grained control might be introduced to scatter the impact while exploit the negative-prediction efficiency, like we could do something like Java 8 revised ConcurrentHashMap, we could setup Bloom Filter on Segment (even available for memory prediction), and apply something like URA (unbalanced resource allocation) to better concentrate messages of the same topic into the same segment. Please review the prototype about the overall architecture of how Bloom Filter (my opinion) could behave, and let's discuss something in details about above factors, if any or more. Do you have any input or idea about above topics? > Structured index storage > > > Key: TUBEMQ-124 > URL: https://issues.apache.org/jira/browse/TUBEMQ-124 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Attachments: screenshot-1.png > > > 1. Structured index storage: optimize the current index storage, for example, > increase the structured index storage, which can be quickly retrieved through > the index when in use to quickly locate the data; the increase in the index > structure may make the write request slower, At the same time, it takes more > time to check and restore the index when the system restarts > -- > To solve this problem, I plan to implement it like this: > !screenshot-1.png! > The first add 2 bytes of version information at the end of the segment file, > then, divide the datas to bucket in index segment file, and use the Bloom > filter algorithm to save the position of the filter item for each data in the > bucket. After this improvement, there are level 2 indexes in the index > segment file, the Bloom filter bitmap is the first level, and the index > bucket with message index information is the second level. > When filtering consumption, the system first searches whether the filter item > exists in the corresponding data bucket from the first level. If it does not > exist, it continues to search for the existence of the next data bucket until > the index segment file is completed and the filter is switched to the next > index segment file; if the filter item is in a data bucket, the data in the > corresponding data bucket will be read according to the current index file > retrieval method. > Implementation effect estimation: The results of using the Bloom filter > algorithm to locate the results are not guaranteed to be unique, but they > should be improved compared to the current item-by-item inspection, at least > in the worst case, the filtering effect is consistent; and it will be a very > good help if the sparse and non-colliding index item collection. The impact > is that we need additional index storage space, and index file recovery > requires special attention. > If the design needs to be implemented, I think the following points need to > be considered: > 1. Due to the addition of a bitmap index, the checkpoint file needs to be > added to the index store, so, when the system is restarted we can know the > starting
[jira] [Comment Edited] (TUBEMQ-124) Structured index storage
[ https://issues.apache.org/jira/browse/TUBEMQ-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128999#comment-17128999 ] Jeff Zhou edited comment on TUBEMQ-124 at 6/9/20, 8:41 AM: --- I think there are several major issues directly from the initial plan, which contains (but not limited to): 1. Bloom Filter, as a probability prediction set, does *not* support a *remove* operation, so we have to choose an *appropriate* time to rebuild it after a long run with massive topics in and out, which is such an expensive thing just like the GC STP (stop-the-world); 2. Fine-grained control might be introduced to *scatter impact* while *exploit efficiency* provided by negative-prediction. Like we could do something like Java 8 revised ConcurrentHashMap, we could setup Bloom Filter on *Segment* (even available for memory prediction), and apply something like URA (unbalanced resource allocation) to better concentrate messages of the same topic into the same segment. Please review the prototype about the overall architecture of how Bloom Filter (my opinion) could behave, and let's discuss something in details about above factors, if any or more. Do you have any input or idea about above topics? was (Author: hystericalhell): I think there are several major issues directly from the initial plan, which contains (but not limited to): 1. Bloom Filter, as a probability prediction set, does *not* support a *remove* operation, so we have to choose an *appropriate* time to rebuild it after a long run with massive topics in and out, which is such an expensive thing just like the GC STP (stop-the-world); 2. Fine-grained control might be introduced to *scatter the impact* while *exploit efficiency* provided by negative-prediction. Like we could do something like Java 8 revised ConcurrentHashMap, we could setup Bloom Filter on *Segment* (even available for memory prediction), and apply something like URA (unbalanced resource allocation) to better concentrate messages of the same topic into the same segment. Please review the prototype about the overall architecture of how Bloom Filter (my opinion) could behave, and let's discuss something in details about above factors, if any or more. Do you have any input or idea about above topics? > Structured index storage > > > Key: TUBEMQ-124 > URL: https://issues.apache.org/jira/browse/TUBEMQ-124 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Attachments: screenshot-1.png > > > 1. Structured index storage: optimize the current index storage, for example, > increase the structured index storage, which can be quickly retrieved through > the index when in use to quickly locate the data; the increase in the index > structure may make the write request slower, At the same time, it takes more > time to check and restore the index when the system restarts > -- > To solve this problem, I plan to implement it like this: > !screenshot-1.png! > The first add 2 bytes of version information at the end of the segment file, > then, divide the datas to bucket in index segment file, and use the Bloom > filter algorithm to save the position of the filter item for each data in the > bucket. After this improvement, there are level 2 indexes in the index > segment file, the Bloom filter bitmap is the first level, and the index > bucket with message index information is the second level. > When filtering consumption, the system first searches whether the filter item > exists in the corresponding data bucket from the first level. If it does not > exist, it continues to search for the existence of the next data bucket until > the index segment file is completed and the filter is switched to the next > index segment file; if the filter item is in a data bucket, the data in the > corresponding data bucket will be read according to the current index file > retrieval method. > Implementation effect estimation: The results of using the Bloom filter > algorithm to locate the results are not guaranteed to be unique, but they > should be improved compared to the current item-by-item inspection, at least > in the worst case, the filtering effect is consistent; and it will be a very > good help if the sparse and non-colliding index item collection. The impact > is that we need additional index storage space, and index file recovery > requires special attention. > If the design needs to be implemented, I think the following points need to > be considered: > 1. Due to the addition of a bitmap index, the checkpoint file needs to be > added to the index store, so, when the system is restarted we can know the > starting
[jira] [Commented] (TUBEMQ-124) Structured index storage
[ https://issues.apache.org/jira/browse/TUBEMQ-124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128999#comment-17128999 ] Jeff Zhou commented on TUBEMQ-124: -- I think there are several major issues directly from the initial plan, which contains (but not limited to): 1. Bloom Filter, as a probability prediction set, does **not** support a **remove** operation, so we have to choose an *appropriate* time to rebuild it after a long run with massive topics in and out, which is such an expensive thing just like the GC STP (stop-the-world); 2. Fine-grained control might be introduced to scatter the impact while exploit the negative-prediction efficiency, like we could do something like Java 8 revised ConcurrentHashMap, we could setup Bloom Filter on Segment (even available for memory prediction), and apply something like URA (unbalanced resource allocation) to better concentrate messages of the same topic into the same segment. Please review the prototype about the overall architecture of how Bloom Filter (my opinion) could behave, and let's discuss something in details about above factors, if any or more. Do you have any input or idea about above topics? > Structured index storage > > > Key: TUBEMQ-124 > URL: https://issues.apache.org/jira/browse/TUBEMQ-124 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Attachments: screenshot-1.png > > > 1. Structured index storage: optimize the current index storage, for example, > increase the structured index storage, which can be quickly retrieved through > the index when in use to quickly locate the data; the increase in the index > structure may make the write request slower, At the same time, it takes more > time to check and restore the index when the system restarts > -- > To solve this problem, I plan to implement it like this: > !screenshot-1.png! > The first add 2 bytes of version information at the end of the segment file, > then, divide the datas to bucket in index segment file, and use the Bloom > filter algorithm to save the position of the filter item for each data in the > bucket. After this improvement, there are level 2 indexes in the index > segment file, the Bloom filter bitmap is the first level, and the index > bucket with message index information is the second level. > When filtering consumption, the system first searches whether the filter item > exists in the corresponding data bucket from the first level. If it does not > exist, it continues to search for the existence of the next data bucket until > the index segment file is completed and the filter is switched to the next > index segment file; if the filter item is in a data bucket, the data in the > corresponding data bucket will be read according to the current index file > retrieval method. > Implementation effect estimation: The results of using the Bloom filter > algorithm to locate the results are not guaranteed to be unique, but they > should be improved compared to the current item-by-item inspection, at least > in the worst case, the filtering effect is consistent; and it will be a very > good help if the sparse and non-colliding index item collection. The impact > is that we need additional index storage space, and index file recovery > requires special attention. > If the design needs to be implemented, I think the following points need to > be considered: > 1. Due to the addition of a bitmap index, the checkpoint file needs to be > added to the index store, so, when the system is restarted we can know the > starting checkpoint of the index file; > 2. Due to the change in file structure, before releasing the version of this > feature, we need to first release a historical version compatible with this > feature to solve the system rollback problem after this feature version is > upgraded abnormally. I think that this is a one-time operation, the price is > worth it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TUBEMQ-124) Structured index storage
[ https://issues.apache.org/jira/browse/TUBEMQ-124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou reassigned TUBEMQ-124: Assignee: Jeff Zhou > Structured index storage > > > Key: TUBEMQ-124 > URL: https://issues.apache.org/jira/browse/TUBEMQ-124 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Attachments: screenshot-1.png > > > 1. Structured index storage: optimize the current index storage, for example, > increase the structured index storage, which can be quickly retrieved through > the index when in use to quickly locate the data; the increase in the index > structure may make the write request slower, At the same time, it takes more > time to check and restore the index when the system restarts > -- > To solve this problem, I plan to implement it like this: > !screenshot-1.png! > The first add 2 bytes of version information at the end of the segment file, > then, divide the datas to bucket in index segment file, and use the Bloom > filter algorithm to save the position of the filter item for each data in the > bucket. After this improvement, there are level 2 indexes in the index > segment file, the Bloom filter bitmap is the first level, and the index > bucket with message index information is the second level. > When filtering consumption, the system first searches whether the filter item > exists in the corresponding data bucket from the first level. If it does not > exist, it continues to search for the existence of the next data bucket until > the index segment file is completed and the filter is switched to the next > index segment file; if the filter item is in a data bucket, the data in the > corresponding data bucket will be read according to the current index file > retrieval method. > Implementation effect estimation: The results of using the Bloom filter > algorithm to locate the results are not guaranteed to be unique, but they > should be improved compared to the current item-by-item inspection, at least > in the worst case, the filtering effect is consistent; and it will be a very > good help if the sparse and non-colliding index item collection. The impact > is that we need additional index storage space, and index file recovery > requires special attention. > If the design needs to be implemented, I think the following points need to > be considered: > 1. Due to the addition of a bitmap index, the checkpoint file needs to be > added to the index store, so, when the system is restarted we can know the > starting checkpoint of the index file; > 2. Due to the change in file structure, before releasing the version of this > feature, we need to first release a historical version compatible with this > feature to solve the system rollback problem after this feature version is > upgraded abnormally. I think that this is a one-time operation, the price is > worth it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TUBEMQ-226) Add Windows startup scripts
[ https://issues.apache.org/jira/browse/TUBEMQ-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou reassigned TUBEMQ-226: Assignee: Jeff Zhou > Add Windows startup scripts > --- > > Key: TUBEMQ-226 > URL: https://issues.apache.org/jira/browse/TUBEMQ-226 > Project: Apache TubeMQ > Issue Type: Improvement >Reporter: Jeff Zhou >Assignee: Jeff Zhou >Priority: High > Labels: ready-to-commit > Fix For: 0.5.0 > > > Currently there's no starting up on Windows platform, one has to set much to > start relating servers, while manually control each options and > specifications. Moreover, it could be difficult to maintain a stable runtime > configurations. > So here it's to add scripts to: > # Setting up the environment on Windows, which is also imported as parameter > settings; > # Startup Master node and Broker node, in the same way as it does in Linux > systems by shell scripts. > At last, it's best to be composed in plain command script (cmd), which could > be widely accepted on various versions of Windows, meanwhile, it's not so > complex to introduce Powershell. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (TUBEMQ-126) Increase the unflushed data bytes control
[ https://issues.apache.org/jira/browse/TUBEMQ-126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou reassigned TUBEMQ-126: Assignee: Jeff Zhou > Increase the unflushed data bytes control > - > > Key: TUBEMQ-126 > URL: https://issues.apache.org/jira/browse/TUBEMQ-126 > Project: Apache TubeMQ > Issue Type: Sub-task >Reporter: Guocheng Zhang >Assignee: Jeff Zhou >Priority: Major > Attachments: screenshot-1.png > > > Currently, there are unflushThreshold indicating the number of unwashed disks > and unflushInterval indicating the maximum interval of unwashed disks. > Through analysis, data flushing check should increase the unflushed data > bytes control: > !screenshot-1.png! > The unflushed indicators are setted to threshold to monitor and make the > possibility of data loss of within the controllable range, but the time > interval and the number of messages are not enough, for example, the same > 1000 messages, in the case of a 100-byte message and a 512K-byte message, we > not only need to check the number of entries, but also need to check the > accumulated total data size of the unwashed disk > For this problem, the new version plans to adjust to increase the > unflushDataSize indicator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (TUBEMQ-226) Add Windows startup scripts
[ https://issues.apache.org/jira/browse/TUBEMQ-226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeff Zhou updated TUBEMQ-226: - Fix Version/s: 0.5.0 > Add Windows startup scripts > --- > > Key: TUBEMQ-226 > URL: https://issues.apache.org/jira/browse/TUBEMQ-226 > Project: Apache TubeMQ > Issue Type: Improvement >Reporter: Jeff Zhou >Priority: High > Labels: ready-to-commit > Fix For: 0.5.0 > > > Currently there's no starting up on Windows platform, one has to set much to > start relating servers, while manually control each options and > specifications. Moreover, it could be difficult to maintain a stable runtime > configurations. > So here it's to add scripts to: > # Setting up the environment on Windows, which is also imported as parameter > settings; > # Startup Master node and Broker node, in the same way as it does in Linux > systems by shell scripts. > At last, it's best to be composed in plain command script (cmd), which could > be widely accepted on various versions of Windows, meanwhile, it's not so > complex to introduce Powershell. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (TUBEMQ-226) Add Windows startup scripts
Jeff Zhou created TUBEMQ-226: Summary: Add Windows startup scripts Key: TUBEMQ-226 URL: https://issues.apache.org/jira/browse/TUBEMQ-226 Project: Apache TubeMQ Issue Type: Improvement Reporter: Jeff Zhou Currently there's no starting up on Windows platform, one has to set much to start relating servers, while manually control each options and specifications. Moreover, it could be difficult to maintain a stable runtime configurations. So here it's to add scripts to: # Setting up the environment on Windows, which is also imported as parameter settings; # Startup Master node and Broker node, in the same way as it does in Linux systems by shell scripts. At last, it's best to be composed in plain command script (cmd), which could be widely accepted on various versions of Windows, meanwhile, it's not so complex to introduce Powershell. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (TUBEMQ-167) Change to relative paths in default configs
[
https://issues.apache.org/jira/browse/TUBEMQ-167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128250#comment-17128250
]
Jeff Zhou commented on TUBEMQ-167:
--
!s15916013621910.png! As shown in above screenshot, current relative setting
would fail the normal start routine on Windows system. Due to further analysis,
I found it's caused that the "BASE_DIR" seems to be the folder in which the
startup command runs, so would you please add a note on *Each* path attributes
that "Please use absolute path on Windows in case of 'Invalid Resources' error
during server starting up"?
> Change to relative paths in default configs
> ---
>
> Key: TUBEMQ-167
> URL: https://issues.apache.org/jira/browse/TUBEMQ-167
> Project: Apache TubeMQ
> Issue Type: Improvement
>Reporter: Ping Yu
>Assignee: Ping Yu
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: s15916013621910.png
>
>
> Change absolute paths in default configuration files to relative, for ease of
> non-productive deployments.
> E.g.
> {code:java}
> [master]
> webResourcePath=E:\\GIT\\TubeMQ\\resources --> ./resources{code}
>
> Solution:
> Change cwd to `$BASE_DIR` before invoke the `Main` class. To preserve user
> cwd, a pair of `pushd/popd` is needed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (TUBEMQ-167) Change to relative paths in default configs
[
https://issues.apache.org/jira/browse/TUBEMQ-167?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeff Zhou updated TUBEMQ-167:
-
Attachment: s15916013621910.png
> Change to relative paths in default configs
> ---
>
> Key: TUBEMQ-167
> URL: https://issues.apache.org/jira/browse/TUBEMQ-167
> Project: Apache TubeMQ
> Issue Type: Improvement
>Reporter: Ping Yu
>Assignee: Ping Yu
>Priority: Trivial
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Attachments: s15916013621910.png
>
>
> Change absolute paths in default configuration files to relative, for ease of
> non-productive deployments.
> E.g.
> {code:java}
> [master]
> webResourcePath=E:\\GIT\\TubeMQ\\resources --> ./resources{code}
>
> Solution:
> Change cwd to `$BASE_DIR` before invoke the `Main` class. To preserve user
> cwd, a pair of `pushd/popd` is needed.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
