Re: Extending the lease duration on messages received with Proton?

[aconway]

On Mon, 2015-12-07 at 10:29 +, Robbie Gemmell wrote:
> On 4 December 2015 at 19:56, Philippe Le Rohellec 
> wrote:
> > I'm using Proton to receive messages from an Azure Service Bus
> > queue.
> > The receiver keeps a lock on each message it receives for a
> > predefined
> > amount of time, which I think matches the "Lock Duration" set on
> > the queue
> > on the Azure side. The max value in Azure is 5 minutes.
> > 
> > What if processing the message takes more than 5 minutes? Is there
> > a way to
> > renew the lock using Proton or are the 5 minutes non negotiable? I
> > didn't
> > find anything in the API.
> > 
> > 
> > 
> > --
> > View this message in context: http://qpid.2158936.n2.nabble.com/Ext
> > ending-the-lease-duration-on-messages-received-with-Proton
> > -tp7634944.html
> > Sent from the Apache Qpid Proton mailing list archive at
> > Nabble.com.
> 
> Hi Philippe,
> 
> I believe the locking you mention is handled server-side, I'm not
> sure
> that the Proton reciever is ever aware of it. I think you'll need to
> ask the Azure folks about whether there is a way to adjust/renew the
> lock.
> 

May be a conflict of terminology here: AMQP doesn't define a "lock" but
it does define "settlement" of a message. Messages can be sent "pre
settled" which means the sender just fires-and-forgets, or "unsettled"
which means the sender is expecting some acknowledgement. I don't know
Azure but on most brokers (or any system with reliable delivery) that
does amount to a "lock" - the broker sends a message unsettled and will
not send it to any other receiver until it is acknolwedged.

AMQP lets the receiver respond to a message with "accept" (I took it),
"reject" (the message is invalid) or "release" (I didn't take it but
you can try somebody else.) Proton lets you control what response the
receiver sends back and when.

AMQP also lets you define a message "timeout" which says "if the
message is not processed by this time, forget it." So some combination
of acks and timeouts will probably let you do what you want.

(Obviously using timeouts in a network creates the possibility of
occasional duplicate processing, but if you have pre-emptive server
-side lock breaking in Aure you probably know that already ;)

Cheers,
Alan.

Re: Acknowledging received messages with Proton and an Azure Service Bus queue

[aconway]

On Fri, 2015-12-04 at 12:20 -0700, plerohel wrote:
> Thanks James, that article is extremely helpful.
> After I set the messenger incoming_window to a positive value,
> messages do
> get acknowledged as expected.

> Rejecting the message doesn't work though. It might be because I'm
> not
> setting the settle mode as described in the article. The ruby API
> doesn't
> seem to allow it. However if the receiver just skips the "accept"
> step then
> the message can be processed again by another receiver.

This may be related to

 https://issues.apache.org/jira/browse/PROTON-1067 

If it is can you add your add your comments/reproducers to it?

If it doesn't seem related can you open another JIRA for the Ruby
binding? I also working on the Ruby also at the moment so would be good
to look at these together even if they're not the same issue since they
are at least in related areas. Please attach code reproducers if you
have any.

Cheers,
Alan.

Re: Function pn_message_encode() returns -2

[aconway]

traverse is intended to walk the data structure from the beginning.
Node "1" is the first child node (node 0 is the "root" of the tree).
Nodes are referred to by number rather than pointer because the node
storage can be re-allocated if the pn_data object grows which changes
the node addresses. So while your fix may solve your immediate problem
it is not correct in general.

Can you post a small C program that illustrates the error? That would
help to figure out the problem.

On Sun, 2015-12-06 at 23:36 -0700, sanddune008 wrote:
> I see that following function pn_message_encode() returns -2 with
> send.c
> example.
> 
> My findings, Kindly review the below code:
> 
> Changes made to fix is highlighted in bold, What is the significance
> of
> passing '1' first node? 
> 
> int *pni_data_traverse*(pn_data_t *data,
>   int (*enter)(void *ctx, pn_data_t *data,
> pni_node_t
> *node),
>   int (*exit)(void *ctx, pn_data_t *data,
> pni_node_t
> *node),
>   void *ctx)
> {
>   *pni_node_t *node = data->size ? pn_data_node(data, data->current)
> :
> NULL;*
>   //pni_node_t *node = data->size ? pn_data_node(data,* 1*) : NULL;
> 
> 
> Thanks in Advance
> 
> 
> 
> 
> 
> --
> View this message in context: http://qpid.2158936.n2.nabble.com/Funct
> ion-pn-message-encode-returns-2-tp7634948.html
> Sent from the Apache Qpid Proton mailing list archive at Nabble.com.

[jira] [Closed] (PROTON-1052) SSL support in C++ reactor client

[Cliff Jansen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PROTON-1052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cliff Jansen closed PROTON-1052.

Resolution: Fixed

> SSL support in C++ reactor client
> -
>
> Key: PROTON-1052
> URL: https://issues.apache.org/jira/browse/PROTON-1052
> Project: Qpid Proton
>  Issue Type: Improvement
>  Components: cpp-binding
>Affects Versions: 0.12.0
>Reporter: Cliff Jansen
>Assignee: Cliff Jansen
> Fix For: 0.12.0
>
>
> Provide SSL configuration at container level and overrides per outgoing 
> connection and listener



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (PROTON-1053) SASL support in C++ reactor client

[Cliff Jansen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PROTON-1053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Cliff Jansen closed PROTON-1053.

Resolution: Fixed

> SASL support in C++ reactor client
> --
>
> Key: PROTON-1053
> URL: https://issues.apache.org/jira/browse/PROTON-1053
> Project: Qpid Proton
>  Issue Type: Improvement
>  Components: cpp-binding
>Affects Versions: 0.12.0
>Reporter: Cliff Jansen
>Assignee: Cliff Jansen
> Fix For: 0.12.0
>
>
> Provide SASL operation and configuration at the container level and overrides 
> per outgoing connection and listener.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Extending the lease duration on messages received with Proton?

[Robbie Gemmell]

On 4 December 2015 at 19:56, Philippe Le Rohellec  wrote:
> I'm using Proton to receive messages from an Azure Service Bus queue.
> The receiver keeps a lock on each message it receives for a predefined
> amount of time, which I think matches the "Lock Duration" set on the queue
> on the Azure side. The max value in Azure is 5 minutes.
>
> What if processing the message takes more than 5 minutes? Is there a way to
> renew the lock using Proton or are the 5 minutes non negotiable? I didn't
> find anything in the API.
>
>
>
> --
> View this message in context: 
> http://qpid.2158936.n2.nabble.com/Extending-the-lease-duration-on-messages-received-with-Proton-tp7634944.html
> Sent from the Apache Qpid Proton mailing list archive at Nabble.com.

Hi Philippe,

I believe the locking you mention is handled server-side, I'm not sure
that the Proton reciever is ever aware of it. I think you'll need to
ask the Azure folks about whether there is a way to adjust/renew the
lock.

Robbie

Re: Function pn_message_encode() returns -2

[sanddune008]

I awaiting review comments.Kindly confirm.



--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Function-pn-message-encode-returns-2-tp7634948p7634950.html
Sent from the Apache Qpid Proton mailing list archive at Nabble.com.

Re: Extending the lease duration on messages received with Proton?

[Philippe Le Rohellec]

Yes, that's my understanding too. In unsettled mode, once a message is
received by a receiver, no other receiver will get it until the first
receiver rejects it OR, if the backend queue has a limit on the lock
duration and the receiver neither accepts or rejects, until the lock
expires. Robbie is right, this lock is a server side thing.

For example I believe RabbitMQ doesn't have such a thing as a lock duration
but it will release the message if the receiver connection is closed before
an accept or reject is received.
Azure Service Bus queue has a lock duration with a 5 minute maximum value.
The lock can be renewed through the Azure API but not through Proton, which
makes sense if that notion of lock duration is not part of the AMQP
protocol.



--
View this message in context: 
http://qpid.2158936.n2.nabble.com/Extending-the-lease-duration-on-messages-received-with-Proton-tp7634944p7634967.html
Sent from the Apache Qpid Proton mailing list archive at Nabble.com.

[jira] [Commented] (PROTON-1071) EventInjector hangs on Windows

[Cliff Jansen (JIRA)]

[ 
https://issues.apache.org/jira/browse/PROTON-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045851#comment-15045851
 ] 

Cliff Jansen commented on PROTON-1071:
--

This JIRA cuts across a number of Proton IO and threading issues.

EventInjector as coded violates Proton's io model: the pipe needs to come from 
pn_pipe() and the reading and writing via pn_read() and pn_write(), all on the 
reactor's pn_io_t object.  It works by fortuitous accident on Posix.

Coding EventInjector to work this way for Posix is presumably simple.  For 
Windows, it further requires tackling deferred thread safety, at least for 
pipes and the associated pn_io_t->selector (but that may pull in everything).

As noted in PROTON-640, there is no thread safety in Proton io on Windows other 
than a weak guarantee that two separate threads may independently work on 
separate pn_io_t objects.

PROTON-688 attempted to document the stronger but still limited thread-safety 
available in the Posix implementation of Proton io.  This is the model of 
concurrency used by Qpid Dispatch Router and perhaps the model that the Windows 
implementation should strive to.  There is an opposing point of view that 
pushing thread safety into Proton is counterproductive: applications know what 
threading/io model works best for their workload.  Hence the rising interest in 
connection_engine.

EventInjector seems like a pretty important use case for coordinating threads.  
Alternatively, a more limited (or dedicated) api extension, perhaps 
pn_reactor_inject() which does platform-specific thread-safe coordination of 
something to be collected and the pn_io_t->selector might be more clear in its 
intention (as opposed to knowing that pn_write/read/select work a certain way 
together).

The Windows io.c and related code could be made to work at the same level of 
thread safety as Posix, but with obvious locking overhead.  If the assumption 
is that an application needing high performance IO will just use the 
connection_engine and manage the IO itself, perhaps there is no need to care 
about the locking penalty.

See also PROTON-889 regarding general thread safety outages in Proton on all 
platforms for pn_io_error() and pn_io_wouldblock().


> EventInjector hangs on Windows
> --
>
> Key: PROTON-1071
> URL: https://issues.apache.org/jira/browse/PROTON-1071
> Project: Qpid Proton
>  Issue Type: Bug
>  Components: proton-c, python-binding
>Affects Versions: 0.11
> Environment: Windows
>Reporter: Ken Giusti
>Assignee: Cliff Jansen
> Fix For: 0.12.0
>
>
> I added a new reactor test that exercises the python-proton ApplicationEvent 
> and EventInjector classes:
> proton_tests.reactor.ApplicationEventTest.test_application_events
> See tests/python/proton_tests/reactor.py
> This test passes on linux, but hangs when run on Windows.
> Poking around a bit, I suspect the problem may be in the Windows selector 
> code.  Description:
> The EventInjector/ApplicationEvent classes provide a way to trigger events 
> from threads external to the reactor main loop.  See 
> proton-c/bindings/python/proton/reactor.py.  A pipe is used to wake up the 
> reactor when a new event is sent to the reactor (see reactor.py in the python 
> bindings).  The EventInjector's trigger method puts the event on a queue and 
> writes to a pipe to wake up the reactor.  The on_selectable_readable callback 
> in the EventInjector is called on the reactor thread to get the event off the 
> queue and clear the pipe.
> On windows it appears as if the EventInjector selectable is made "readable" 
> even though nothing has been written to the pipe.  This causes the os.read() 
> call in the on_selectable_readable() callback to hang.
> Best I can tell the windows selector code doesn't work properly with a pipe.  
> The pn_selector_next() function is returning a read event on the pipe's read 
> descriptor even though the pipe is empty.  But I'm not familiar with the 
> window's selector implementation, so this is a best guess.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (PROTON-1072) Proton build fails if epydoc is installed

[Ganesh Murthy (JIRA)]

Ganesh Murthy created PROTON-1072:
-

 Summary: Proton build fails if epydoc is installed
 Key: PROTON-1072
 URL: https://issues.apache.org/jira/browse/PROTON-1072
 Project: Qpid Proton
  Issue Type: Bug
  Components: proton-c
Affects Versions: 0.11
Reporter: Ganesh Murthy


Steps to reproduce 
Install epydoc on your system (dnf -y install epydoc)
Now run the proton build
You will see the following error - 
 CMake Warning (dev) at proton-c/bindings/python/CMakeLists.txt:87 
(add_custom_target):
   Syntax error in cmake code at

 /main/qpid-proton/proton-c/bindings/python/CMakeLists.txt:91

   when parsing string

 ${SWIG_MODULE_${cproton}_REAL_NAME

   syntax error, unexpected $end, expecting } (34)

   Policy CMP0010 is not set: Bad variable reference syntax is an error.  Run
   "cmake --help-policy CMP0010" for policy details.  Use the cmake_policy
   command to set the policy and suppress this warning.
 This warning is for project developers.  Use -Wno-dev to suppress it.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (PROTON-1072) Proton build fails if epydoc is installed

[Ganesh Murthy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/PROTON-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ganesh Murthy updated PROTON-1072:
--
Description: 
Steps to reproduce 
Get the latest proton from master.
Install epydoc on your system (dnf -y install epydoc  (I did this on Docker 
using Fedora 23 environment))
Now run the proton build
You will see the following error - 
 CMake Warning (dev) at proton-c/bindings/python/CMakeLists.txt:87 
(add_custom_target):
   Syntax error in cmake code at

 /main/qpid-proton/proton-c/bindings/python/CMakeLists.txt:91

   when parsing string

 ${SWIG_MODULE_${cproton}_REAL_NAME

   syntax error, unexpected $end, expecting } (34)

   Policy CMP0010 is not set: Bad variable reference syntax is an error.  Run
   "cmake --help-policy CMP0010" for policy details.  Use the cmake_policy
   command to set the policy and suppress this warning.
 This warning is for project developers.  Use -Wno-dev to suppress it.


  was:
Steps to reproduce 
Install epydoc on your system (dnf -y install epydoc)
Now run the proton build
You will see the following error - 
 CMake Warning (dev) at proton-c/bindings/python/CMakeLists.txt:87 
(add_custom_target):
   Syntax error in cmake code at

 /main/qpid-proton/proton-c/bindings/python/CMakeLists.txt:91

   when parsing string

 ${SWIG_MODULE_${cproton}_REAL_NAME

   syntax error, unexpected $end, expecting } (34)

   Policy CMP0010 is not set: Bad variable reference syntax is an error.  Run
   "cmake --help-policy CMP0010" for policy details.  Use the cmake_policy
   command to set the policy and suppress this warning.
 This warning is for project developers.  Use -Wno-dev to suppress it.



> Proton build fails if epydoc is installed
> -
>
> Key: PROTON-1072
> URL: https://issues.apache.org/jira/browse/PROTON-1072
> Project: Qpid Proton
>  Issue Type: Bug
>  Components: proton-c
>Affects Versions: 0.11
>Reporter: Ganesh Murthy
>
> Steps to reproduce 
> Get the latest proton from master.
> Install epydoc on your system (dnf -y install epydoc  (I did this on Docker 
> using Fedora 23 environment))
> Now run the proton build
> You will see the following error - 
>  CMake Warning (dev) at proton-c/bindings/python/CMakeLists.txt:87 
> (add_custom_target):
>Syntax error in cmake code at
>  /main/qpid-proton/proton-c/bindings/python/CMakeLists.txt:91
>when parsing string
>  ${SWIG_MODULE_${cproton}_REAL_NAME
>syntax error, unexpected $end, expecting } (34)
>Policy CMP0010 is not set: Bad variable reference syntax is an error.  Run
>"cmake --help-policy CMP0010" for policy details.  Use the cmake_policy
>command to set the policy and suppress this warning.
>  This warning is for project developers.  Use -Wno-dev to suppress it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PROTON-1072) Proton build fails if epydoc is installed

[Ganesh Murthy (JIRA)]

[ 
https://issues.apache.org/jira/browse/PROTON-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045883#comment-15045883
 ] 

Ganesh Murthy commented on PROTON-1072:
---

The build works fine when I commented out the following lines in 
/main/qpid-proton/proton-c/bindings/python/CMakeLists.txt

if (EPYDOC_EXE)
   foreach (py_src_doc ${pysrc})
 list(APPEND PY_DOC_FILES "${CMAKE_CURRENT_SOURCE_DIR}/${py_src_doc}")
   endforeach(py_src_doc)
   add_custom_target(docs-py COMMAND ${PYTHON_EXECUTABLE} 
${CMAKE_CURRENT_SOURCE_DIR}/../../env.py --
 PYTHONPATH=${CMAKE_CURRENT_BINARY_DIR}:${CMAKE_CURRENT_SOURCE_DIR}
 ${EPYDOC_EXE} -v --no-private --html -o ${CMAKE_CURRENT_BINARY_DIR}/html
 ${PY_DOC_FILES}
 DEPENDENCIES ${SWIG_MODULE_${cproton}_REAL_NAME)
   add_dependencies(docs docs-py)
   install(DIRECTORY "${CMAKE_CURRENT_BINARY_DIR}/html/"
   DESTINATION "${PROTON_SHARE}/docs/api-py"
   COMPONENT documentation
   ${OPTIONAL_ARG})
endif (EPYDOC_EXE

> Proton build fails if epydoc is installed
> -
>
> Key: PROTON-1072
> URL: https://issues.apache.org/jira/browse/PROTON-1072
> Project: Qpid Proton
>  Issue Type: Bug
>  Components: proton-c
>Affects Versions: 0.11
>Reporter: Ganesh Murthy
>
> Steps to reproduce 
> Get the latest proton from master.
> Install epydoc on your system (dnf -y install epydoc  (I did this on Docker 
> using Fedora 23 environment))
> Now run the proton build
> You will see the following error - 
>  CMake Warning (dev) at proton-c/bindings/python/CMakeLists.txt:87 
> (add_custom_target):
>Syntax error in cmake code at
>  /main/qpid-proton/proton-c/bindings/python/CMakeLists.txt:91
>when parsing string
>  ${SWIG_MODULE_${cproton}_REAL_NAME
>syntax error, unexpected $end, expecting } (34)
>Policy CMP0010 is not set: Bad variable reference syntax is an error.  Run
>"cmake --help-policy CMP0010" for policy details.  Use the cmake_policy
>command to set the policy and suppress this warning.
>  This warning is for project developers.  Use -Wno-dev to suppress it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PROTON-1068) c++ remove counted_ptr and context types from public API

[ASF subversion and git services (JIRA)]

[ 
https://issues.apache.org/jira/browse/PROTON-1068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045905#comment-15045905
 ] 

ASF subversion and git services commented on PROTON-1068:
-

Commit 73aad7123a3f35785ebde0b961eda2c1c7223d94 in qpid-proton's branch 
refs/heads/master from [~astitcher]
[ https://git-wip-us.apache.org/repos/asf?p=qpid-proton.git;h=73aad71 ]

PROTON-1068: Export encapsulated incref/decref functions
- C++ Build fails on windows otherwise.


> c++ remove counted_ptr and context types from public API
> 
>
> Key: PROTON-1068
> URL: https://issues.apache.org/jira/browse/PROTON-1068
> Project: Qpid Proton
>  Issue Type: Improvement
>Affects Versions: 0.11
>Reporter: Alan Conway
>Assignee: Alan Conway
> Fix For: 0.12.0
>
>
> Remove counted_ptr and context types from public API.
> Remove counted_ptr entirely, use object<> ref-counted pointer internally for  
> uniform, automated memory management.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (PROTON-1071) EventInjector hangs on Windows

[Gordon Sim (JIRA)]

[ 
https://issues.apache.org/jira/browse/PROTON-1071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15045930#comment-15045930
 ] 

Gordon Sim commented on PROTON-1071:


{quote}
EventInjector as coded violates Proton's io model: the pipe needs to come from 
pn_pipe() and the reading and writing via pn_read() and pn_write(), all on the 
reactor's pn_io_t object. It works by fortuitous accident on Posix.
{quote}

I'm not sure I agree. What is your justification for this? The purpose of 
pn_selectable_set_fd surely is to associate any file descriptor with the 
pn_selectable. Having to use proton io operations defeats what I understood to 
be a key purpose of the integration of selectables with the reactor, namely 
providing a way to integrate other things into protons reactor event loop.

> EventInjector hangs on Windows
> --
>
> Key: PROTON-1071
> URL: https://issues.apache.org/jira/browse/PROTON-1071
> Project: Qpid Proton
>  Issue Type: Bug
>  Components: proton-c, python-binding
>Affects Versions: 0.11
> Environment: Windows
>Reporter: Ken Giusti
>Assignee: Cliff Jansen
> Fix For: 0.12.0
>
>
> I added a new reactor test that exercises the python-proton ApplicationEvent 
> and EventInjector classes:
> proton_tests.reactor.ApplicationEventTest.test_application_events
> See tests/python/proton_tests/reactor.py
> This test passes on linux, but hangs when run on Windows.
> Poking around a bit, I suspect the problem may be in the Windows selector 
> code.  Description:
> The EventInjector/ApplicationEvent classes provide a way to trigger events 
> from threads external to the reactor main loop.  See 
> proton-c/bindings/python/proton/reactor.py.  A pipe is used to wake up the 
> reactor when a new event is sent to the reactor (see reactor.py in the python 
> bindings).  The EventInjector's trigger method puts the event on a queue and 
> writes to a pipe to wake up the reactor.  The on_selectable_readable callback 
> in the EventInjector is called on the reactor thread to get the event off the 
> queue and clear the pipe.
> On windows it appears as if the EventInjector selectable is made "readable" 
> even though nothing has been written to the pipe.  This causes the os.read() 
> call in the on_selectable_readable() callback to hang.
> Best I can tell the windows selector code doesn't work properly with a pipe.  
> The pn_selector_next() function is returning a read event on the pipe's read 
> descriptor even though the pipe is empty.  But I'm not familiar with the 
> window's selector implementation, so this is a best guess.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)