Re: Branch 3.7.0 failing install related to Kryo version...perhaps
Thanks! On Wed, Jun 20, 2018 at 2:28 PM Pramod Immaneni wrote: > I believe it is StatsListenerWithContext > > On Wed, Jun 20, 2018 at 11:25 AM Aaron Bossert > wrote: > > > Sorry, a bit off topic, but what replaces > > com.datatorrent.api.StatsListener? It is marked as deprecated. > > > > On Wed, Jun 20, 2018 at 2:12 PM Aaron Bossert > > wrote: > > > > > I agree. I am porting a prototype that I built using Storm to Apex. > The > > > runtime config changes were the impetus for the switch. I need to be > > able > > > to adjust settings, compute resources, and topology components (add, > > > remove, modify) on the fly...Implementing those features from scratch > in > > > Storm would have been a massive undertaking. Currently, there are > Apache > > > Metron and Spot, which are backed by Hortonworks and Cloudera, > > > respectively...this will be a competitor to both of them. The project > > will > > > be open-sourced once sufficient progress is made. > > > > > > On Wed, Jun 20, 2018 at 2:02 PM Pramod Immaneni < > > pramod.imman...@gmail.com> > > > wrote: > > > > > >> That would be awesome, it would be a good use case for the platform. > > >> > > >> On Wed, Jun 20, 2018 at 10:54 AM Aaron Bossert > > >> wrote: > > >> > > >> > I'm with you. I'm hoping to have it underpin a fairly large network > > >> > security pipeline... > > >> > > > >> > On Wed, Jun 20, 2018 at 1:29 PM Vlad Rozov > wrote: > > >> > > > >> > > +1. Apex does not have as large community as other Apache project, > > but > > >> > > let's try to build it. > > >> > > > > >> > > Thank you, > > >> > > > > >> > > Vlad > > >> > > > > >> > > On 6/20/18 10:24, Pramod Immaneni wrote: > > >> > > > Aaron, > > >> > > > > > >> > > > Your concerns are legitimate, the number of folks contributing > in > > >> terms > > >> > > of > > >> > > > code commits has reduced quite a bit. I do however still see > > >> interest > > >> > > from > > >> > > > ex-contributors and committers based on their participation in > the > > >> > > > discussions on dev list when important topics come up. It is up > to > > >> be > > >> > > seen > > >> > > > how the recent changes with DataTorrent will play out and if > > >> > > ex-DataTorrent > > >> > > > folks will contribute in the upcoming months. There was a new > > >> > initiative > > >> > > > started by an ex-DataTorrent employee a few days back on adding > a > > >> basic > > >> > > UI > > >> > > > to Apex so maybe there is hope. > > >> > > > > > >> > > > Having said that a few of us long timers have stuck it out > through > > >> > thick > > >> > > > and thin and willing to help whenever it is possible. > > >> > > > > > >> > > > Thanks > > >> > > > > > >> > > > On Wed, Jun 20, 2018 at 9:19 AM Aaron Bossert < > > aa...@punchcyber.com > > >> > > > >> > > wrote: > > >> > > > > > >> > > >> Gentlemen, > > >> > > >> > > >> > > >> I am working at the start of a fairly large project. I have > some > > >> > > questions > > >> > > >> related to the general health of Apex...need to get a warm and > > >> fuzzy > > >> > > >> feeling that the project is not going to die on the vine as it > > >> were. > > >> > > >> > > >> > > >> I am seeing the volume of commits and contributor activity > > dropped > > >> off > > >> > > >> significantly since early 2016 and there has been a drop again > > >> after > > >> > > >> Datatorrent folded...What is your sense of the project? I > really > > >> like > > >> > > the > > >> > > >> framework and definitely would prefer to use it as well as > > >> contribute > > >> > > >> back...just want to make sure I am not going to have work on it > > >> solo > > >> > or > > >> > > >> worse, end up having to switch to something else later... > > >> > > >> > > >> > > >> Your thoughts? > > >> > > >> > > >> > > >> Aaron > > >> > > >> > > >> > > >> On Wed, Jun 20, 2018 at 12:10 PM Pramod Immaneni < > > >> > > >> pramod.imman...@gmail.com> > > >> > > >> wrote: > > >> > > >> > > >> > > >>> There are hadoop IPC calls are failing possibly because of its > > >> > reliance > > >> > > >> on > > >> > > >>> kryo for serializing the payload and there is some > > incompatibility > > >> > with > > >> > > >> the > > >> > > >>> new version. I will dig in more to see what is going on. > > >> > > >>> > > >> > > >>> On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert < > > >> aa...@punchcyber.com> > > >> > > >>> wrote: > > >> > > >>> > > >> > > Pramod, > > >> > > > > >> > > Thanks for taking the time to help! > > >> > > > > >> > > Here is the output (just failed parts) when running full > > install > > >> > > (clean > > >> > > install -X) on the Master branch: > > >> > > > > >> > > Running com.datatorrent.stram.StramRecoveryTest > > >> > > 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest > > >> > > testRpcFailover - Mock server listening at macbook-pro-6.lan/ > > >> > > 192.168.87.125:62154 > > >> > > 2018-06-19 21:34:28,678 [main] ERROR > stram.RecoverableRpcProxy >
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
I believe it is StatsListenerWithContext On Wed, Jun 20, 2018 at 11:25 AM Aaron Bossert wrote: > Sorry, a bit off topic, but what replaces > com.datatorrent.api.StatsListener? It is marked as deprecated. > > On Wed, Jun 20, 2018 at 2:12 PM Aaron Bossert > wrote: > > > I agree. I am porting a prototype that I built using Storm to Apex. The > > runtime config changes were the impetus for the switch. I need to be > able > > to adjust settings, compute resources, and topology components (add, > > remove, modify) on the fly...Implementing those features from scratch in > > Storm would have been a massive undertaking. Currently, there are Apache > > Metron and Spot, which are backed by Hortonworks and Cloudera, > > respectively...this will be a competitor to both of them. The project > will > > be open-sourced once sufficient progress is made. > > > > On Wed, Jun 20, 2018 at 2:02 PM Pramod Immaneni < > pramod.imman...@gmail.com> > > wrote: > > > >> That would be awesome, it would be a good use case for the platform. > >> > >> On Wed, Jun 20, 2018 at 10:54 AM Aaron Bossert > >> wrote: > >> > >> > I'm with you. I'm hoping to have it underpin a fairly large network > >> > security pipeline... > >> > > >> > On Wed, Jun 20, 2018 at 1:29 PM Vlad Rozov wrote: > >> > > >> > > +1. Apex does not have as large community as other Apache project, > but > >> > > let's try to build it. > >> > > > >> > > Thank you, > >> > > > >> > > Vlad > >> > > > >> > > On 6/20/18 10:24, Pramod Immaneni wrote: > >> > > > Aaron, > >> > > > > >> > > > Your concerns are legitimate, the number of folks contributing in > >> terms > >> > > of > >> > > > code commits has reduced quite a bit. I do however still see > >> interest > >> > > from > >> > > > ex-contributors and committers based on their participation in the > >> > > > discussions on dev list when important topics come up. It is up to > >> be > >> > > seen > >> > > > how the recent changes with DataTorrent will play out and if > >> > > ex-DataTorrent > >> > > > folks will contribute in the upcoming months. There was a new > >> > initiative > >> > > > started by an ex-DataTorrent employee a few days back on adding a > >> basic > >> > > UI > >> > > > to Apex so maybe there is hope. > >> > > > > >> > > > Having said that a few of us long timers have stuck it out through > >> > thick > >> > > > and thin and willing to help whenever it is possible. > >> > > > > >> > > > Thanks > >> > > > > >> > > > On Wed, Jun 20, 2018 at 9:19 AM Aaron Bossert < > aa...@punchcyber.com > >> > > >> > > wrote: > >> > > > > >> > > >> Gentlemen, > >> > > >> > >> > > >> I am working at the start of a fairly large project. I have some > >> > > questions > >> > > >> related to the general health of Apex...need to get a warm and > >> fuzzy > >> > > >> feeling that the project is not going to die on the vine as it > >> were. > >> > > >> > >> > > >> I am seeing the volume of commits and contributor activity > dropped > >> off > >> > > >> significantly since early 2016 and there has been a drop again > >> after > >> > > >> Datatorrent folded...What is your sense of the project? I really > >> like > >> > > the > >> > > >> framework and definitely would prefer to use it as well as > >> contribute > >> > > >> back...just want to make sure I am not going to have work on it > >> solo > >> > or > >> > > >> worse, end up having to switch to something else later... > >> > > >> > >> > > >> Your thoughts? > >> > > >> > >> > > >> Aaron > >> > > >> > >> > > >> On Wed, Jun 20, 2018 at 12:10 PM Pramod Immaneni < > >> > > >> pramod.imman...@gmail.com> > >> > > >> wrote: > >> > > >> > >> > > >>> There are hadoop IPC calls are failing possibly because of its > >> > reliance > >> > > >> on > >> > > >>> kryo for serializing the payload and there is some > incompatibility > >> > with > >> > > >> the > >> > > >>> new version. I will dig in more to see what is going on. > >> > > >>> > >> > > >>> On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert < > >> aa...@punchcyber.com> > >> > > >>> wrote: > >> > > >>> > >> > > Pramod, > >> > > > >> > > Thanks for taking the time to help! > >> > > > >> > > Here is the output (just failed parts) when running full > install > >> > > (clean > >> > > install -X) on the Master branch: > >> > > > >> > > Running com.datatorrent.stram.StramRecoveryTest > >> > > 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest > >> > > testRpcFailover - Mock server listening at macbook-pro-6.lan/ > >> > > 192.168.87.125:62154 > >> > > 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy > >> > invoke > >> > > - > >> > > Giving up RPC connection recovery after 507 ms > >> > > java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ > >> > > 192.168.87.125 > >> > > to macbook-pro-6.lan:62154 failed on socket timeout exception: > >> > > java.net.SocketTimeoutException: 500 millis timeout while > >> waiting > >> >
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
Sorry, a bit off topic, but what replaces com.datatorrent.api.StatsListener? It is marked as deprecated. On Wed, Jun 20, 2018 at 2:12 PM Aaron Bossert wrote: > I agree. I am porting a prototype that I built using Storm to Apex. The > runtime config changes were the impetus for the switch. I need to be able > to adjust settings, compute resources, and topology components (add, > remove, modify) on the fly...Implementing those features from scratch in > Storm would have been a massive undertaking. Currently, there are Apache > Metron and Spot, which are backed by Hortonworks and Cloudera, > respectively...this will be a competitor to both of them. The project will > be open-sourced once sufficient progress is made. > > On Wed, Jun 20, 2018 at 2:02 PM Pramod Immaneni > wrote: > >> That would be awesome, it would be a good use case for the platform. >> >> On Wed, Jun 20, 2018 at 10:54 AM Aaron Bossert >> wrote: >> >> > I'm with you. I'm hoping to have it underpin a fairly large network >> > security pipeline... >> > >> > On Wed, Jun 20, 2018 at 1:29 PM Vlad Rozov wrote: >> > >> > > +1. Apex does not have as large community as other Apache project, but >> > > let's try to build it. >> > > >> > > Thank you, >> > > >> > > Vlad >> > > >> > > On 6/20/18 10:24, Pramod Immaneni wrote: >> > > > Aaron, >> > > > >> > > > Your concerns are legitimate, the number of folks contributing in >> terms >> > > of >> > > > code commits has reduced quite a bit. I do however still see >> interest >> > > from >> > > > ex-contributors and committers based on their participation in the >> > > > discussions on dev list when important topics come up. It is up to >> be >> > > seen >> > > > how the recent changes with DataTorrent will play out and if >> > > ex-DataTorrent >> > > > folks will contribute in the upcoming months. There was a new >> > initiative >> > > > started by an ex-DataTorrent employee a few days back on adding a >> basic >> > > UI >> > > > to Apex so maybe there is hope. >> > > > >> > > > Having said that a few of us long timers have stuck it out through >> > thick >> > > > and thin and willing to help whenever it is possible. >> > > > >> > > > Thanks >> > > > >> > > > On Wed, Jun 20, 2018 at 9:19 AM Aaron Bossert > > >> > > wrote: >> > > > >> > > >> Gentlemen, >> > > >> >> > > >> I am working at the start of a fairly large project. I have some >> > > questions >> > > >> related to the general health of Apex...need to get a warm and >> fuzzy >> > > >> feeling that the project is not going to die on the vine as it >> were. >> > > >> >> > > >> I am seeing the volume of commits and contributor activity dropped >> off >> > > >> significantly since early 2016 and there has been a drop again >> after >> > > >> Datatorrent folded...What is your sense of the project? I really >> like >> > > the >> > > >> framework and definitely would prefer to use it as well as >> contribute >> > > >> back...just want to make sure I am not going to have work on it >> solo >> > or >> > > >> worse, end up having to switch to something else later... >> > > >> >> > > >> Your thoughts? >> > > >> >> > > >> Aaron >> > > >> >> > > >> On Wed, Jun 20, 2018 at 12:10 PM Pramod Immaneni < >> > > >> pramod.imman...@gmail.com> >> > > >> wrote: >> > > >> >> > > >>> There are hadoop IPC calls are failing possibly because of its >> > reliance >> > > >> on >> > > >>> kryo for serializing the payload and there is some incompatibility >> > with >> > > >> the >> > > >>> new version. I will dig in more to see what is going on. >> > > >>> >> > > >>> On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert < >> aa...@punchcyber.com> >> > > >>> wrote: >> > > >>> >> > > Pramod, >> > > >> > > Thanks for taking the time to help! >> > > >> > > Here is the output (just failed parts) when running full install >> > > (clean >> > > install -X) on the Master branch: >> > > >> > > Running com.datatorrent.stram.StramRecoveryTest >> > > 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest >> > > testRpcFailover - Mock server listening at macbook-pro-6.lan/ >> > > 192.168.87.125:62154 >> > > 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy >> > invoke >> > > - >> > > Giving up RPC connection recovery after 507 ms >> > > java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ >> > > 192.168.87.125 >> > > to macbook-pro-6.lan:62154 failed on socket timeout exception: >> > > java.net.SocketTimeoutException: 500 millis timeout while >> waiting >> > for >> > > channel to be ready for read. ch : >> > > java.nio.channels.SocketChannel[connected local=/ >> > 192.168.87.125:62155 >> > > remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details >> > see: >> > > http://wiki.apache.org/hadoop/SocketTimeout >> > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native >> > > >> Method) >> > > at >> > > >> > > >> > > >> >> >
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
I agree. I am porting a prototype that I built using Storm to Apex. The runtime config changes were the impetus for the switch. I need to be able to adjust settings, compute resources, and topology components (add, remove, modify) on the fly...Implementing those features from scratch in Storm would have been a massive undertaking. Currently, there are Apache Metron and Spot, which are backed by Hortonworks and Cloudera, respectively...this will be a competitor to both of them. The project will be open-sourced once sufficient progress is made. On Wed, Jun 20, 2018 at 2:02 PM Pramod Immaneni wrote: > That would be awesome, it would be a good use case for the platform. > > On Wed, Jun 20, 2018 at 10:54 AM Aaron Bossert > wrote: > > > I'm with you. I'm hoping to have it underpin a fairly large network > > security pipeline... > > > > On Wed, Jun 20, 2018 at 1:29 PM Vlad Rozov wrote: > > > > > +1. Apex does not have as large community as other Apache project, but > > > let's try to build it. > > > > > > Thank you, > > > > > > Vlad > > > > > > On 6/20/18 10:24, Pramod Immaneni wrote: > > > > Aaron, > > > > > > > > Your concerns are legitimate, the number of folks contributing in > terms > > > of > > > > code commits has reduced quite a bit. I do however still see interest > > > from > > > > ex-contributors and committers based on their participation in the > > > > discussions on dev list when important topics come up. It is up to be > > > seen > > > > how the recent changes with DataTorrent will play out and if > > > ex-DataTorrent > > > > folks will contribute in the upcoming months. There was a new > > initiative > > > > started by an ex-DataTorrent employee a few days back on adding a > basic > > > UI > > > > to Apex so maybe there is hope. > > > > > > > > Having said that a few of us long timers have stuck it out through > > thick > > > > and thin and willing to help whenever it is possible. > > > > > > > > Thanks > > > > > > > > On Wed, Jun 20, 2018 at 9:19 AM Aaron Bossert > > > wrote: > > > > > > > >> Gentlemen, > > > >> > > > >> I am working at the start of a fairly large project. I have some > > > questions > > > >> related to the general health of Apex...need to get a warm and fuzzy > > > >> feeling that the project is not going to die on the vine as it were. > > > >> > > > >> I am seeing the volume of commits and contributor activity dropped > off > > > >> significantly since early 2016 and there has been a drop again after > > > >> Datatorrent folded...What is your sense of the project? I really > like > > > the > > > >> framework and definitely would prefer to use it as well as > contribute > > > >> back...just want to make sure I am not going to have work on it solo > > or > > > >> worse, end up having to switch to something else later... > > > >> > > > >> Your thoughts? > > > >> > > > >> Aaron > > > >> > > > >> On Wed, Jun 20, 2018 at 12:10 PM Pramod Immaneni < > > > >> pramod.imman...@gmail.com> > > > >> wrote: > > > >> > > > >>> There are hadoop IPC calls are failing possibly because of its > > reliance > > > >> on > > > >>> kryo for serializing the payload and there is some incompatibility > > with > > > >> the > > > >>> new version. I will dig in more to see what is going on. > > > >>> > > > >>> On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert < > aa...@punchcyber.com> > > > >>> wrote: > > > >>> > > > Pramod, > > > > > > Thanks for taking the time to help! > > > > > > Here is the output (just failed parts) when running full install > > > (clean > > > install -X) on the Master branch: > > > > > > Running com.datatorrent.stram.StramRecoveryTest > > > 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest > > > testRpcFailover - Mock server listening at macbook-pro-6.lan/ > > > 192.168.87.125:62154 > > > 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy > > invoke > > > - > > > Giving up RPC connection recovery after 507 ms > > > java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ > > > 192.168.87.125 > > > to macbook-pro-6.lan:62154 failed on socket timeout exception: > > > java.net.SocketTimeoutException: 500 millis timeout while waiting > > for > > > channel to be ready for read. ch : > > > java.nio.channels.SocketChannel[connected local=/ > > 192.168.87.125:62155 > > > remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details > > see: > > > http://wiki.apache.org/hadoop/SocketTimeout > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > > >> Method) > > > at > > > > > > > > > >> > > > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > > at > > > > > > > > > >> > > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > > at
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
That would be awesome, it would be a good use case for the platform. On Wed, Jun 20, 2018 at 10:54 AM Aaron Bossert wrote: > I'm with you. I'm hoping to have it underpin a fairly large network > security pipeline... > > On Wed, Jun 20, 2018 at 1:29 PM Vlad Rozov wrote: > > > +1. Apex does not have as large community as other Apache project, but > > let's try to build it. > > > > Thank you, > > > > Vlad > > > > On 6/20/18 10:24, Pramod Immaneni wrote: > > > Aaron, > > > > > > Your concerns are legitimate, the number of folks contributing in terms > > of > > > code commits has reduced quite a bit. I do however still see interest > > from > > > ex-contributors and committers based on their participation in the > > > discussions on dev list when important topics come up. It is up to be > > seen > > > how the recent changes with DataTorrent will play out and if > > ex-DataTorrent > > > folks will contribute in the upcoming months. There was a new > initiative > > > started by an ex-DataTorrent employee a few days back on adding a basic > > UI > > > to Apex so maybe there is hope. > > > > > > Having said that a few of us long timers have stuck it out through > thick > > > and thin and willing to help whenever it is possible. > > > > > > Thanks > > > > > > On Wed, Jun 20, 2018 at 9:19 AM Aaron Bossert > > wrote: > > > > > >> Gentlemen, > > >> > > >> I am working at the start of a fairly large project. I have some > > questions > > >> related to the general health of Apex...need to get a warm and fuzzy > > >> feeling that the project is not going to die on the vine as it were. > > >> > > >> I am seeing the volume of commits and contributor activity dropped off > > >> significantly since early 2016 and there has been a drop again after > > >> Datatorrent folded...What is your sense of the project? I really like > > the > > >> framework and definitely would prefer to use it as well as contribute > > >> back...just want to make sure I am not going to have work on it solo > or > > >> worse, end up having to switch to something else later... > > >> > > >> Your thoughts? > > >> > > >> Aaron > > >> > > >> On Wed, Jun 20, 2018 at 12:10 PM Pramod Immaneni < > > >> pramod.imman...@gmail.com> > > >> wrote: > > >> > > >>> There are hadoop IPC calls are failing possibly because of its > reliance > > >> on > > >>> kryo for serializing the payload and there is some incompatibility > with > > >> the > > >>> new version. I will dig in more to see what is going on. > > >>> > > >>> On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert > > >>> wrote: > > >>> > > Pramod, > > > > Thanks for taking the time to help! > > > > Here is the output (just failed parts) when running full install > > (clean > > install -X) on the Master branch: > > > > Running com.datatorrent.stram.StramRecoveryTest > > 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest > > testRpcFailover - Mock server listening at macbook-pro-6.lan/ > > 192.168.87.125:62154 > > 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy > invoke > > - > > Giving up RPC connection recovery after 507 ms > > java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ > > 192.168.87.125 > > to macbook-pro-6.lan:62154 failed on socket timeout exception: > > java.net.SocketTimeoutException: 500 millis timeout while waiting > for > > channel to be ready for read. ch : > > java.nio.channels.SocketChannel[connected local=/ > 192.168.87.125:62155 > > remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details > see: > > http://wiki.apache.org/hadoop/SocketTimeout > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > > >> Method) > > at > > > > > > >> > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > > > > > > >> > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at org.apache.hadoop.net > .NetUtils.wrapWithMessage(NetUtils.java:791) > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) > > at org.apache.hadoop.ipc.Client.call(Client.java:1472) > > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > > at > > > > > > >> > > > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) > > at com.sun.proxy.$Proxy138.log(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > > >> > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > > > > >> > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > >
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
I'm with you. I'm hoping to have it underpin a fairly large network security pipeline... On Wed, Jun 20, 2018 at 1:29 PM Vlad Rozov wrote: > +1. Apex does not have as large community as other Apache project, but > let's try to build it. > > Thank you, > > Vlad > > On 6/20/18 10:24, Pramod Immaneni wrote: > > Aaron, > > > > Your concerns are legitimate, the number of folks contributing in terms > of > > code commits has reduced quite a bit. I do however still see interest > from > > ex-contributors and committers based on their participation in the > > discussions on dev list when important topics come up. It is up to be > seen > > how the recent changes with DataTorrent will play out and if > ex-DataTorrent > > folks will contribute in the upcoming months. There was a new initiative > > started by an ex-DataTorrent employee a few days back on adding a basic > UI > > to Apex so maybe there is hope. > > > > Having said that a few of us long timers have stuck it out through thick > > and thin and willing to help whenever it is possible. > > > > Thanks > > > > On Wed, Jun 20, 2018 at 9:19 AM Aaron Bossert > wrote: > > > >> Gentlemen, > >> > >> I am working at the start of a fairly large project. I have some > questions > >> related to the general health of Apex...need to get a warm and fuzzy > >> feeling that the project is not going to die on the vine as it were. > >> > >> I am seeing the volume of commits and contributor activity dropped off > >> significantly since early 2016 and there has been a drop again after > >> Datatorrent folded...What is your sense of the project? I really like > the > >> framework and definitely would prefer to use it as well as contribute > >> back...just want to make sure I am not going to have work on it solo or > >> worse, end up having to switch to something else later... > >> > >> Your thoughts? > >> > >> Aaron > >> > >> On Wed, Jun 20, 2018 at 12:10 PM Pramod Immaneni < > >> pramod.imman...@gmail.com> > >> wrote: > >> > >>> There are hadoop IPC calls are failing possibly because of its reliance > >> on > >>> kryo for serializing the payload and there is some incompatibility with > >> the > >>> new version. I will dig in more to see what is going on. > >>> > >>> On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert > >>> wrote: > >>> > Pramod, > > Thanks for taking the time to help! > > Here is the output (just failed parts) when running full install > (clean > install -X) on the Master branch: > > Running com.datatorrent.stram.StramRecoveryTest > 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest > testRpcFailover - Mock server listening at macbook-pro-6.lan/ > 192.168.87.125:62154 > 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy invoke > - > Giving up RPC connection recovery after 507 ms > java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ > 192.168.87.125 > to macbook-pro-6.lan:62154 failed on socket timeout exception: > java.net.SocketTimeoutException: 500 millis timeout while waiting for > channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=/192.168.87.125:62155 > remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details see: > http://wiki.apache.org/hadoop/SocketTimeout > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > >> Method) > at > > > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > > > >> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) > at org.apache.hadoop.ipc.Client.call(Client.java:1472) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > > > >> > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) > at com.sun.proxy.$Proxy138.log(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > > > >> > com.datatorrent.stram.RecoverableRpcProxy.invoke(RecoverableRpcProxy.java:157) > at com.sun.proxy.$Proxy138.log(Unknown Source) > at > > > >> > com.datatorrent.stram.StramRecoveryTest.testRpcFailover(StramRecoveryTest.java:561) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > > >> >
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
+1. Apex does not have as large community as other Apache project, but let's try to build it. Thank you, Vlad On 6/20/18 10:24, Pramod Immaneni wrote: Aaron, Your concerns are legitimate, the number of folks contributing in terms of code commits has reduced quite a bit. I do however still see interest from ex-contributors and committers based on their participation in the discussions on dev list when important topics come up. It is up to be seen how the recent changes with DataTorrent will play out and if ex-DataTorrent folks will contribute in the upcoming months. There was a new initiative started by an ex-DataTorrent employee a few days back on adding a basic UI to Apex so maybe there is hope. Having said that a few of us long timers have stuck it out through thick and thin and willing to help whenever it is possible. Thanks On Wed, Jun 20, 2018 at 9:19 AM Aaron Bossert wrote: Gentlemen, I am working at the start of a fairly large project. I have some questions related to the general health of Apex...need to get a warm and fuzzy feeling that the project is not going to die on the vine as it were. I am seeing the volume of commits and contributor activity dropped off significantly since early 2016 and there has been a drop again after Datatorrent folded...What is your sense of the project? I really like the framework and definitely would prefer to use it as well as contribute back...just want to make sure I am not going to have work on it solo or worse, end up having to switch to something else later... Your thoughts? Aaron On Wed, Jun 20, 2018 at 12:10 PM Pramod Immaneni < pramod.imman...@gmail.com> wrote: There are hadoop IPC calls are failing possibly because of its reliance on kryo for serializing the payload and there is some incompatibility with the new version. I will dig in more to see what is going on. On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert wrote: Pramod, Thanks for taking the time to help! Here is the output (just failed parts) when running full install (clean install -X) on the Master branch: Running com.datatorrent.stram.StramRecoveryTest 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest testRpcFailover - Mock server listening at macbook-pro-6.lan/ 192.168.87.125:62154 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy invoke - Giving up RPC connection recovery after 507 ms java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ 192.168.87.125 to macbook-pro-6.lan:62154 failed on socket timeout exception: java.net.SocketTimeoutException: 500 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.87.125:62155 remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) at com.sun.proxy.$Proxy138.log(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.datatorrent.stram.RecoverableRpcProxy.invoke(RecoverableRpcProxy.java:157) at com.sun.proxy.$Proxy138.log(Unknown Source) at com.datatorrent.stram.StramRecoveryTest.testRpcFailover(StramRecoveryTest.java:561) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
Aaron, Your concerns are legitimate, the number of folks contributing in terms of code commits has reduced quite a bit. I do however still see interest from ex-contributors and committers based on their participation in the discussions on dev list when important topics come up. It is up to be seen how the recent changes with DataTorrent will play out and if ex-DataTorrent folks will contribute in the upcoming months. There was a new initiative started by an ex-DataTorrent employee a few days back on adding a basic UI to Apex so maybe there is hope. Having said that a few of us long timers have stuck it out through thick and thin and willing to help whenever it is possible. Thanks On Wed, Jun 20, 2018 at 9:19 AM Aaron Bossert wrote: > Gentlemen, > > I am working at the start of a fairly large project. I have some questions > related to the general health of Apex...need to get a warm and fuzzy > feeling that the project is not going to die on the vine as it were. > > I am seeing the volume of commits and contributor activity dropped off > significantly since early 2016 and there has been a drop again after > Datatorrent folded...What is your sense of the project? I really like the > framework and definitely would prefer to use it as well as contribute > back...just want to make sure I am not going to have work on it solo or > worse, end up having to switch to something else later... > > Your thoughts? > > Aaron > > On Wed, Jun 20, 2018 at 12:10 PM Pramod Immaneni < > pramod.imman...@gmail.com> > wrote: > > > There are hadoop IPC calls are failing possibly because of its reliance > on > > kryo for serializing the payload and there is some incompatibility with > the > > new version. I will dig in more to see what is going on. > > > > On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert > > wrote: > > > > > Pramod, > > > > > > Thanks for taking the time to help! > > > > > > Here is the output (just failed parts) when running full install (clean > > > install -X) on the Master branch: > > > > > > Running com.datatorrent.stram.StramRecoveryTest > > > 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest > > > testRpcFailover - Mock server listening at macbook-pro-6.lan/ > > > 192.168.87.125:62154 > > > 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy invoke - > > > Giving up RPC connection recovery after 507 ms > > > java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ > > > 192.168.87.125 > > > to macbook-pro-6.lan:62154 failed on socket timeout exception: > > > java.net.SocketTimeoutException: 500 millis timeout while waiting for > > > channel to be ready for read. ch : > > > java.nio.channels.SocketChannel[connected local=/192.168.87.125:62155 > > > remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details see: > > > http://wiki.apache.org/hadoop/SocketTimeout > > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native > Method) > > > at > > > > > > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > > at > > > > > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) > > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) > > > at org.apache.hadoop.ipc.Client.call(Client.java:1472) > > > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > > > at > > > > > > > > > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) > > > at com.sun.proxy.$Proxy138.log(Unknown Source) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > at > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > at > > > > > > > > > com.datatorrent.stram.RecoverableRpcProxy.invoke(RecoverableRpcProxy.java:157) > > > at com.sun.proxy.$Proxy138.log(Unknown Source) > > > at > > > > > > > > > com.datatorrent.stram.StramRecoveryTest.testRpcFailover(StramRecoveryTest.java:561) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > > > > > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > at > > > > > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:498) > > > at > > > > > > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > > > at > > > > > > > > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > > > at > > > > > > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > > > at > > > > > > > >
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
I see flush was getting called twice.. On Wed, Jun 20, 2018 at 9:30 AM Vlad Rozov wrote: > AFAIK, those RPC calls are expected to fail (test for RPC timeout). The > test fails due to the change in Kryo flush behavior, please see my other > response. > > Thank you, > > Vlad > > On 6/20/18 09:09, Pramod Immaneni wrote: > > There are hadoop IPC calls are failing possibly because of its reliance > on > > kryo for serializing the payload and there is some incompatibility with > the > > new version. I will dig in more to see what is going on. > > > > On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert > wrote: > > > >> Pramod, > >> > >> Thanks for taking the time to help! > >> > >> Here is the output (just failed parts) when running full install (clean > >> install -X) on the Master branch: > >> > >> Running com.datatorrent.stram.StramRecoveryTest > >> 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest > >> testRpcFailover - Mock server listening at macbook-pro-6.lan/ > >> 192.168.87.125:62154 > >> 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy invoke - > >> Giving up RPC connection recovery after 507 ms > >> java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ > >> 192.168.87.125 > >> to macbook-pro-6.lan:62154 failed on socket timeout exception: > >> java.net.SocketTimeoutException: 500 millis timeout while waiting for > >> channel to be ready for read. ch : > >> java.nio.channels.SocketChannel[connected local=/192.168.87.125:62155 > >> remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details see: > >> http://wiki.apache.org/hadoop/SocketTimeout > >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > >> at > >> > >> > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > >> at > >> > >> > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > >> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > >> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) > >> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) > >> at org.apache.hadoop.ipc.Client.call(Client.java:1472) > >> at org.apache.hadoop.ipc.Client.call(Client.java:1399) > >> at > >> > >> > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) > >> at com.sun.proxy.$Proxy138.log(Unknown Source) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> at > >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at > >> > >> > com.datatorrent.stram.RecoverableRpcProxy.invoke(RecoverableRpcProxy.java:157) > >> at com.sun.proxy.$Proxy138.log(Unknown Source) > >> at > >> > >> > com.datatorrent.stram.StramRecoveryTest.testRpcFailover(StramRecoveryTest.java:561) > >> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > >> at > >> > >> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > >> at > >> > >> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > >> at java.lang.reflect.Method.invoke(Method.java:498) > >> at > >> > >> > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > >> at > >> > >> > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > >> at > >> > >> > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > >> at > >> > >> > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > >> at > >> > >> > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > >> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > >> at org.junit.rules.RunRules.evaluate(RunRules.java:20) > >> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > >> at > >> > >> > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > >> at > >> > >> > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > >> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > >> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > >> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > >> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > >> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > >> at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > >> at org.junit.runners.Suite.runChild(Suite.java:127) > >> at org.junit.runners.Suite.runChild(Suite.java:26) > >> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > >> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > >> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > >>
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
AFAIK, those RPC calls are expected to fail (test for RPC timeout). The test fails due to the change in Kryo flush behavior, please see my other response. Thank you, Vlad On 6/20/18 09:09, Pramod Immaneni wrote: There are hadoop IPC calls are failing possibly because of its reliance on kryo for serializing the payload and there is some incompatibility with the new version. I will dig in more to see what is going on. On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert wrote: Pramod, Thanks for taking the time to help! Here is the output (just failed parts) when running full install (clean install -X) on the Master branch: Running com.datatorrent.stram.StramRecoveryTest 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest testRpcFailover - Mock server listening at macbook-pro-6.lan/ 192.168.87.125:62154 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy invoke - Giving up RPC connection recovery after 507 ms java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ 192.168.87.125 to macbook-pro-6.lan:62154 failed on socket timeout exception: java.net.SocketTimeoutException: 500 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/192.168.87.125:62155 remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details see: http://wiki.apache.org/hadoop/SocketTimeout at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) at org.apache.hadoop.ipc.Client.call(Client.java:1472) at org.apache.hadoop.ipc.Client.call(Client.java:1399) at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) at com.sun.proxy.$Proxy138.log(Unknown Source) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at com.datatorrent.stram.RecoverableRpcProxy.invoke(RecoverableRpcProxy.java:157) at com.sun.proxy.$Proxy138.log(Unknown Source) at com.datatorrent.stram.StramRecoveryTest.testRpcFailover(StramRecoveryTest.java:561) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.junit.runners.Suite.runChild(Suite.java:127) at org.junit.runners.Suite.runChild(Suite.java:26) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) at org.junit.runners.ParentRunner.run(ParentRunner.java:309) at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) at
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
Gentlemen, I am working at the start of a fairly large project. I have some questions related to the general health of Apex...need to get a warm and fuzzy feeling that the project is not going to die on the vine as it were. I am seeing the volume of commits and contributor activity dropped off significantly since early 2016 and there has been a drop again after Datatorrent folded...What is your sense of the project? I really like the framework and definitely would prefer to use it as well as contribute back...just want to make sure I am not going to have work on it solo or worse, end up having to switch to something else later... Your thoughts? Aaron On Wed, Jun 20, 2018 at 12:10 PM Pramod Immaneni wrote: > There are hadoop IPC calls are failing possibly because of its reliance on > kryo for serializing the payload and there is some incompatibility with the > new version. I will dig in more to see what is going on. > > On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert > wrote: > > > Pramod, > > > > Thanks for taking the time to help! > > > > Here is the output (just failed parts) when running full install (clean > > install -X) on the Master branch: > > > > Running com.datatorrent.stram.StramRecoveryTest > > 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest > > testRpcFailover - Mock server listening at macbook-pro-6.lan/ > > 192.168.87.125:62154 > > 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy invoke - > > Giving up RPC connection recovery after 507 ms > > java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ > > 192.168.87.125 > > to macbook-pro-6.lan:62154 failed on socket timeout exception: > > java.net.SocketTimeoutException: 500 millis timeout while waiting for > > channel to be ready for read. ch : > > java.nio.channels.SocketChannel[connected local=/192.168.87.125:62155 > > remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details see: > > http://wiki.apache.org/hadoop/SocketTimeout > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > > > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > > > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) > > at org.apache.hadoop.ipc.Client.call(Client.java:1472) > > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > > at > > > > > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) > > at com.sun.proxy.$Proxy138.log(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > > > com.datatorrent.stram.RecoverableRpcProxy.invoke(RecoverableRpcProxy.java:157) > > at com.sun.proxy.$Proxy138.log(Unknown Source) > > at > > > > > com.datatorrent.stram.StramRecoveryTest.testRpcFailover(StramRecoveryTest.java:561) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > > > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > > at > > > > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > > at > > > > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > > at > > > > > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > > at > > > > > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > > at > > > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > > at > > > > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > > at
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
There are hadoop IPC calls are failing possibly because of its reliance on kryo for serializing the payload and there is some incompatibility with the new version. I will dig in more to see what is going on. On Tue, Jun 19, 2018 at 6:54 PM Aaron Bossert wrote: > Pramod, > > Thanks for taking the time to help! > > Here is the output (just failed parts) when running full install (clean > install -X) on the Master branch: > > Running com.datatorrent.stram.StramRecoveryTest > 2018-06-19 21:34:28,137 [main] INFO stram.StramRecoveryTest > testRpcFailover - Mock server listening at macbook-pro-6.lan/ > 192.168.87.125:62154 > 2018-06-19 21:34:28,678 [main] ERROR stram.RecoverableRpcProxy invoke - > Giving up RPC connection recovery after 507 ms > java.net.SocketTimeoutException: Call From macbook-pro-6.lan/ > 192.168.87.125 > to macbook-pro-6.lan:62154 failed on socket timeout exception: > java.net.SocketTimeoutException: 500 millis timeout while waiting for > channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=/192.168.87.125:62155 > remote=macbook-pro-6.lan/192.168.87.125:62154]; For more details see: > http://wiki.apache.org/hadoop/SocketTimeout > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) > at org.apache.hadoop.ipc.Client.call(Client.java:1472) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) > at com.sun.proxy.$Proxy138.log(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > > com.datatorrent.stram.RecoverableRpcProxy.invoke(RecoverableRpcProxy.java:157) > at com.sun.proxy.$Proxy138.log(Unknown Source) > at > > com.datatorrent.stram.StramRecoveryTest.testRpcFailover(StramRecoveryTest.java:561) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runners.Suite.runChild(Suite.java:127) > at org.junit.runners.Suite.runChild(Suite.java:26) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55) > at > > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137) > at > > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:107) > at > > org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:83) > at > >
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
Do you see the same errors when you run the individual tests in question in isolation, such as using mvn test -Dtest=. If you do can you paste the full logs of what you see when the individual tests fail. Thanks On Mon, Jun 18, 2018 at 11:41 AM Aaron Bossert wrote: > please disregard the first iteration...this ended up being related to a > hung build running in the background causing timeouts, I think. I am still > having failures, but there are two and are still mysterious to me as to > their root cause. Here are the actual failures: > > I don't immediately see how these are related to Kryo at all...but then > again, I am still familiarizing myself with the code base. I am hoping > that someone out there has a lightbulb turn on and has some notion of how > they are related... > > > --- > Test set: com.datatorrent.stram.StramRecoveryTest > > --- > Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.119 sec > <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest > testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed: > 0.105 sec <<< FAILURE! > java.lang.AssertionError: flush count expected:<1> but was:<2> > at > > com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326) > > > --- > Test set: com.datatorrent.stram.engine.StatsTest > > --- > Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.051 sec > <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest > > testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) > Time elapsed: 3.277 sec <<< FAILURE! > java.lang.AssertionError: Validate input port queue size -1 > at > > com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270) > at > > com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285) > > On Mon, Jun 18, 2018 at 1:20 PM Aaron Bossert > wrote: > > > I recently attempted to update Kryo from 2.24.0 to 4.0.2 to address a > > serialization issue related to support for Java Instant and a couple of > > other classes that are supported in newer Kryo versions. My test build > and > > install (vanilla, no changes of any kind, just download apex-core and > > "clean install") works fine, however, when updating the Kryo dependency > to > > 4.0.2, getting this non-obvious (to me) error (running "clean install > -X). > > I also identified a bug or perhaps a feature? When building on my macOS > > laptop, I have an Idea project folder in iCloud which is locally stored > in > > a directory that contains a space in the name, which needs to be escaped. > > When I initially built, I kept running into errors related to that...not > > sure if that is something that should be fixed (it is not as > > straightforward as I had hoped) or simply require that directory names > not > > include any spaces. I have no control of the iCloud local folder > > name...otherwise, would have just fixed that. > > > > 2018-06-18 12:43:24,485 [main] ERROR stram.RecoverableRpcProxy invoke - > > Giving up RPC connection recovery after 504 ms > > java.net.SocketTimeoutException: Call From MacBook-Pro-6.local/ > 10.37.129.2 > > to MacBook-Pro-6.local:65136 failed on socket timeout exception: > > java.net.SocketTimeoutException: 500 millis timeout while waiting for > > channel to be ready for read. ch : > > java.nio.channels.SocketChannel[connected local=/10.37.129.2:65137 > > remote=MacBook-Pro-6.local/10.37.129.2:65136]; For more details see: > > http://wiki.apache.org/hadoop/SocketTimeout > > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > > at > > > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > > at > > > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) > > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) > > at org.apache.hadoop.ipc.Client.call(Client.java:1472) > > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > > at > > > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) > > at com.sun.proxy.$Proxy138.log(Unknown Source) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > at > > > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > at > > > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > > at > > >
Re: Branch 3.7.0 failing install related to Kryo version...perhaps
please disregard the first iteration...this ended up being related to a hung build running in the background causing timeouts, I think. I am still having failures, but there are two and are still mysterious to me as to their root cause. Here are the actual failures: I don't immediately see how these are related to Kryo at all...but then again, I am still familiarizing myself with the code base. I am hoping that someone out there has a lightbulb turn on and has some notion of how they are related... --- Test set: com.datatorrent.stram.StramRecoveryTest --- Tests run: 8, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 6.119 sec <<< FAILURE! - in com.datatorrent.stram.StramRecoveryTest testWriteAheadLog(com.datatorrent.stram.StramRecoveryTest) Time elapsed: 0.105 sec <<< FAILURE! java.lang.AssertionError: flush count expected:<1> but was:<2> at com.datatorrent.stram.StramRecoveryTest.testWriteAheadLog(StramRecoveryTest.java:326) --- Test set: com.datatorrent.stram.engine.StatsTest --- Tests run: 6, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 22.051 sec <<< FAILURE! - in com.datatorrent.stram.engine.StatsTest testQueueSizeForContainerLocalOperators(com.datatorrent.stram.engine.StatsTest) Time elapsed: 3.277 sec <<< FAILURE! java.lang.AssertionError: Validate input port queue size -1 at com.datatorrent.stram.engine.StatsTest.baseTestForQueueSize(StatsTest.java:270) at com.datatorrent.stram.engine.StatsTest.testQueueSizeForContainerLocalOperators(StatsTest.java:285) On Mon, Jun 18, 2018 at 1:20 PM Aaron Bossert wrote: > I recently attempted to update Kryo from 2.24.0 to 4.0.2 to address a > serialization issue related to support for Java Instant and a couple of > other classes that are supported in newer Kryo versions. My test build and > install (vanilla, no changes of any kind, just download apex-core and > "clean install") works fine, however, when updating the Kryo dependency to > 4.0.2, getting this non-obvious (to me) error (running "clean install -X). > I also identified a bug or perhaps a feature? When building on my macOS > laptop, I have an Idea project folder in iCloud which is locally stored in > a directory that contains a space in the name, which needs to be escaped. > When I initially built, I kept running into errors related to that...not > sure if that is something that should be fixed (it is not as > straightforward as I had hoped) or simply require that directory names not > include any spaces. I have no control of the iCloud local folder > name...otherwise, would have just fixed that. > > 2018-06-18 12:43:24,485 [main] ERROR stram.RecoverableRpcProxy invoke - > Giving up RPC connection recovery after 504 ms > java.net.SocketTimeoutException: Call From MacBook-Pro-6.local/10.37.129.2 > to MacBook-Pro-6.local:65136 failed on socket timeout exception: > java.net.SocketTimeoutException: 500 millis timeout while waiting for > channel to be ready for read. ch : > java.nio.channels.SocketChannel[connected local=/10.37.129.2:65137 > remote=MacBook-Pro-6.local/10.37.129.2:65136]; For more details see: > http://wiki.apache.org/hadoop/SocketTimeout > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) > at > sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) > at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > at java.lang.reflect.Constructor.newInstance(Constructor.java:423) > at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791) > at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:750) > at org.apache.hadoop.ipc.Client.call(Client.java:1472) > at org.apache.hadoop.ipc.Client.call(Client.java:1399) > at > org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:244) > at com.sun.proxy.$Proxy138.log(Unknown Source) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > com.datatorrent.stram.RecoverableRpcProxy.invoke(RecoverableRpcProxy.java:157) > at com.sun.proxy.$Proxy138.log(Unknown Source) > at > com.datatorrent.stram.StramRecoveryTest.testRpcFailover(StramRecoveryTest.java:561) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at
