Re: Beam Summit at ApacheCon
Hi, Kenn and I have submitted a talk on state and timers and dealing with timeseries. Cheers Reza *From: *Austin Bennett *Date: *Sun, 12 May 2019, 00:47 *To: *dev *Cc: * Hi All, > > Deadline for CfP is the morning of 13 May (this Monday) Pacific Time, as > decided by ApacheCon. Please submit if you have anything. Also, do write > if you have questions/concerns, etc. > > Cheers, > Austin > > > > On Tue, Apr 30, 2019 at 7:59 AM Austin Bennett < > whatwouldausti...@gmail.com> wrote: > >> Hi Users and Devs, >> >> The CfP deadline approaches. Do submit your technical and/or use case >> talks, etc etc. Feel free to reach out if you have any questions. >> >> Cheers, >> Austin >> >> On Tue, Apr 23, 2019 at 2:49 AM Maximilian Michels >> wrote: >> >>> Hi Austin, >>> >>> Thanks for the heads-up! I just want to highlight that this is a great >>> chance for Beam. There will be a _dedicated_ Beam track which means that >>> there is potential for lots of new people to learn about Beam. Of >>> course, there will also be many people already involved in Beam. >>> >>> -Max >>> >>> On 23.04.19 02:47, Austin Bennett wrote: >>> > Beam Summit will be at ApacheCon this year -- please consider >>> submitting! >>> > >>> > Dates for Beam Summit 11 and 12 September 2019. There are other >>> tracks >>> > at ApacheCon during this and on other dates too. >>> > >>> > https://www.apachecon.com/acna19/cfp.html >>> > >>> > >>> >>
Re: Beam Summit at ApacheCon
The paper submission deadline doesn't have a concrete time. Morning pacific time is all that is on their website, we're doing this collaboratively, thus we (Beam) doesn't have full control over everything. >From what I have seen elsewhere, it will be cut when the guy managing this comes into the office for the day not necessarily a specific time on the clock. On Sat, May 11, 2019 at 9:54 AM Suneel Marthi wrote: > Could u please further quantify the 'morning pacific time' part of it? > Its just not clear what the deadline is now from that. > > On Sat, May 11, 2019 at 12:47 PM Austin Bennett < > whatwouldausti...@gmail.com> wrote: > >> Hi All, >> >> Deadline for CfP is the morning of 13 May (this Monday) Pacific Time, as >> decided by ApacheCon. Please submit if you have anything. Also, do write >> if you have questions/concerns, etc. >> >> Cheers, >> Austin >> >> >> >> On Tue, Apr 30, 2019 at 7:59 AM Austin Bennett < >> whatwouldausti...@gmail.com> wrote: >> >>> Hi Users and Devs, >>> >>> The CfP deadline approaches. Do submit your technical and/or use case >>> talks, etc etc. Feel free to reach out if you have any questions. >>> >>> Cheers, >>> Austin >>> >>> On Tue, Apr 23, 2019 at 2:49 AM Maximilian Michels >>> wrote: >>> Hi Austin, Thanks for the heads-up! I just want to highlight that this is a great chance for Beam. There will be a _dedicated_ Beam track which means that there is potential for lots of new people to learn about Beam. Of course, there will also be many people already involved in Beam. -Max On 23.04.19 02:47, Austin Bennett wrote: > Beam Summit will be at ApacheCon this year -- please consider submitting! > > Dates for Beam Summit 11 and 12 September 2019. There are other tracks > at ApacheCon during this and on other dates too. > > https://www.apachecon.com/acna19/cfp.html > > >>>
Re: Beam Summit at ApacheCon
Could u please further quantify the 'morning pacific time' part of it? Its just not clear what the deadline is now from that. On Sat, May 11, 2019 at 12:47 PM Austin Bennett wrote: > Hi All, > > Deadline for CfP is the morning of 13 May (this Monday) Pacific Time, as > decided by ApacheCon. Please submit if you have anything. Also, do write > if you have questions/concerns, etc. > > Cheers, > Austin > > > > On Tue, Apr 30, 2019 at 7:59 AM Austin Bennett < > whatwouldausti...@gmail.com> wrote: > >> Hi Users and Devs, >> >> The CfP deadline approaches. Do submit your technical and/or use case >> talks, etc etc. Feel free to reach out if you have any questions. >> >> Cheers, >> Austin >> >> On Tue, Apr 23, 2019 at 2:49 AM Maximilian Michels >> wrote: >> >>> Hi Austin, >>> >>> Thanks for the heads-up! I just want to highlight that this is a great >>> chance for Beam. There will be a _dedicated_ Beam track which means that >>> there is potential for lots of new people to learn about Beam. Of >>> course, there will also be many people already involved in Beam. >>> >>> -Max >>> >>> On 23.04.19 02:47, Austin Bennett wrote: >>> > Beam Summit will be at ApacheCon this year -- please consider >>> submitting! >>> > >>> > Dates for Beam Summit 11 and 12 September 2019. There are other >>> tracks >>> > at ApacheCon during this and on other dates too. >>> > >>> > https://www.apachecon.com/acna19/cfp.html >>> > >>> > >>> >>
Re: Beam Summit at ApacheCon
Hi All, Deadline for CfP is the morning of 13 May (this Monday) Pacific Time, as decided by ApacheCon. Please submit if you have anything. Also, do write if you have questions/concerns, etc. Cheers, Austin On Tue, Apr 30, 2019 at 7:59 AM Austin Bennett wrote: > Hi Users and Devs, > > The CfP deadline approaches. Do submit your technical and/or use case > talks, etc etc. Feel free to reach out if you have any questions. > > Cheers, > Austin > > On Tue, Apr 23, 2019 at 2:49 AM Maximilian Michels wrote: > >> Hi Austin, >> >> Thanks for the heads-up! I just want to highlight that this is a great >> chance for Beam. There will be a _dedicated_ Beam track which means that >> there is potential for lots of new people to learn about Beam. Of >> course, there will also be many people already involved in Beam. >> >> -Max >> >> On 23.04.19 02:47, Austin Bennett wrote: >> > Beam Summit will be at ApacheCon this year -- please consider >> submitting! >> > >> > Dates for Beam Summit 11 and 12 September 2019. There are other tracks >> > at ApacheCon during this and on other dates too. >> > >> > https://www.apachecon.com/acna19/cfp.html >> > >> > >> >
Re: Wordcount using Python with Flink runner and Kafka source
Hello Maximilian, I followed your guide with the wordcount.py example, to the step where I connect to a remote Flink cluster from my laptop, and got stuck. On the Flink server side, I saw that there's a connection established from my laptop to the port 8081 of the server (shown in *netstat),* but nothing showed up in Flink GUI console. On my python console, I could see that my job state changed to *RUNNING*. On the other side, the JobService, the last log entry is "*Submitting job to...*" Are there any logs that could help me debug? Would that be any chance that the Flink job jar file is too big to be sent to my Flink cluster? More interesting, if I change the port in the command to start the JobService to an invalid one (e.g: from 8081 to 8082), the output on my python console as well as on the JobService console stayed the same. I also have another question regarding writing my stream into parquet files. As mentioned in this site, https://beam.apache.org/documentation/io/built-in/, there's no file-based connectors for Python streaming yet, does that mean I also need to use Java for the IO? Thanks and best regards, Averell Here's the output on my python console - it stuck there: (beam_env) Averell-Macbook:wordcount Averell$ python wordcount.py INFO:root:Using latest locally built Python SDK docker image. INFO:root: INFO:root: INFO:root:Job state changed to RUNNING And here is the output on the JobService: (beam_env) Averell-Macbook:beam Averell$ ./gradlew :beam-runners-flink-1.7-job-server:runShadow -PflinkMasterUrl= 10.10.64.121:8081 Configuration on demand is an incubating feature. > Task :beam-runners-flink-1.7-job-server:runShadow Listening for transport dt_socket at address: 5005 [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - ArtifactStagingService started on localhost:8098 [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - Java ExpansionService started on localhost:8097 [main] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobServerDriver - JobService started on localhost:8099 [grpc-default-executor-0] ERROR org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService - Encountered Unexpected Exception for Invocation job_28626ac7-5339-4694-84c5-7e85f3b51a0 org.apache.beam.vendor.grpc.v1p13p1.io.grpc.StatusException: NOT_FOUND at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Status.asException(Status.java:534) at org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.getInvocation(InMemoryJobService.java:341) at org.apache.beam.runners.fnexecution.jobsubmission.InMemoryJobService.getStateStream(InMemoryJobService.java:262) at org.apache.beam.model.jobmanagement.v1.JobServiceGrpc$MethodHandlers.invoke(JobServiceGrpc.java:770) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.stub.ServerCalls$UnaryServerCallHandler$UnaryServerCallListener.onHalfClose(ServerCalls.java:171) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.PartialForwardingServerCallListener.onHalfClose(PartialForwardingServerCallListener.java:35) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:23) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onHalfClose(ForwardingServerCallListener.java:40) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.Contexts$ContextualizedServerCallListener.onHalfClose(Contexts.java:86) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.halfClosed(ServerCallImpl.java:283) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1HalfClosed.runInContext(ServerImpl.java:707) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) at org.apache.beam.vendor.grpc.v1p13p1.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) [grpc-default-executor-0] INFO org.apache.beam.runners.flink.FlinkJobInvoker - Invoking job your-wordcount-job_de5850b5-e92f-4179-bcff-19169554aaef [grpc-default-executor-0] INFO org.apache.beam.runners.fnexecution.jobsubmission.JobInvocation - Starting job invocation your-wordcount-job_de5850b5-e92f-4179-bcff-19169554aaef [flink-runner-job-invoker] INFO org.apache.beam.runners.flink.FlinkPipelineRunner - Translating pipeline to Flink program. [flink-runner-job-invoker] INFO org.apache.beam.runners.flink.FlinkExecutionEnvironments - Creating a Batch Execution Environment. [flink-runner-job-invoker]
