date:20180302

[jira] [Created] (FLINK-8830) YarnResourceManager throws NullPointerException

2018-03-02 Thread Piotr Nowojski (JIRA)

Piotr Nowojski created FLINK-8830:
-

 Summary: YarnResourceManager throws NullPointerException
 Key: FLINK-8830
 URL: https://issues.apache.org/jira/browse/FLINK-8830
 Project: Flink
  Issue Type: Bug
  Components: YARN
Affects Versions: 1.5.0
Reporter: Piotr Nowojski


 
{code:java}
java.lang.NullPointerException
at java.io.File.(File.java:277)
at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:502)
at 
org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:445)
at 
org.apache.flink.yarn.YarnResourceManager.onContainersAllocated(YarnResourceManager.java:338)
at 
org.apache.flink.yarn.YarnResourceManagerTest$1.(YarnResourceManagerTest.java:340)
at 
org.apache.flink.yarn.YarnResourceManagerTest.testStopWorker(YarnResourceManagerTest.java:317)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:51)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
{code}
 

This exception is being thrown in 
`org.apache.flink.yarn.YarnResourceManagerTest#testStopWorker`. Exception 
apparently is being ignored, since the test completes. It seems like this line:
{code:java}
String fileLocation = 
System.getenv(UserGroupInformation.HADOOP_TOKEN_FILE_LOCATION){code}
Is not guarded against returned null value. I don't know if that's a test or 
production code issue.

CC [~till.rohrmann]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Closed] (FLINK-8806) Failure in UnionInputGate getNextBufferOrEvent()

2018-03-02 Thread Nico Kruber (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nico Kruber closed FLINK-8806.
--
Resolution: Duplicate

> Failure in UnionInputGate getNextBufferOrEvent()
> 
>
> Key: FLINK-8806
> URL: https://issues.apache.org/jira/browse/FLINK-8806
> Project: Flink
>  Issue Type: Bug
>  Components: Network
>Affects Versions: 1.5.0, 1.6.0
>Reporter: Stephan Ewen
>Priority: Blocker
> Fix For: 1.5.0, 1.6.0
>
>
> Error occurs in {{SelfConnectionITCase}}:
> Full log: https://api.travis-ci.org/v3/job/346847455/log.txt
> Exception Stack Trace
> {code}
> org.apache.flink.runtime.client.JobExecutionException: 
> java.lang.IllegalStateException
>   at 
> org.apache.flink.runtime.minicluster.MiniCluster.executeJobBlocking(MiniCluster.java:527)
>   at 
> org.apache.flink.streaming.util.TestStreamEnvironment.execute(TestStreamEnvironment.java:79)
>   at 
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1510)
>   at 
> org.apache.flink.test.streaming.runtime.SelfConnectionITCase.differentDataStreamDifferentChain(SelfConnectionITCase.java:158)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
>   at org.junit.rules.RunRules.evaluate(RunRules.java:20)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runners.Suite.runChild(Suite.java:128)
>   at org.junit.runners.Suite.runChild(Suite.java:27)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
>   at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
>   at 
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
>   at 
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.IllegalStateException
>   at 
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:179)
>   at 
> org.apache.flink.runtime.io.network.partition.consumer.UnionInputGate.getNextBufferOrEvent(UnionInputGate.java:173)
>   at 
> org.apache.flink.streaming.runtime.io.BarrierTracker.getNextNonBlocked(BarrierTracker.java:94)
>   at 
> org.apache.flink.streaming.runtime.io.StreamTwoInpu

[GitHub] flink pull request #5608: [FLINK-8821][TableAPI && SQL] Fix non-terminating ...

2018-03-02 Thread asfgit

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5608


---

[jira] [Commented] (FLINK-8821) Fix non-terminating decimal error

2018-03-02 Thread ASF GitHub Bot (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383351#comment-16383351
 ] 

ASF GitHub Bot commented on FLINK-8821:
---

Github user asfgit closed the pull request at:

https://github.com/apache/flink/pull/5608


> Fix non-terminating decimal error
> -
>
> Key: FLINK-8821
> URL: https://issues.apache.org/jira/browse/FLINK-8821
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Reporter: Ruidong Li
>Assignee: Ruidong Li
>Priority: Major
>
> The DecimalAvgAggFunction lacks precision protection



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (FLINK-8821) Fix non-terminating decimal error

2018-03-02 Thread Timo Walther (JIRA)


 [ 
https://issues.apache.org/jira/browse/FLINK-8821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Timo Walther resolved FLINK-8821.
-
   Resolution: Fixed
Fix Version/s: 1.6.0
   1.5.0

Fixed in 1.6.0: b31b707cb20f34633815718ff356e187f3397620
Fixed in 1.5.0: 347ec3848ec603ac452e394a5211cf888db6663f

> Fix non-terminating decimal error
> -
>
> Key: FLINK-8821
> URL: https://issues.apache.org/jira/browse/FLINK-8821
> Project: Flink
>  Issue Type: Bug
>  Components: Table API & SQL
>Reporter: Ruidong Li
>Assignee: Ruidong Li
>Priority: Major
> Fix For: 1.5.0, 1.6.0
>
>
> The DecimalAvgAggFunction lacks precision protection



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

2018-03-02 Thread yanxiaobin (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383356#comment-16383356
 ] 

yanxiaobin commented on FLINK-8794:
---

hi, [~pnowojski] ! Thank you for your suggestion！ The downstream processor can 
ignore the files with "*pending" or "*in-progress" sufixes and "_" prefix, but 
I don't think it's a good way to deal with it. We can change this behaviour/add 
an option for BucketingSink to use temporary "in-progress" and "pending" 
directories instead of prefixes， but the temporary "in-progress" and "pending" 
directories is still also a subdirectory of the base directory, and the 
downstream processor may still read the base directory recursively, It also 
results in reading redundant dirty data. I think the temporary data produced 
during the program should be isolated from the final output data. Thanks!

Also [~kkl0u] could you elaborate why rescaling forced us to keep lingering 
files? 

 

> When using BucketingSink, it happens that one of the files is always in the 
> [.in-progress] state
> 
>
> Key: FLINK-8794
> URL: https://issues.apache.org/jira/browse/FLINK-8794
> Project: Flink
>  Issue Type: Bug
>  Components: filesystem-connector
>Affects Versions: 1.4.0, 1.4.1
>Reporter: yanxiaobin
>Priority: Major
>
> When using BucketingSink, it happens that one of the files is always in the 
> [.in-progress] state. And this state has never changed after that.  The 
> underlying use of S3 as storage.
>  
> {code:java}
> // code placeholder
> {code}
> 2018-02-28 11:58:42  147341619 {color:#d04437}_part-28-0.in-progress{color}
> 2018-02-28 12:06:27  147315059 part-0-0
> 2018-02-28 12:06:27  147462359 part-1-0
> 2018-02-28 12:06:27  147316006 part-10-0
> 2018-02-28 12:06:28  147349854 part-100-0
> 2018-02-28 12:06:27  147421625 part-101-0
> 2018-02-28 12:06:27  147443830 part-102-0
> 2018-02-28 12:06:27  147372801 part-103-0
> 2018-02-28 12:06:27  147343670 part-104-0
> ..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (FLINK-8794) When using BucketingSink, it happens that one of the files is always in the [.in-progress] state

2018-03-02 Thread yanxiaobin (JIRA)


[ 
https://issues.apache.org/jira/browse/FLINK-8794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16383356#comment-16383356
 ] 

yanxiaobin edited comment on FLINK-8794 at 3/2/18 8:40 AM:
---

hi, [~pnowojski] Thank you for your suggestion！ The downstream processor can 
ignore the files with "*pending" or "*in-progress" sufixes and "_" prefix, but 
I don't think it's a good way to deal with it. We can change this behaviour/add 
an option for BucketingSink to use temporary "in-progress" and "pending" 
directories instead of prefixes， but the temporary "in-progress" and "pending" 
directories is still also a subdirectory of the base directory, and the 
downstream processor may still read the base directory recursively, It also 
results in reading redundant dirty data. I think the temporary data produced 
during the program should be isolated from the final output data. Thanks！


Also [~kkl0u] could you elaborate why rescaling forced us to keep lingering 
files? 

 


was (Author: backlight):
hi, [~pnowojski] ! Thank you for your suggestion！ The downstream processor can 
ignore the files with "*pending" or "*in-progress" sufixes and "_" prefix, but 
I don't think it's a good way to deal with it. We can change this behaviour/add 
an option for BucketingSink to use temporary "in-progress" and "pending" 
directories instead of prefixes， but the temporary "in-progress" and "pending" 
directories is still also a subdirectory of the base directory, and the 
downstream processor may still read the base directory recursively, It also 
results in reading redundant dirty data. I think the temporary data produced 
during the program should be isolated from the final output data. Thanks!

Also [~kkl0u] could you elaborate why rescaling forced us to keep lingering 
files? 

 

> When using BucketingSink, it happens that one of the files is always in the 
> [.in-progress] state
> 
>
> Key: FLINK-8794
> URL: https://issues.apache.org/jira/browse/FLINK-8794
> Project: Flink
>  Issue Type: Bug
>  Components: filesystem-connector
>Affects Versions: 1.4.0, 1.4.1
>Reporter: yanxiaobin
>Priority: Major
>
> When using BucketingSink, it happens that one of the files is always in the 
> [.in-progress] state. And this state has never changed after that.  The 
> underlying use of S3 as storage.
>  
> {code:java}
> // code placeholder
> {code}
> 2018-02-28 11:58:42  147341619 {color:#d04437}_part-28-0.in-progress{color}
> 2018-02-28 12:06:27  147315059 part-0-0
> 2018-02-28 12:06:27  147462359 part-1-0
> 2018-02-28 12:06:27  147316006 part-10-0
> 2018-02-28 12:06:28  147349854 part-100-0
> 2018-02-28 12:06:27  147421625 part-101-0
> 2018-02-28 12:06:27  147443830 part-102-0
> 2018-02-28 12:06:27  147372801 part-103-0
> 2018-02-28 12:06:27  147343670 part-104-0
> ..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (FLINK-8831) Create SQL Client dependencies

2018-03-02 Thread Timo Walther (JIRA)

Timo Walther created FLINK-8831:
---

Summary: Create SQL Client dependencies
Key: FLINK-8831
URL: https://issues.apache.org/jira/browse/FLINK-8831
Project: Flink
Issue Type: New Feature
Components: Table API & SQL
Reporter: Timo Walther
Assignee: Timo Walther

A first minimum version of FLIP-24 for the upcoming
Flink SQL Client has been merged to the master. We also merged
possibilities to discover and configure table sources without a single
line of code using string-based properties and Java service provider
discovery.

We are now facing the issue of how to manage dependencies in this new
environment. It is different from how regular Flink projects are created
(by setting up a a new Maven project and build a jar or fat jar).
Ideally, a user should be able to select from a set of prepared
connectors, catalogs, and formats. E.g., if a Kafka connector and Avro
format is needed, all that should be required is to move a
"flink-kafka.jar" and "flink-avro.jar" into the "sql_lib" directory that
is shipped to a Flink cluster together with the SQL query.

[As discussed on
ML|http://mail-archives.apache.org/mod_mbox/flink-dev/201802.mbox/%3C9c73518b-ec8e-3b01-f200-dea816c75efc%40apache.org%3E],
we will build fat jars for these modules with every Flink release that can be
hostet somewhere (e.g. Apache infrastructure, but not Maven central). This
would make it very easy to add a dependency by downloading the prepared JAR
files.

1 2 >

1 - 100 of 187 matches

Mail list logo