[GitHub] flink pull request: [FLINK-377] [FLINK-671] Generic Interface / PA...
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72640916 Just implemented the basic triangle enumeration job and figured out that this example is already included in this PR ;-) However, when trying to run both programs, I encountered two problems: 1. I had to manually create a file `/tmp/flink_data/output` and give it a certain size (100MB worked for me). I'm on OS X. 2. After I had that file, the ./pyflink2.sh command did not print any error message but did not respond either. Seemed kind of deadlocked. The error message for the missing file was not really helpful and could be improved. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-377) Create a general purpose framework for language bindings
[ https://issues.apache.org/jira/browse/FLINK-377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303164#comment-14303164 ] ASF GitHub Bot commented on FLINK-377: -- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72640916 Just implemented the basic triangle enumeration job and figured out that this example is already included in this PR ;-) However, when trying to run both programs, I encountered two problems: 1. I had to manually create a file `/tmp/flink_data/output` and give it a certain size (100MB worked for me). I'm on OS X. 2. After I had that file, the ./pyflink2.sh command did not print any error message but did not respond either. Seemed kind of deadlocked. The error message for the missing file was not really helpful and could be improved. Create a general purpose framework for language bindings Key: FLINK-377 URL: https://issues.apache.org/jira/browse/FLINK-377 Project: Flink Issue Type: Improvement Reporter: GitHub Import Assignee: Chesnay Schepler Labels: github-import Fix For: pre-apache A general purpose API to run operators with arbitrary binaries. This will allow to run Stratosphere programs written in Python, JavaScript, Ruby, Go or whatever you like. We suggest using Google Protocol Buffers for data serialization. This is the list of languages that currently support ProtoBuf: https://code.google.com/p/protobuf/wiki/ThirdPartyAddOns Very early prototype with python: https://github.com/rmetzger/scratch/tree/learn-protobuf (basically testing protobuf) For Ruby: https://github.com/infochimps-labs/wukong Two new students working at Stratosphere (@skunert and @filiphaase) are working on this. The reference binding language will be for Python, but other bindings are very welcome. The best name for this so far is stratosphere-lang-bindings. I created this issue to track the progress (and give everybody a chance to comment on this) Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/377 Created by: [rmetzger|https://github.com/rmetzger] Labels: enhancement, Assignee: [filiphaase|https://github.com/filiphaase] Created at: Tue Jan 07 19:47:20 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1437][Java API] Fixes copy() methods in...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/342 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1193][java-api][scala-api] Complete sup...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/348 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1464) Added ResultTypeQueryable interface to TypeSerializerInputFormat.
[
https://issues.apache.org/jira/browse/FLINK-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303177#comment-14303177
]
ASF GitHub Bot commented on FLINK-1464:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/349
Added ResultTypeQueryable interface to TypeSerializerInputFormat.
-
Key: FLINK-1464
URL: https://issues.apache.org/jira/browse/FLINK-1464
Project: Flink
Issue Type: Improvement
Components: Distributed Runtime, Optimizer
Affects Versions: 0.8, 0.9, 0.8.1
Reporter: Alexander Alexandrov
Assignee: Alexander Alexandrov
Priority: Minor
Labels: easyfix
Fix For: 0.9, 0.8.1
Original Estimate: 6h
Remaining Estimate: 6h
It is currently impossible to use the {{TypeSerializerInputFormat}} with
generic Tuple types.
For example, [this example
gist|https://gist.github.com/aalexandrov/90bf21f66bf604676f37] fails with a
{quote}
Exception in thread main
org.apache.flink.api.common.InvalidProgramException: The type returned by the
input format could not be automatically determined. Please specify the
TypeInformation of the produced type explicitly.
at
org.apache.flink.api.java.ExecutionEnvironment.readFile(ExecutionEnvironment.java:341)
at SerializedFormatExample$.main(SerializedFormatExample.scala:48)
at SerializedFormatExample.main(SerializedFormatExample.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
{quote}
exaception.
To fix the issue, I changed the constructor to take a {{TypeInformationT}}
instad of a {{TypeSerializerT}} argument. If this is indeed a bug, I think
that this is a good solution.
Unfortunately the fix breaks the API. Feel free to change it if you find a
more elegant solution compatible with the 0.8 branch.
The suggested fix can be found in the GitHub
[PR#349|https://github.com/apache/flink/pull/349].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (FLINK-1437) Bug in PojoSerializer's copy() method
[
https://issues.apache.org/jira/browse/FLINK-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303175#comment-14303175
]
ASF GitHub Bot commented on FLINK-1437:
---
Github user asfgit closed the pull request at:
https://github.com/apache/flink/pull/342
Bug in PojoSerializer's copy() method
-
Key: FLINK-1437
URL: https://issues.apache.org/jira/browse/FLINK-1437
Project: Flink
Issue Type: Bug
Components: Java API
Reporter: Timo Walther
Assignee: Timo Walther
The PojoSerializer's {{copy()}} method does not work properly with {{null}}
values. An exception could look like:
{code}
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: null
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:792)
Caused by: java.io.EOFException
at
org.apache.flink.runtime.io.disk.RandomAccessInputView.nextSegment(RandomAccessInputView.java:83)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.advance(AbstractPagedInputView.java:159)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readByte(AbstractPagedInputView.java:270)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readUnsignedByte(AbstractPagedInputView.java:277)
at org.apache.flink.types.StringValue.copyString(StringValue.java:839)
at
org.apache.flink.api.common.typeutils.base.StringSerializer.copy(StringSerializer.java:83)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.copy(PojoSerializer.java:261)
at
org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1303)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:788)
{code}
I'm working on a fix for that...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: Improved exception for missing type of InputFo...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/351 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1330) Restructure directory layout
[ https://issues.apache.org/jira/browse/FLINK-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303260#comment-14303260 ] ASF GitHub Bot commented on FLINK-1330: --- Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/333 Restructure directory layout Key: FLINK-1330 URL: https://issues.apache.org/jira/browse/FLINK-1330 Project: Flink Issue Type: Improvement Components: Build System, Documentation Reporter: Max Michels Priority: Minor Labels: usability When building Flink, the build results can currently be found under flink-root/flink-dist/target/flink-$FLINKVERSION-incubating-SNAPSHOT-bin/flink-$YARNVERSION-$FLINKVERSION-incubating-SNAPSHOT/. I think we could improve the directory layout with the following: - provide the bin folder in the root by default - let the start up and submissions scripts in bin assemble the class path - in case the project hasn't been build yet, inform the user The changes would make it easier to work with Flink from source. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-377) Create a general purpose framework for language bindings
[ https://issues.apache.org/jira/browse/FLINK-377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303170#comment-14303170 ] ASF GitHub Bot commented on FLINK-377: -- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72641447 btw. implementing the program felt quite good. Very nice API, IMO! Create a general purpose framework for language bindings Key: FLINK-377 URL: https://issues.apache.org/jira/browse/FLINK-377 Project: Flink Issue Type: Improvement Reporter: GitHub Import Assignee: Chesnay Schepler Labels: github-import Fix For: pre-apache A general purpose API to run operators with arbitrary binaries. This will allow to run Stratosphere programs written in Python, JavaScript, Ruby, Go or whatever you like. We suggest using Google Protocol Buffers for data serialization. This is the list of languages that currently support ProtoBuf: https://code.google.com/p/protobuf/wiki/ThirdPartyAddOns Very early prototype with python: https://github.com/rmetzger/scratch/tree/learn-protobuf (basically testing protobuf) For Ruby: https://github.com/infochimps-labs/wukong Two new students working at Stratosphere (@skunert and @filiphaase) are working on this. The reference binding language will be for Python, but other bindings are very welcome. The best name for this so far is stratosphere-lang-bindings. I created this issue to track the progress (and give everybody a chance to comment on this) Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/377 Created by: [rmetzger|https://github.com/rmetzger] Labels: enhancement, Assignee: [filiphaase|https://github.com/filiphaase] Created at: Tue Jan 07 19:47:20 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-377] [FLINK-671] Generic Interface / PA...
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72641447 btw. implementing the program felt quite good. Very nice API, IMO! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-1193) InvalidTypesException for Array[Array[Tuple2[Int, Double]]]
[
https://issues.apache.org/jira/browse/FLINK-1193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1193.
-
Resolution: Fixed
Fix Version/s: 0.9
Fixed via 57b3e9f9e50705033200dcca0762a5a65d1aba60
InvalidTypesException for Array[Array[Tuple2[Int, Double]]]
---
Key: FLINK-1193
URL: https://issues.apache.org/jira/browse/FLINK-1193
Project: Flink
Issue Type: Bug
Reporter: Till Rohrmann
Assignee: Timo Walther
Fix For: 0.9
The TypeExtractor cannot handle nested arrays with non-trivial element types.
For example, the following code cannot be run
{code}
val input = env.fromCollection(List(1,2,3,4))
val mapped = input.map{
id = {
val pair = (1,1.0)
val a = Array(pair)
val b = Array(a)
b
}
}
{code}
The resulting type Array[Array[Tuple2[Int, Double]]] cannot be extracted. It
would be nice, if Flink supported these types.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-377] [FLINK-671] Generic Interface / PA...
Github user zentol commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72652471 hmm...you are now the second person to report that creating the tmp files does not work on OS X. i don't know why that doesn't work. the file creation is done from java, is there any magic required there? i can't debug OS X error myself at the moment. all i can do on that front is add sanity checks for better error reporting. the included triangle enumeration is kinda odd, even if it runs the output is empty; Ive already checked the implementation yesterday and it appears equal to the java counterpart. will give it another go. the plan execution is one of the more fragile parts. generally, when the process exits with an error it is noticed. but if for example something is missing (like the call to execute) things just get stuck. this is due to the fact that information is only ever sent to java, but never received, a complete one-way street. since accumulators nor actions were supposed to be implemented anytime soon this seemed appropriate, but it seems that requires a change already. some timeouts could be useful as well. @fhueske Thanks for trying it out! --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1455) ExternalSortLargeRecordsITCase.testSortWithShortMediumAndLargeRecords: Potential Memory leak
[
https://issues.apache.org/jira/browse/FLINK-1455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303334#comment-14303334
]
Stephan Ewen commented on FLINK-1455:
-
Hmm, this seems a problem of the JVM setup on the test cluster.
The memory that it cannot allocate is not Flink memory, it is Java direct
memory, which the NIO code internally needs for I/O: Buffers are copied to
pooled DirectByteBuffers before issuing the I/O request.
Not sure what is happening there, this is a standalone test, no other NIO
memory consumers in the test. Seems like an issue with the Travis VMs.
ExternalSortLargeRecordsITCase.testSortWithShortMediumAndLargeRecords:
Potential Memory leak
Key: FLINK-1455
URL: https://issues.apache.org/jira/browse/FLINK-1455
Project: Flink
Issue Type: Bug
Components: Local Runtime
Affects Versions: 0.9
Reporter: Robert Metzger
Priority: Minor
This error occurred in one of my Travis jobs:
https://travis-ci.org/rmetzger/flink/jobs/48343022
Would be cool if somebody who knows the sorter better could verify/invalidate
the issue.
{code}
Running org.apache.flink.runtime.operators.sort.ExternalSortLargeRecordsITCase
java.lang.RuntimeException: Error obtaining the sorted input: Thread
'SortMerger spilling thread' terminated due to an exception: Cannot allocate
memory
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger.getIterator(UnilateralSortMerger.java:593)
at
org.apache.flink.runtime.operators.sort.ExternalSortLargeRecordsITCase.testSortWithShortMediumAndLargeRecords(ExternalSortLargeRecordsITCase.java:285)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229)
at org.junit.runners.ParentRunner.run(ParentRunner.java:309)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:264)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:200)
at
org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:153)
at
org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: Cannot allocate memory
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:770)
Caused by: java.io.IOException: Cannot allocate memory
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:205)
at
org.apache.flink.runtime.io.disk.iomanager.SegmentWriteRequest.write(AsynchronousFileIOChannel.java:267)
at
org.apache.flink.runtime.io.disk.iomanager.IOManagerAsync$WriterThread.run(IOManagerAsync.java:440)
Tests run: 5, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 56.089 sec
FAILURE! - in
[GitHub] flink pull request: Allow KeySelectors to implement ResultTypeQuer...
Github user aalexandrov commented on the pull request:
https://github.com/apache/flink/pull/354#issuecomment-72660318
I would advocate to adding this one as well as a fallback option.
I have a situation where I want to use KeySelector that might return Java
TupleXX instances parameterized with Scala types, e.g.:
```
class SelectFoo extends KeySelector[Tuple3[Int, Int, String], Tuple3[Int,
String]] {
override def getKey(v: Tuple3[Int, Int, Int]) = new Tuple2(v.f0, v.f2)
}
``
Even though in this cases the generic parameters are available, the Scala
types cannot be inferred because the actual type field type parameters are
erased by Scala and are seen only as java.lang.Object from the Java reflection
API.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] flink pull request: [FLINK-377] [FLINK-671] Generic Interface / PA...
Github user qmlmoon commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72668635 I would also like to try out some examples. I'm starting with TPCH queries:D --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1472) Web frontend config overview shows wrong value
Ufuk Celebi created FLINK-1472: -- Summary: Web frontend config overview shows wrong value Key: FLINK-1472 URL: https://issues.apache.org/jira/browse/FLINK-1472 Project: Flink Issue Type: Bug Components: Webfrontend Affects Versions: master Reporter: Ufuk Celebi Priority: Minor The web frontend shows configuration values even if they could not be correctly parsed. For example I've configured the number of buffers as 123.000, which cannot be parsed as an Integer by GlobalConfiguration and the default value is used. Still, the web frontend shows the not used 123.000. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Allow KeySelectors to implement ResultTypeQuer...
Github user aalexandrov commented on the pull request: https://github.com/apache/flink/pull/354#issuecomment-72660810 I think that [this StackOverflow article explains my problem](http://stackoverflow.com/questions/11586944/how-to-obtain-the-raw-datatype-of-a-parameter-of-a-field-that-is-specialized-in). --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1318] CsvInputFormat: Made quoted strin...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/265#issuecomment-72612157 Looks good so far. What about this PR is API breaking? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Allow KeySelectors to implement ResultTypeQuer...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/354#issuecomment-72614564 So, this pull request can be closed? --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1458) Interfaces and abstract classes are not valid types
[
https://issues.apache.org/jira/browse/FLINK-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302966#comment-14302966
]
ASF GitHub Bot commented on FLINK-1458:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/357#issuecomment-72614433
Looks good to me so far, modulo two comments:
- Since the type extractor code is changed a lot, it would be good to
have a test validating that the TypeExtractor allows abstract types and
interfaces.
- I would change the log level of the messages (abstract types are less
performant, type is no POJO) to INFO, since WARN implies to me that something
is wrong (which is not)
Interfaces and abstract classes are not valid types
---
Key: FLINK-1458
URL: https://issues.apache.org/jira/browse/FLINK-1458
Project: Flink
Issue Type: Bug
Reporter: John Sandiford
Assignee: Aljoscha Krettek
I don't know whether this is by design or is a bug, but I am having trouble
working with DataSet and traits in scala which is a major limitation. A
simple example is shown below.
Compile time warning is 'Type Main.SimpleTrait has no fields that are visible
from Scala Type analysis. Falling back to Java Type Analysis...'
Run time error is 'Interfaces and abstract classes are not valid types:
interface Main$SimpleTrait'
Regards, John
val env = ExecutionEnvironment.getExecutionEnvironment
trait SimpleTrait {
def contains(x: String): Boolean
}
class SimpleClass extends SimpleTrait {
def contains(x: String) = true
}
val data: DataSet[Double] = env.fromElements(1.0, 2.0, 3.0, 4.0)
def f(data: DataSet[Double]): DataSet[SimpleTrait] = {
data.mapPartition(iterator = {
Iterator(new SimpleClass)
})
}
val g = f(data)
g.print()
env.execute(Simple example)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1318] CsvInputFormat: Made quoted strin...
Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/265#issuecomment-72613154 The API was just extended, but the parsing logic for strings changed (see PR description). So programs that relied on the previous way of parsing will fail now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1318) Make quoted String parsing optional and configurable for CSVInputFormats
[ https://issues.apache.org/jira/browse/FLINK-1318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302954#comment-14302954 ] ASF GitHub Bot commented on FLINK-1318: --- Github user fhueske commented on the pull request: https://github.com/apache/flink/pull/265#issuecomment-72613154 The API was just extended, but the parsing logic for strings changed (see PR description). So programs that relied on the previous way of parsing will fail now. Make quoted String parsing optional and configurable for CSVInputFormats Key: FLINK-1318 URL: https://issues.apache.org/jira/browse/FLINK-1318 Project: Flink Issue Type: Improvement Components: Java API, Scala API Affects Versions: 0.8 Reporter: Fabian Hueske Assignee: Fabian Hueske Priority: Minor With the current implementation of the CSVInputFormat, quoted string parsing kicks in, if the first non-whitespace character of a field is a double quote. I see two issues with this implementation: 1. Quoted String parsing cannot be disabled 2. The quoting character is fixed to double quotes () I propose to add parameters to disable quoted String parsing and set the quote character. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-377] [FLINK-671] Generic Interface / PA...
Github user zentol commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72744279 about error messages going to command-line: the only way i see for that to work is by wrapping the complete error message into an exception, since they do show up on the command-line. wc deadlock: i just can't reproduce it. i tried small files (4 words) and went up to 750mb with dop=1. can you send me the test data you used? @qmlmoon THANK YOU! that would have taken me ages to figure out. working on a fix right now. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-377) Create a general purpose framework for language bindings
[ https://issues.apache.org/jira/browse/FLINK-377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304096#comment-14304096 ] ASF GitHub Bot commented on FLINK-377: -- Github user zentol commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72744279 about error messages going to command-line: the only way i see for that to work is by wrapping the complete error message into an exception, since they do show up on the command-line. wc deadlock: i just can't reproduce it. i tried small files (4 words) and went up to 750mb with dop=1. can you send me the test data you used? @qmlmoon THANK YOU! that would have taken me ages to figure out. working on a fix right now. Create a general purpose framework for language bindings Key: FLINK-377 URL: https://issues.apache.org/jira/browse/FLINK-377 Project: Flink Issue Type: Improvement Reporter: GitHub Import Assignee: Chesnay Schepler Labels: github-import Fix For: pre-apache A general purpose API to run operators with arbitrary binaries. This will allow to run Stratosphere programs written in Python, JavaScript, Ruby, Go or whatever you like. We suggest using Google Protocol Buffers for data serialization. This is the list of languages that currently support ProtoBuf: https://code.google.com/p/protobuf/wiki/ThirdPartyAddOns Very early prototype with python: https://github.com/rmetzger/scratch/tree/learn-protobuf (basically testing protobuf) For Ruby: https://github.com/infochimps-labs/wukong Two new students working at Stratosphere (@skunert and @filiphaase) are working on this. The reference binding language will be for Python, but other bindings are very welcome. The best name for this so far is stratosphere-lang-bindings. I created this issue to track the progress (and give everybody a chance to comment on this) Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/377 Created by: [rmetzger|https://github.com/rmetzger] Labels: enhancement, Assignee: [filiphaase|https://github.com/filiphaase] Created at: Tue Jan 07 19:47:20 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-377] [FLINK-671] Generic Interface / PA...
Github user qmlmoon commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72749480 Great. Now it works for me --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1330] [build] Build creates a link in t...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/333#issuecomment-72678344 I modified the merged version to exclude it from rat. I also tried subsequent builds, it worked. See here: https://github.com/apache/flink/commit/0c726f833892a03a12cc92d35e7d249369eb0dc9#diff-600376dffeb79835ede4a0b285078036R1046 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1314) Update website about #flink chat room in freenode IRC
[ https://issues.apache.org/jira/browse/FLINK-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303569#comment-14303569 ] Robert Metzger commented on FLINK-1314: --- Sorry, I was too busy today. The change looks good to merge. Thank you! Update website about #flink chat room in freenode IRC - Key: FLINK-1314 URL: https://issues.apache.org/jira/browse/FLINK-1314 Project: Flink Issue Type: Task Components: Project Website Reporter: Henry Saputra Assignee: Henry Saputra Priority: Minor Attachments: FLINK-1314.patch Update Flink website to mention the #flink chat room in freenode IRC -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-1314) Update website about #flink chat room in freenode IRC
[ https://issues.apache.org/jira/browse/FLINK-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303567#comment-14303567 ] Henry Saputra commented on FLINK-1314: -- If no comment, will merge this end of day Update website about #flink chat room in freenode IRC - Key: FLINK-1314 URL: https://issues.apache.org/jira/browse/FLINK-1314 Project: Flink Issue Type: Task Components: Project Website Reporter: Henry Saputra Assignee: Henry Saputra Priority: Minor Attachments: FLINK-1314.patch Update Flink website to mention the #flink chat room in freenode IRC -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1458] Allow Interfaces and abstract typ...
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/357#issuecomment-72642370 You're right. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Resolved] (FLINK-1437) Bug in PojoSerializer's copy() method
[
https://issues.apache.org/jira/browse/FLINK-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen resolved FLINK-1437.
-
Resolution: Fixed
Fix Version/s: 0.9
Fixed via fb7ce0e315e270e10c605dcfa269286bc2add47f
Bug in PojoSerializer's copy() method
-
Key: FLINK-1437
URL: https://issues.apache.org/jira/browse/FLINK-1437
Project: Flink
Issue Type: Bug
Components: Java API
Reporter: Timo Walther
Assignee: Timo Walther
Fix For: 0.9
The PojoSerializer's {{copy()}} method does not work properly with {{null}}
values. An exception could look like:
{code}
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: null
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:792)
Caused by: java.io.EOFException
at
org.apache.flink.runtime.io.disk.RandomAccessInputView.nextSegment(RandomAccessInputView.java:83)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.advance(AbstractPagedInputView.java:159)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readByte(AbstractPagedInputView.java:270)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readUnsignedByte(AbstractPagedInputView.java:277)
at org.apache.flink.types.StringValue.copyString(StringValue.java:839)
at
org.apache.flink.api.common.typeutils.base.StringSerializer.copy(StringSerializer.java:83)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.copy(PojoSerializer.java:261)
at
org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1303)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:788)
{code}
I'm working on a fix for that...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1330] [build] Build creates a link in t...
Github user asfgit closed the pull request at: https://github.com/apache/flink/pull/333 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1458) Interfaces and abstract classes are not valid types
[
https://issues.apache.org/jira/browse/FLINK-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303187#comment-14303187
]
ASF GitHub Bot commented on FLINK-1458:
---
Github user aljoscha commented on the pull request:
https://github.com/apache/flink/pull/357#issuecomment-72642370
You're right.
Interfaces and abstract classes are not valid types
---
Key: FLINK-1458
URL: https://issues.apache.org/jira/browse/FLINK-1458
Project: Flink
Issue Type: Bug
Reporter: John Sandiford
Assignee: Aljoscha Krettek
I don't know whether this is by design or is a bug, but I am having trouble
working with DataSet and traits in scala which is a major limitation. A
simple example is shown below.
Compile time warning is 'Type Main.SimpleTrait has no fields that are visible
from Scala Type analysis. Falling back to Java Type Analysis...'
Run time error is 'Interfaces and abstract classes are not valid types:
interface Main$SimpleTrait'
Regards, John
val env = ExecutionEnvironment.getExecutionEnvironment
trait SimpleTrait {
def contains(x: String): Boolean
}
class SimpleClass extends SimpleTrait {
def contains(x: String) = true
}
val data: DataSet[Double] = env.fromElements(1.0, 2.0, 3.0, 4.0)
def f(data: DataSet[Double]): DataSet[SimpleTrait] = {
data.mapPartition(iterator = {
Iterator(new SimpleClass)
})
}
val g = f(data)
g.print()
env.execute(Simple example)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (FLINK-1330) Restructure directory layout
[ https://issues.apache.org/jira/browse/FLINK-1330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-1330. - Resolution: Fixed Fix Version/s: 0.9 Assignee: Stephan Ewen Fixed via 0c726f833892a03a12cc92d35e7d249369eb0dc9 Restructure directory layout Key: FLINK-1330 URL: https://issues.apache.org/jira/browse/FLINK-1330 Project: Flink Issue Type: Improvement Components: Build System, Documentation Reporter: Max Michels Assignee: Stephan Ewen Priority: Minor Labels: usability Fix For: 0.9 When building Flink, the build results can currently be found under flink-root/flink-dist/target/flink-$FLINKVERSION-incubating-SNAPSHOT-bin/flink-$YARNVERSION-$FLINKVERSION-incubating-SNAPSHOT/. I think we could improve the directory layout with the following: - provide the bin folder in the root by default - let the start up and submissions scripts in bin assemble the class path - in case the project hasn't been build yet, inform the user The changes would make it easier to work with Flink from source. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (FLINK-377) Create a general purpose framework for language bindings
[ https://issues.apache.org/jira/browse/FLINK-377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303266#comment-14303266 ] ASF GitHub Bot commented on FLINK-377: -- Github user zentol commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72652471 hmm...you are now the second person to report that creating the tmp files does not work on OS X. i don't know why that doesn't work. the file creation is done from java, is there any magic required there? i can't debug OS X error myself at the moment. all i can do on that front is add sanity checks for better error reporting. the included triangle enumeration is kinda odd, even if it runs the output is empty; Ive already checked the implementation yesterday and it appears equal to the java counterpart. will give it another go. the plan execution is one of the more fragile parts. generally, when the process exits with an error it is noticed. but if for example something is missing (like the call to execute) things just get stuck. this is due to the fact that information is only ever sent to java, but never received, a complete one-way street. since accumulators nor actions were supposed to be implemented anytime soon this seemed appropriate, but it seems that requires a change already. some timeouts could be useful as well. @fhueske Thanks for trying it out! Create a general purpose framework for language bindings Key: FLINK-377 URL: https://issues.apache.org/jira/browse/FLINK-377 Project: Flink Issue Type: Improvement Reporter: GitHub Import Assignee: Chesnay Schepler Labels: github-import Fix For: pre-apache A general purpose API to run operators with arbitrary binaries. This will allow to run Stratosphere programs written in Python, JavaScript, Ruby, Go or whatever you like. We suggest using Google Protocol Buffers for data serialization. This is the list of languages that currently support ProtoBuf: https://code.google.com/p/protobuf/wiki/ThirdPartyAddOns Very early prototype with python: https://github.com/rmetzger/scratch/tree/learn-protobuf (basically testing protobuf) For Ruby: https://github.com/infochimps-labs/wukong Two new students working at Stratosphere (@skunert and @filiphaase) are working on this. The reference binding language will be for Python, but other bindings are very welcome. The best name for this so far is stratosphere-lang-bindings. I created this issue to track the progress (and give everybody a chance to comment on this) Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/377 Created by: [rmetzger|https://github.com/rmetzger] Labels: enhancement, Assignee: [filiphaase|https://github.com/filiphaase] Created at: Tue Jan 07 19:47:20 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1471) Allow KeySelectors to implement ResultTypeQueryable
[ https://issues.apache.org/jira/browse/FLINK-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-1471. - Resolution: Fixed Fix Version/s: 0.9 Fixed via e3f6c9ba69a3e545fdd8f18b7b652fa111ade93e Thanks for the patch! Allow KeySelectors to implement ResultTypeQueryable --- Key: FLINK-1471 URL: https://issues.apache.org/jira/browse/FLINK-1471 Project: Flink Issue Type: Bug Components: Java API Affects Versions: 0.9 Reporter: Robert Metzger Assignee: Timo Walther Fix For: 0.9 See https://github.com/apache/flink/pull/354 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (FLINK-1464) Added ResultTypeQueryable interface to TypeSerializerInputFormat.
[
https://issues.apache.org/jira/browse/FLINK-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger resolved FLINK-1464.
---
Resolution: Fixed
Fixed via e3f6c9ba69a3e545fdd8f18b7b652fa111ade93e
Thanks for the patch!
The fix has been merged by Stephan Ewen.
Added ResultTypeQueryable interface to TypeSerializerInputFormat.
-
Key: FLINK-1464
URL: https://issues.apache.org/jira/browse/FLINK-1464
Project: Flink
Issue Type: Improvement
Components: Distributed Runtime, Optimizer
Affects Versions: 0.8, 0.9, 0.8.1
Reporter: Alexander Alexandrov
Assignee: Alexander Alexandrov
Priority: Minor
Labels: easyfix
Fix For: 0.9, 0.8.1
Original Estimate: 6h
Remaining Estimate: 6h
It is currently impossible to use the {{TypeSerializerInputFormat}} with
generic Tuple types.
For example, [this example
gist|https://gist.github.com/aalexandrov/90bf21f66bf604676f37] fails with a
{quote}
Exception in thread main
org.apache.flink.api.common.InvalidProgramException: The type returned by the
input format could not be automatically determined. Please specify the
TypeInformation of the produced type explicitly.
at
org.apache.flink.api.java.ExecutionEnvironment.readFile(ExecutionEnvironment.java:341)
at SerializedFormatExample$.main(SerializedFormatExample.scala:48)
at SerializedFormatExample.main(SerializedFormatExample.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
{quote}
exaception.
To fix the issue, I changed the constructor to take a {{TypeInformationT}}
instad of a {{TypeSerializerT}} argument. If this is indeed a bug, I think
that this is a good solution.
Unfortunately the fix breaks the API. Feel free to change it if you find a
more elegant solution compatible with the 0.8 branch.
The suggested fix can be found in the GitHub
[PR#349|https://github.com/apache/flink/pull/349].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Reopened] (FLINK-1471) Allow KeySelectors to implement ResultTypeQueryable
[ https://issues.apache.org/jira/browse/FLINK-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger reopened FLINK-1471: --- The commit Stephan mentions fixes another issue. This is still unresolved. Allow KeySelectors to implement ResultTypeQueryable --- Key: FLINK-1471 URL: https://issues.apache.org/jira/browse/FLINK-1471 Project: Flink Issue Type: Bug Components: Java API Affects Versions: 0.9 Reporter: Robert Metzger Assignee: Timo Walther Fix For: 0.9 See https://github.com/apache/flink/pull/354 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-592] Add support for Kerberos secured Y...
GitHub user rmetzger opened a pull request: https://github.com/apache/flink/pull/358 [FLINK-592] Add support for Kerberos secured YARN setups to Flink. This pull request is basically a port of @warneke's branch (https://github.com/warneke/flink/tree/security) to the latest `master` of Flink. The port has been done mostly by @mxm. We tested the change on google compute engine (non-secure setup, to ensure that everything is working as before) and a local secure YARN setup with Kerberos. Open issues: - Test token renewal Once the open issues have been resolved, I would like to merge this asap because a user was asking for this on the mailing list. You can merge this pull request into a Git repository by running: $ git pull https://github.com/rmetzger/flink flink592 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/358.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #358 commit 3fc8d47f3f7322285539454c7a80a8cec4ba043f Author: Max m...@posteo.de Date: 2015-02-02T15:09:18Z [FLINK-592] Add support for Kerberos secured YARN setups to Flink. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-377) Create a general purpose framework for language bindings
[ https://issues.apache.org/jira/browse/FLINK-377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304192#comment-14304192 ] ASF GitHub Bot commented on FLINK-377: -- Github user zentol commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72752968 @qmlmoon sweet. @rmetzger errors should show up on the console now. and in the .out file. and i suppose by extension in the .log file aswell. Create a general purpose framework for language bindings Key: FLINK-377 URL: https://issues.apache.org/jira/browse/FLINK-377 Project: Flink Issue Type: Improvement Reporter: GitHub Import Assignee: Chesnay Schepler Labels: github-import Fix For: pre-apache A general purpose API to run operators with arbitrary binaries. This will allow to run Stratosphere programs written in Python, JavaScript, Ruby, Go or whatever you like. We suggest using Google Protocol Buffers for data serialization. This is the list of languages that currently support ProtoBuf: https://code.google.com/p/protobuf/wiki/ThirdPartyAddOns Very early prototype with python: https://github.com/rmetzger/scratch/tree/learn-protobuf (basically testing protobuf) For Ruby: https://github.com/infochimps-labs/wukong Two new students working at Stratosphere (@skunert and @filiphaase) are working on this. The reference binding language will be for Python, but other bindings are very welcome. The best name for this so far is stratosphere-lang-bindings. I created this issue to track the progress (and give everybody a chance to comment on this) Imported from GitHub Url: https://github.com/stratosphere/stratosphere/issues/377 Created by: [rmetzger|https://github.com/rmetzger] Labels: enhancement, Assignee: [filiphaase|https://github.com/filiphaase] Created at: Tue Jan 07 19:47:20 CET 2014 State: open -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-377] [FLINK-671] Generic Interface / PA...
Github user zentol commented on the pull request: https://github.com/apache/flink/pull/202#issuecomment-72752968 @qmlmoon sweet. @rmetzger errors should show up on the console now. and in the .out file. and i suppose by extension in the .log file aswell. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: Allow KeySelectors to implement ResultTypeQuer...
Github user aljoscha commented on the pull request: https://github.com/apache/flink/pull/354#issuecomment-72752435 Nope, sorry, also have no Idea why this is happening. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] flink pull request: [FLINK-1376] [runtime] Add proper shared slot ...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/317#issuecomment-72618925 I think this is a good fix, overall. There is one issue I would really like to fix, and that is the serializability of the `Instance` class. This class is not meant to be serialized and moved around, which can be reflected by the fact that it holds an Actor Ref, and the necessity to make a lot of the fields transient. I assume that the instance needs to be serialized as part of the ExechutionGraph archiving, where the ExecutionGraph is sent via an actor message to the archiver. I would like to solve that differently. The execution graph is cleaned before archiving (see #344 ) to reduce memory footprint. At this point, I would replace the `Instance` in the Executions with the `Instance Connection Info`, which holds all info necessary. Then we won't have to send instances through actor messages, which would be the cleaner solution. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1471) Allow KeySelectors to implement ResultTypeQueryable
[ https://issues.apache.org/jira/browse/FLINK-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303007#comment-14303007 ] ASF GitHub Bot commented on FLINK-1471: --- Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/354#issuecomment-72620298 I'll close it. I've filed a jira for the issue and assigned it to @twalthr: https://issues.apache.org/jira/browse/FLINK-1471 Allow KeySelectors to implement ResultTypeQueryable --- Key: FLINK-1471 URL: https://issues.apache.org/jira/browse/FLINK-1471 Project: Flink Issue Type: Bug Components: Java API Affects Versions: 0.9 Reporter: Robert Metzger Assignee: Timo Walther See https://github.com/apache/flink/pull/354 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: [FLINK-1464] Add ResultTypeQueryable interface...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/349#issuecomment-72631504 Looks good to me. I think the API breaking is minor, since it does not affect the methods on `DataSet` or on any of the operators. The TypeSerializerInputFormat is only used explicitly and is rather new, so it should be okay. +1 Will merge this. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1464) Added ResultTypeQueryable interface to TypeSerializerInputFormat.
[
https://issues.apache.org/jira/browse/FLINK-1464?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303075#comment-14303075
]
ASF GitHub Bot commented on FLINK-1464:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/349#issuecomment-72631504
Looks good to me. I think the API breaking is minor, since it does not
affect the methods on `DataSet` or on any of the operators. The
TypeSerializerInputFormat is only used explicitly and is rather new, so it
should be okay.
+1
Will merge this.
Added ResultTypeQueryable interface to TypeSerializerInputFormat.
-
Key: FLINK-1464
URL: https://issues.apache.org/jira/browse/FLINK-1464
Project: Flink
Issue Type: Improvement
Components: Distributed Runtime, Optimizer
Affects Versions: 0.8, 0.9, 0.8.1
Reporter: Alexander Alexandrov
Assignee: Alexander Alexandrov
Priority: Minor
Labels: easyfix
Fix For: 0.9, 0.8.1
Original Estimate: 6h
Remaining Estimate: 6h
It is currently impossible to use the {{TypeSerializerInputFormat}} with
generic Tuple types.
For example, [this example
gist|https://gist.github.com/aalexandrov/90bf21f66bf604676f37] fails with a
{quote}
Exception in thread main
org.apache.flink.api.common.InvalidProgramException: The type returned by the
input format could not be automatically determined. Please specify the
TypeInformation of the produced type explicitly.
at
org.apache.flink.api.java.ExecutionEnvironment.readFile(ExecutionEnvironment.java:341)
at SerializedFormatExample$.main(SerializedFormatExample.scala:48)
at SerializedFormatExample.main(SerializedFormatExample.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:134)
{quote}
exaception.
To fix the issue, I changed the constructor to take a {{TypeInformationT}}
instad of a {{TypeSerializerT}} argument. If this is indeed a bug, I think
that this is a good solution.
Unfortunately the fix breaks the API. Feel free to change it if you find a
more elegant solution compatible with the 0.8 branch.
The suggested fix can be found in the GitHub
[PR#349|https://github.com/apache/flink/pull/349].
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: Improved exception for missing type of InputFo...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/351#issuecomment-72629129 I tried to implement Henry's idea, but I noticed the error message is very specific to the site where it is created, since it refers to an alternate method of `DataSet`. It does not make sense to have that specific message in a different place. I would leave it as it is for now... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1376) SubSlots are not properly released in case that a TaskManager fatally fails, leaving the system in a corrupted state
[ https://issues.apache.org/jira/browse/FLINK-1376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14302997#comment-14302997 ] ASF GitHub Bot commented on FLINK-1376: --- Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/317#issuecomment-72618925 I think this is a good fix, overall. There is one issue I would really like to fix, and that is the serializability of the `Instance` class. This class is not meant to be serialized and moved around, which can be reflected by the fact that it holds an Actor Ref, and the necessity to make a lot of the fields transient. I assume that the instance needs to be serialized as part of the ExechutionGraph archiving, where the ExecutionGraph is sent via an actor message to the archiver. I would like to solve that differently. The execution graph is cleaned before archiving (see #344 ) to reduce memory footprint. At this point, I would replace the `Instance` in the Executions with the `Instance Connection Info`, which holds all info necessary. Then we won't have to send instances through actor messages, which would be the cleaner solution. SubSlots are not properly released in case that a TaskManager fatally fails, leaving the system in a corrupted state Key: FLINK-1376 URL: https://issues.apache.org/jira/browse/FLINK-1376 Project: Flink Issue Type: Bug Reporter: Till Rohrmann Assignee: Till Rohrmann In case that the TaskManager fatally fails and some of the failing node's slots are SharedSlots, then the slots are not properly released by the JobManager. This causes that the corresponding job will not be properly failed, leaving the system in a corrupted state. The reason for that is that the AllocatedSlot is not aware of being treated as a SharedSlot and thus he cannot release the associated SubSlots. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Improved exception for missing type of InputFo...
Github user StephanEwen commented on the pull request: https://github.com/apache/flink/pull/351#issuecomment-72623931 I'll merge this... --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1437) Bug in PojoSerializer's copy() method
[
https://issues.apache.org/jira/browse/FLINK-1437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14303028#comment-14303028
]
ASF GitHub Bot commented on FLINK-1437:
---
Github user StephanEwen commented on the pull request:
https://github.com/apache/flink/pull/342#issuecomment-72623753
Ah, just saw that the tests are there. My bad...
Bug in PojoSerializer's copy() method
-
Key: FLINK-1437
URL: https://issues.apache.org/jira/browse/FLINK-1437
Project: Flink
Issue Type: Bug
Components: Java API
Reporter: Timo Walther
Assignee: Timo Walther
The PojoSerializer's {{copy()}} method does not work properly with {{null}}
values. An exception could look like:
{code}
Caused by: java.io.IOException: Thread 'SortMerger spilling thread'
terminated due to an exception: null
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:792)
Caused by: java.io.EOFException
at
org.apache.flink.runtime.io.disk.RandomAccessInputView.nextSegment(RandomAccessInputView.java:83)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.advance(AbstractPagedInputView.java:159)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readByte(AbstractPagedInputView.java:270)
at
org.apache.flink.runtime.memorymanager.AbstractPagedInputView.readUnsignedByte(AbstractPagedInputView.java:277)
at org.apache.flink.types.StringValue.copyString(StringValue.java:839)
at
org.apache.flink.api.common.typeutils.base.StringSerializer.copy(StringSerializer.java:83)
at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.copy(PojoSerializer.java:261)
at
org.apache.flink.runtime.operators.sort.NormalizedKeySorter.writeToOutput(NormalizedKeySorter.java:449)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$SpillingThread.go(UnilateralSortMerger.java:1303)
at
org.apache.flink.runtime.operators.sort.UnilateralSortMerger$ThreadBase.run(UnilateralSortMerger.java:788)
{code}
I'm working on a fix for that...
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[GitHub] flink pull request: Allow KeySelectors to implement ResultTypeQuer...
Github user rmetzger closed the pull request at: https://github.com/apache/flink/pull/354 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Created] (FLINK-1471) Allow KeySelectors to implement ResultTypeQueryable
Robert Metzger created FLINK-1471: - Summary: Allow KeySelectors to implement ResultTypeQueryable Key: FLINK-1471 URL: https://issues.apache.org/jira/browse/FLINK-1471 Project: Flink Issue Type: Bug Components: Java API Affects Versions: 0.9 Reporter: Robert Metzger Assignee: Timo Walther See https://github.com/apache/flink/pull/354 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] flink pull request: Allow KeySelectors to implement ResultTypeQuer...
Github user rmetzger commented on the pull request: https://github.com/apache/flink/pull/354#issuecomment-72620298 I'll close it. I've filed a jira for the issue and assigned it to @twalthr: https://issues.apache.org/jira/browse/FLINK-1471 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (FLINK-1471) Allow KeySelectors to implement ResultTypeQueryable
[ https://issues.apache.org/jira/browse/FLINK-1471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14304088#comment-14304088 ] ASF GitHub Bot commented on FLINK-1471: --- GitHub user twalthr opened a pull request: https://github.com/apache/flink/pull/359 [FLINK-1471][java-api] Fixes wrong input validation if function has no generics FLINK-1471 was not implemented properly. See also #354. This PR skips the input validation if no generic parameters are available. You can merge this pull request into a Git repository by running: $ git pull https://github.com/twalthr/flink UnParameterizedFunctions Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/359.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #359 commit 7f70d7ef500cbb572a66f775d35609ac5903f767 Author: twalthr twal...@apache.org Date: 2015-02-03T21:51:10Z [FLINK-1471][java-api] Fixes wrong input validation if function has no generics Allow KeySelectors to implement ResultTypeQueryable --- Key: FLINK-1471 URL: https://issues.apache.org/jira/browse/FLINK-1471 Project: Flink Issue Type: Bug Components: Java API Affects Versions: 0.9 Reporter: Robert Metzger Assignee: Timo Walther Fix For: 0.9 See https://github.com/apache/flink/pull/354 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
