[jira] [Created] (FLINK-13419) TableEnvironment.explain() has side-effects on ExecutionConfig

[Timo Walther (JIRA)]

Timo Walther created FLINK-13419:


 Summary: TableEnvironment.explain() has side-effects on 
ExecutionConfig
 Key: FLINK-13419
 URL: https://issues.apache.org/jira/browse/FLINK-13419
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Planner
Reporter: Timo Walther


The Blink Planner seems to set batch properties in 
{{org.apache.flink.table.planner.delegation.BatchExecutor#generateStreamGraph}} 
which remain after {{TableEnvironment.explain()}}.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13420) Supported hive versions are hard coded

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13420:
--

 Summary: Supported hive versions are hard coded
 Key: FLINK-13420
 URL: https://issues.apache.org/jira/browse/FLINK-13420
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Affects Versions: 1.9.0
Reporter: Jeff Zhang


Currently, the supported hive versions are hardcoded in [HiveShimLoader 
|[https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-hive/src/main/java/org/apache/flink/table/catalog/hive/client/HiveShimLoader.java]].
 That makes me unable to run hive connector for 1.2.2, I believe we only need 
to check the major.minor version number, but ignore the bugfix version number.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] Support temporary tables in SQL API

[Aljoscha Krettek]

Thanks for pushing the discussion, Dawid! I’m also fine with option #3.

Aljoscha

> On 24. Jul 2019, at 12:04, Dawid Wysakowicz  wrote:
> 
> Hi all,
> 
> Thank you Xuefu for clarifying your opinion. Now we have 3 votes for both of 
> the options. To conclude this discussion I am willing to change my vote to 
> option 3 as I had only a slight preference towards option 2.
> 
> Therefore the final results of the poll are as follows:
> 
> option #2: 2 votes(Kurt, Aljoscha)
> 
> option #3: 4 votes(Timo, Jingsong, Xuefu, me)
> 
> I will prepare appropriate PRs according to the decision (unless somebody 
> objects). We will revisit the long-term solution in a separate thread as part 
> of the 1.10 release after 1.9 is released.
> 
> Thank you all for your opinions!
> 
> Best,
> 
> Dawid
> 
> On 24/07/2019 09:35, Aljoscha Krettek wrote:
>> Isn’t https://issues.apache.org/jira/browse/FLINK-13279 
>>  
>>  
>>  already a sign that 
>> there are surprises for users if we go with option #3?
>> 
>> Aljoscha
>> 
>>> On 24. Jul 2019, at 00:33, Xuefu Z  
>>>  wrote:
>>> 
>>> I favored #3 if that wasn't obvious.
>>> 
>>> Usability issue with #2 makes Hive  too hard to use. #3 keeps the old
>>> behavior for existing users who don't have Hive and thus there is only one,
>>> in-memory catalog. If a user does register Hive, he/she understands that
>>> there are multiple catalogs and that fully qualified table name is
>>> necessary. Thus, #3 has no impact (and no surprises) for existing users,
>>> and new requirement of fully qualified names is for only for users of the
>>> new feature (multiple catalogs), which seems very natural.
>>> 
>>> Thanks,
>>> Xuefu
>>> 
>>> On Tue, Jul 23, 2019 at 5:47 AM Dawid Wysakowicz >>   
>>> >
>>> wrote:
>>> 
 I think we all agree so far that we should implement one of the short term
 solutions for 1.9 release (#2 or #3) and continue the discussion on option
 #1 for the next release. Personally I prefer option #2, because it is
 closest to the current behavior and as Kurt said it is the most intuitive
 one, but I am also fine with option #3
 
 To sum up the opinions so far:
 
 *option #2: 3 votes(Kurt, Aljoscha, me)*
 
 *option #3: 2 votes(Timo, Jingsong)*
 
 I wasn't sure which option out of the two Xuefu prefers.
 
 I would like to conclude the discussion by the end of tomorrow, so that we
 can prepare a proper fix as soon as possible. Therefore I would suggest to
 proceed with the option that gets the most votes until tomorrow (*July
 24th 12:00 CET*), unless there are some hard objections.
 
 
 Comment on option #1 concerns:
 
 I agree with Jingsong on that. I think there are some benefits of the
 approach, as it makes Flink in control of the temporary tables.
 
 1. We have a unified behavior across all catalogs. Also for the catalogs
 that do not support temporary tables natively.
 
 2. As Flink is in control of the temporary tables it makes it easier to
 control their lifecycle.
 
 Best,
 
 Dawid
 On 23/07/2019 11:40, JingsongLee wrote:
 
 And I think we should recommend user to use catalog api to
 createTable and createFunction,(I guess most scenarios do
 not use temporary objects) in this way, it is good to option #3
 
 Best, JingsongLee
 
 
 --
 From:JingsongLee >>>  
  
 > >>>  
  
 >
 Send Time:2019年7月23日(星期二) 17:35
 To:dev mailto:dev@flink.apache.org> 
  > 
 mailto:dev@flink.apache.org> 
  >
 Subject:Re: [DISCUSS] Support temporary tables in SQL API
 
 Thanks Dawid and other people.
 +1 for using option #3 for 1.9.0 and go with option #1
 in 1.10.0.
 
 Regarding Xuefu's concern, I don't know how necessary it is for each 
 catalog to
 deal with tmpView. I think Catalog is different from DB, we can have 
 single concept for tmpView, that make user easier to understand.
 
 Regarding option #2, It is hard to use if we let user to use fully 
 qualified name for hive catalog. Would this experience be too bad to use?
 
 Best, Jingsong Lee
 
 
 --
 From:Kurt You

[jira] [Created] (FLINK-13421) Unexpected ConcurrentModificationException when RM notify JM about allocation failure

[Zhu Zhu (JIRA)]

Zhu Zhu created FLINK-13421:
---

 Summary: Unexpected ConcurrentModificationException when RM notify 
JM about allocation failure
 Key: FLINK-13421
 URL: https://issues.apache.org/jira/browse/FLINK-13421
 Project: Flink
  Issue Type: Bug
Reporter: Zhu Zhu


When a TM lost and RM identified it first, it will notify JM about it through 
JobMaster#notifyAllocationFailure. 

We observed unexpected ConcurrentModificationException in this process, stack 
as below:

 

Caused by: java.util.ConcurrentModificationException

        at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)

        at java.util.HashMap$ValueIterator.next(HashMap.java:1466)

        at 
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:477)

        at 
org.apache.flink.runtime.jobmaster.slotpool.AllocatedSlot.releasePayload(AllocatedSlot.java:149)

        at 
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.tryFailingAllocatedSlot(SlotPoolImpl.java:712)

        at 
org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl.failAllocation(SlotPoolImpl.java:692)

        at 
org.apache.flink.runtime.jobmaster.JobMaster.internalFailAllocation(JobMaster.java:538)

        at 
org.apache.flink.runtime.jobmaster.JobMaster.notifyAllocationFailure(JobMaster.java:664)

        ... 26 more

 

This can cause a allocated slot removed from SlotPool#allocatedSlots but not 
all of its payload tasks get failed. Tasks may hang in scheduled forever, as in 
the attached log.

It is not figured out yet that how a concurrent modification can happen. We do 
not have a debug log for it and is not able to re-pro it with debug log enabled 
yet.

However, we can let SlotSharingManager$MultiTaskSlot do not iterate on its 
children directly to avoid ConcurrentModificationException to occur in any case.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] FLIP-27: Refactor Source Interface

[Biao Liu]

Hi devs,

Since 1.9 is nearly released, I think we could get back to FLIP-27. I
believe it should be included in 1.10.

There are so many things mentioned in document of FLIP-27. [1] I think we'd
better discuss them separately. However the wiki is not a good place to
discuss. I wrote google doc about SplitReader API which misses some details
in the document. [2]

1.
https://cwiki.apache.org/confluence/display/FLINK/FLIP-27:+Refactor+Source+Interface
2.
https://docs.google.com/document/d/1R1s_89T4S3CZwq7Tf31DciaMCqZwrLHGZFqPASu66oE/edit?usp=sharing

CC Stephan, Aljoscha, Piotrek, Becket


On Thu, Mar 28, 2019 at 4:38 PM Biao Liu  wrote:

> Hi Steven,
> Thank you for the feedback. Please take a look at the document FLIP-27
> 
>  which
> is updated recently. A lot of details of enumerator were added in this
> document. I think it would help.
>
> Steven Wu  于2019年3月28日周四 下午12:52写道：
>
>> This proposal mentioned that SplitEnumerator might run on the JobManager
>> or
>> in a single task on a TaskManager.
>>
>> if enumerator is a single task on a taskmanager, then the job DAG can
>> never
>> been embarrassingly parallel anymore. That will nullify the leverage of
>> fine-grained recovery for embarrassingly parallel jobs.
>>
>> It's not clear to me what's the implication of running enumerator on the
>> jobmanager. So I will leave that out for now.
>>
>> On Mon, Jan 28, 2019 at 3:05 AM Biao Liu  wrote:
>>
>> > Hi Stephan & Piotrek,
>> >
>> > Thank you for feedback.
>> >
>> > It seems that there are a lot of things to do in community. I am just
>> > afraid that this discussion may be forgotten since there so many
>> proposals
>> > recently.
>> > Anyway, wish to see the split topics soon :)
>> >
>> > Piotr Nowojski  于2019年1月24日周四 下午8:21写道：
>> >
>> > > Hi Biao!
>> > >
>> > > This discussion was stalled because of preparations for the open
>> sourcing
>> > > & merging Blink. I think before creating the tickets we should split
>> this
>> > > discussion into topics/areas outlined by Stephan and create Flips for
>> > that.
>> > >
>> > > I think there is no chance for this to be completed in couple of
>> > remaining
>> > > weeks/1 month before 1.8 feature freeze, however it would be good to
>> aim
>> > > with those changes for 1.9.
>> > >
>> > > Piotrek
>> > >
>> > > > On 20 Jan 2019, at 16:08, Biao Liu  wrote:
>> > > >
>> > > > Hi community,
>> > > > The summary of Stephan makes a lot sense to me. It is much clearer
>> > indeed
>> > > > after splitting the complex topic into small ones.
>> > > > I was wondering is there any detail plan for next step? If not, I
>> would
>> > > > like to push this thing forward by creating some JIRA issues.
>> > > > Another question is that should version 1.8 include these features?
>> > > >
>> > > > Stephan Ewen  于2018年12月1日周六 上午4:20写道：
>> > > >
>> > > >> Thanks everyone for the lively discussion. Let me try to summarize
>> > > where I
>> > > >> see convergence in the discussion and open issues.
>> > > >> I'll try to group this by design aspect of the source. Please let
>> me
>> > > know
>> > > >> if I got things wrong or missed something crucial here.
>> > > >>
>> > > >> For issues 1-3, if the below reflects the state of the discussion,
>> I
>> > > would
>> > > >> try and update the FLIP in the next days.
>> > > >> For the remaining ones we need more discussion.
>> > > >>
>> > > >> I would suggest to fork each of these aspects into a separate mail
>> > > thread,
>> > > >> or will loose sight of the individual aspects.
>> > > >>
>> > > >> *(1) Separation of Split Enumerator and Split Reader*
>> > > >>
>> > > >>  - All seem to agree this is a good thing
>> > > >>  - Split Enumerator could in the end live on JobManager (and assign
>> > > splits
>> > > >> via RPC) or in a task (and assign splits via data streams)
>> > > >>  - this discussion is orthogonal and should come later, when the
>> > > interface
>> > > >> is agreed upon.
>> > > >>
>> > > >> *(2) Split Readers for one or more splits*
>> > > >>
>> > > >>  - Discussion seems to agree that we need to support one reader
>> that
>> > > >> possibly handles multiple splits concurrently.
>> > > >>  - The requirement comes from sources where one poll()-style call
>> > > fetches
>> > > >> data from different splits / partitions
>> > > >>--> example sources that require that would be for example
>> Kafka,
>> > > >> Pravega, Pulsar
>> > > >>
>> > > >>  - Could have one split reader per source, or multiple split
>> readers
>> > > that
>> > > >> share the "poll()" function
>> > > >>  - To not make it too complicated, we can start with thinking about
>> > one
>> > > >> split reader for all splits initially and see if that covers all
>> > > >> requirements
>> > > >>
>> > > >> *(3) Threading model of the Split Reader*
>> > > >>
>> > > >>  - Most active part of the discussion ;-)
>> > > >>
>> > > >>  - A non-blocking way for Flink's task code to interact with the
>

[jira] [Created] (FLINK-13422) git doc file's link display correct

[richt richt (JIRA)]

richt richt created FLINK-13422:
---

 Summary: git doc file's link  display correct
 Key: FLINK-13422
 URL: https://issues.apache.org/jira/browse/FLINK-13422
 Project: Flink
  Issue Type: Wish
  Components: Documentation
Reporter: richt richt
 Attachments: image-2019-07-25-16-22-40-208.png

eg.

[https://github.com/apache/flink/blob/master/docs/dev/table/hive_integration.md]

 

!image-2019-07-25-16-22-40-208.png!

it maybe a hyperlink or some picture . but i cannot read id now 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13423) Unable to find aggregation function in hive 1

[Jeff Zhang (JIRA)]

Jeff Zhang created FLINK-13423:
--

 Summary: Unable to find aggregation function in hive 1
 Key: FLINK-13423
 URL: https://issues.apache.org/jira/browse/FLINK-13423
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / Hive
Affects Versions: 1.9.0
Reporter: Jeff Zhang


I hit the following error when I try to use count in sql on hive1
{code:java}
btenv.sqlQuery("select count(1) from date_dim").toDataSet[Row].print(){code}
{code:java}
org.apache.flink.table.api.ValidationException: SQL validation failed. Failed 
to get function tpcds_text_2.COUNT
at 
org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:127)
at 
org.apache.flink.table.api.internal.TableEnvImpl.sqlQuery(TableEnvImpl.scala:427)
... 30 elided
Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: Failed 
to get function tpcds_text_2.COUNT
at 
org.apache.flink.table.catalog.hive.HiveCatalog.getFunction(HiveCatalog.java:1033)
at 
org.apache.flink.table.catalog.FunctionCatalog.lookupFunction(FunctionCatalog.java:167)
at 
org.apache.flink.table.catalog.FunctionCatalogOperatorTable.lookupOperatorOverloads(FunctionCatalogOperatorTable.java:74)
at 
org.apache.calcite.sql.util.ChainedSqlOperatorTable.lookupOperatorOverloads(ChainedSqlOperatorTable.java:73)
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1183)
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1198)
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.performUnconditionalRewrites(SqlValidatorImpl.java:1168)
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:925)
at 
org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:639)
at 
org.apache.flink.table.calcite.FlinkPlannerImpl.validate(FlinkPlannerImpl.scala:123)
... 31 more{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13424) HiveCatalog should add hive version in conf

[Rui Li (JIRA)]

Rui Li created FLINK-13424:
--

 Summary: HiveCatalog should add hive version in conf
 Key: FLINK-13424
 URL: https://issues.apache.org/jira/browse/FLINK-13424
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Reporter: Rui Li
 Fix For: 1.9.0


{{HiveTableSource}} and {{HiveTableSink}} retrieve hive version from conf. 
Therefore {{HiveCatalog}} has to add it.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13425) Table api sqlUpdate execute case when expression occur data error

[pingle wang (JIRA)]

pingle wang created FLINK-13425:
---

 Summary: Table api sqlUpdate execute case when expression occur 
data error
 Key: FLINK-13425
 URL: https://issues.apache.org/jira/browse/FLINK-13425
 Project: Flink
  Issue Type: Improvement
  Components: Table SQL / API, Table SQL / Planner
Affects Versions: 1.7.2, 1.6.3
Reporter: pingle wang
 Attachments: image-2019-07-25-16-50-02-632.png, 
image-2019-07-25-17-01-26-076.png

The flink job running in our online environment, from the beginning of the 1st 
to the 23rd, found that the data written to mysql on the 17th has abnormal 
changes, such as the case when the converted constant value has extra space to 
write.

flink execute sql :

 
{code:java}
insert into
date_pos_area_version_source
select
Date_Format(receive_time, _UTF-16LE'MMdd') AS `date`,
(
case
when country_id is null
or country_id = 10184 then 'mainland'
when country_id = 10239 then 'hongkong'
when country_id = 10248 then 'taiwan'
when country_id = 10257 then 'macau'
else 'overseas'
end
) as big_area,
sum(ad_action) as click_count
from
date_pos_area_version_source_view
where
ad_position_id is not null
group by Date_Format(receive_time, _UTF-16LE'MMdd'), big_area{code}
flink graph:
!image-2019-07-25-17-01-26-076.png!

mysql result like :
{code:java}
9ab1c5afa4946ca0040271736f38c83a hongkong 20190717
0acfa9f5133f5b558e4642ce0870ea77 macau20190717
cc571067754687a72ee0e8d224c6115a mainland 20190717
adb9f8b618195e195c90b09815a94842 overseas 20190717
aff685603b0f02debc8329a1dc7905d0 taiwan   20190717
9ab1c5afa4946ca0040271736f38c83a hongkong 20190630
9690a92f29519fbfef104011784221e7 macau 20190630
cc571067754687a72ee0e8d224c6115a mainland 20190630
adb9f8b618195e195c90b09815a94842 overseas 20190630
31779ba135934ed036644deb47eb1e54 taiwan 20190630
{code}
  !image-2019-07-25-16-50-02-632.png!

 

 

 

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13426) TaskExecutor uses the wrong Registrationid in the heartbeat with RM.

[Guowei Ma (JIRA)]

Guowei Ma created FLINK-13426:
-

 Summary: TaskExecutor uses the wrong Registrationid in the 
heartbeat with RM.
 Key: FLINK-13426
 URL: https://issues.apache.org/jira/browse/FLINK-13426
 Project: Flink
  Issue Type: Bug
  Components: Runtime / Coordination
Affects Versions: 1.8.1, 1.9.0
Reporter: Guowei Ma


1. First-time TaskExecutor register to rm successfully. If it fails to send 
SlotReport to SlotMaanger, TaskExecutor will reconnect to RM again. However, 
TaskExecutor still uses the old registration id in the 
EstablishedResourceManagerConnection.

2. Second-time TaskExecutor registers to rm successfully and gets a new 
registration id.

3. First-round and second-round has a race condition. Since that the task 
executor maybe use the old registration id in heartbeat with rm.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Probing of simple repartition hash join

[Benjamin Burkhardt]

Hi,

while doing a join, a MutableHashTable is created and filled while building. 
After building it is closed and the probing can begin.
I would like to start probing while building the hash table still runs. 
(ignoring the fact that this would lead to join misses...)

Anyone having an idea how one could do that?

Am 24. Juli 2019, 04:14 +0200 schrieb Caizhi Weng :
> Hi Benjamin,
>
> As you mentioned hash join I assume that you are referring to
> `HashJoinOperator` in blink planner.
>
> The input is selected by `nextSelection` method. As you can see, it will
> first read all records in the build side then read all records in the probe
> side. So the probing will only start after the build side has been fully
> received.
>
> Apart from this specific question, I'm interested in how you're going to
> implement a hash join where any of the two sides can be read. Could you
> share your ideas or give some hints about this? Thanks a lot.
>
> Benjamin Burkhardt  于2019年7月24日周三 上午1:24写道：
>
> > Hi all,
> >
> > Let’s imagine a simple repartition hash Join oft two tables.
> >
> > As soon as the first table is hashed completely (all EndOfPartition Events
> > sent) the shipping and probing of the second table starts.
> >
> > What I can’t find:
> >
> > 1. What triggers to start the probing exactly?
> > 2. Where can I find it in the code?
> >
> >
> > My final goal is to change the 2-phase join mechanism to a mixed
> > implementation where probing for finished subpartitions begins earlier.
> >
> > I appreciate any help.
> >
> > Thanks.
> >
> > Benjamin
> >

[jira] [Created] (FLINK-13427) HiveCatalog's createFunction not work when function name is upper

[Jingsong Lee (JIRA)]

Jingsong Lee created FLINK-13427:


 Summary: HiveCatalog's createFunction not work when function name 
is upper
 Key: FLINK-13427
 URL: https://issues.apache.org/jira/browse/FLINK-13427
 Project: Flink
  Issue Type: Bug
  Components: Connectors / Hive
Reporter: Jingsong Lee
 Fix For: 1.9.0, 1.10.0


 
{code:java}
hiveCatalog.createFunction(
  new ObjectPath(HiveCatalog.DEFAULT_DB, "myUdf"),
  new CatalogFunctionImpl(TestSimpleUDF.class.getCanonicalName(), new 
HashMap<>()),
  false);

hiveCatalog.getFunction(new ObjectPath(HiveCatalog.DEFAULT_DB, "myUdf"));
{code}
There is an exception now:

 

 
{code:java}
org.apache.flink.table.catalog.exceptions.FunctionNotExistException: Function 
default.myUdf does not exist in Catalog test-catalog.

at 
org.apache.flink.table.catalog.hive.HiveCatalog.getFunction(HiveCatalog.java:1030)
at 
org.apache.flink.table.catalog.hive.HiveCatalogITCase.testGenericTable(HiveCatalogITCase.java:146)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at 
org.apache.flink.batch.connectors.hive.FlinkStandaloneHiveRunner.runTestMethod(FlinkStandaloneHiveRunner.java:170)
at 
org.apache.flink.batch.connectors.hive.FlinkStandaloneHiveRunner.runChild(FlinkStandaloneHiveRunner.java:155)
at 
org.apache.flink.batch.connectors.hive.FlinkStandaloneHiveRunner.runChild(FlinkStandaloneHiveRunner.java:93)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
at 
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
at 
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
at 
com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: NoSuchObjectException(message:Function default.myUdf does not exist)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_function_result$get_function_resultStandardScheme.read(ThriftHiveMetastore.java)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_function_result$get_function_resultStandardScheme.read(ThriftHiveMetastore.java)
at 
org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_function_result.read(ThriftHiveMetastore.java)
{code}
Seems there are some bugs in HiveCatalog when use upper.

 

Maybe we should normalizeName in createFunction...



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13428) StreamingFileSink allow part file name to be configurable

[Joao Boto (JIRA)]

Joao Boto created FLINK-13428:
-

 Summary: StreamingFileSink allow part file name to be configurable
 Key: FLINK-13428
 URL: https://issues.apache.org/jira/browse/FLINK-13428
 Project: Flink
  Issue Type: Improvement
  Components: Connectors / FileSystem
Reporter: Joao Boto


Allow that part file name could be configurable:
 * partPrefix can be passed
 * allow to add extension if writer define one

 

the part prefix allow to set a better name to file

the extension allow system like Athena or Presto to automatic detect the type 
of file and the compression if applied



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (FLINK-13429) SQL Client end-to-end test fails on Travis

[Timo Walther (JIRA)]

Timo Walther created FLINK-13429:


 Summary: SQL Client end-to-end test fails on Travis
 Key: FLINK-13429
 URL: https://issues.apache.org/jira/browse/FLINK-13429
 Project: Flink
  Issue Type: Bug
  Components: Table SQL / Client
Reporter: Timo Walther
Assignee: Timo Walther


The SQL Client test does not work on the current master and hangs when 
executing CEP SQL. We reproduced this on two machines.

At commit 475c30cd4064a7bc2e32c963b6ca58e7623251c6 it was working.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [DISCUSS] Setup a bui...@flink.apache.org mailing list for travis builds

[Robert Metzger]

The mailing list has been created, you can now subscribe to it.

On Wed, Jul 24, 2019 at 1:43 PM Jark Wu  wrote:

> Thanks Robert for helping out that.
>
> Best,
> Jark
>
> On Wed, 24 Jul 2019 at 19:16, Robert Metzger  wrote:
>
> > I've requested the creation of the list, and made Jark, Chesnay and me
> > moderators of it.
> >
> > On Wed, Jul 24, 2019 at 1:12 PM Robert Metzger 
> > wrote:
> >
> > > @Jark: Yes, I will request the creation of a mailing list!
> > >
> > > On Tue, Jul 23, 2019 at 4:48 PM Hugo Louro  wrote:
> > >
> > >> +1
> > >>
> > >> > On Jul 23, 2019, at 6:15 AM, Till Rohrmann 
> > >> wrote:
> > >> >
> > >> > Good idea Jark. +1 for the proposal.
> > >> >
> > >> > Cheers,
> > >> > Till
> > >> >
> > >> >> On Tue, Jul 23, 2019 at 1:59 PM Hequn Cheng 
> > >> wrote:
> > >> >>
> > >> >> Hi Jark,
> > >> >>
> > >> >> Good idea. +1!
> > >> >>
> > >> >>> On Tue, Jul 23, 2019 at 6:23 PM Jark Wu  wrote:
> > >> >>>
> > >> >>> Thank you all for your positive feedback.
> > >> >>>
> > >> >>> We have three binding +1s, so I think, we can proceed with this.
> > >> >>>
> > >> >>> Hi @Robert Metzger  , could you create a
> > >> request to
> > >> >>> INFRA for the mailing list?
> > >> >>> I'm not sure if this needs a PMC permission.
> > >> >>>
> > >> >>> Thanks,
> > >> >>> Jark
> > >> >>>
> > >> >>> On Tue, 23 Jul 2019 at 16:42, jincheng sun <
> > sunjincheng...@gmail.com>
> > >> >>> wrote:
> > >> >>>
> > >>  +1
> > >> 
> > >>  Robert Metzger  于2019年7月23日周二 下午4:01写道：
> > >> 
> > >> > +1
> > >> >
> > >> > On Mon, Jul 22, 2019 at 10:27 AM Biao Liu 
> > >> >> wrote:
> > >> >
> > >> >> +1, make sense to me.
> > >> >> Mailing list seems to be a more "community" way.
> > >> >>
> > >> >> Timo Walther  于2019年7月22日周一 下午4:06写道：
> > >> >>
> > >> >>> +1 sounds good to inform people about instabilities or other
> > >> >> issues
> > >> >>>
> > >> >>> Regards,
> > >> >>> Timo
> > >> >>>
> > >> >>>
> > >>  Am 22.07.19 um 09:58 schrieb Haibo Sun:
> > >>  +1. Sounds good.Letting the PR creators know the build
> results
> > >> >> of
> > >>  the
> > >> >>> master branch can help to determine more quickly whether
> Travis
> > >> > failures
> > >> >> of
> > >> >>> their PR are an exact failure or because of the instability of
> > >> >> test
> > >> > case.
> > >> >>> By the way, if the PR creator can abort their own Travis
> build,
> > I
> > >>  think
> > >> >> it
> > >> >>> can improve the efficient use of Travis resources (of course,
> > >> >> this
> > >> >>> discussion does not deal with this issue).
> > >> 
> > >> 
> > >>  Best,
> > >>  Haibo
> > >>  At 2019-07-22 12:36:31, "Yun Tang"  wrote:
> > >> > +1 sounds good, more people could be encouraged to involve
> in
> > >>  paying
> > >> >>> attention to failure commit.
> > >> >
> > >> > Best
> > >> > Yun Tang
> > >> > 
> > >> > From: Becket Qin 
> > >> > Sent: Monday, July 22, 2019 9:44
> > >> > To: dev 
> > >> > Subject: Re: [DISCUSS] Setup a bui...@flink.apache.org
> > >> >> mailing
> > >>  list
> > >> >>> for travis builds
> > >> >
> > >> > +1. Sounds a good idea to me.
> > >> >
> > >> > On Sat, Jul 20, 2019 at 7:07 PM Dian Fu <
> > >> >> dian0511...@gmail.com>
> > >> >> wrote:
> > >> >
> > >> >> Thanks Jark for the proposal, sounds reasonable for me. +1.
> > >> >>> This
> > >>  ML
> > >> >>> could
> > >> >> be used for all the build notifications including master
> and
> > >> >>> CRON
> > >> >> jobs.
> > >> >>
> > >> >>> 在 2019年7月20日，下午2:55，Xu Forward 
> 写道：
> > >> >>>
> > >> >>> +1 ,Thanks jark for the proposal.
> > >> >>> Best
> > >> >>> Forward
> > >> >>>
> > >> >>> Jark Wu  于2019年7月20日周六 下午12:14写道：
> > >> >>>
> > >>  Hi all,
> > >> 
> > >>  As far as I know, currently, email notifications of
> Travis
> > >>  builds
> > >> >> for
> > >>  master branch are sent to the commit author when a build
> > >> >> was
> > >>  just
> > >> >> broken or
> > >>  still is broken. And there is no email notifications for
> > >> >> CRON
> > >> >> builds.
> > >> 
> > >>  Recently, we are suffering from compile errors for
> > >> >> scala-2.12
> > >>  and
> > >> >>> java-9
> > >>  which are only ran in CRON jobs. So I'm figuring out a
> way
> > >> >> to
> > >>  get
> > >>  notifications of CRON builds (or all builds) to quick fix
> > >>  compile
> > >> >>> errors
> > >>  and failed cron tests.
> > >> 
> > >>  After reaching out to @Chesnay Schepler <
> > >> >> ches...@apache.org>

[jira] [Created] (FLINK-13430) Configure sending travis build notifications to bui...@flink.apache.org

[Jark Wu (JIRA)]

Jark Wu created FLINK-13430:
---

 Summary: Configure sending travis build notifications to 
bui...@flink.apache.org
 Key: FLINK-13430
 URL: https://issues.apache.org/jira/browse/FLINK-13430
 Project: Flink
  Issue Type: Task
  Components: Build System
Reporter: Jark Wu
Assignee: Jark Wu


As discussed in the mailing list[1], the community want to send travis build 
notifications to bui...@flink.apache.org mailing list. 



[1]: 
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Setup-a-builds-flink-apache-org-mailing-list-for-travis-builds-tt30778.html



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Probing of simple repartition hash join

[Stephan Ewen]

Hi!

The join implementations used for the DataSet API and for the Blink Planner
are quite intricate. They make use of these custom memory segments, to
operate as much as possible on bytes, to control JVM memory utilization and
to save serialization costs.
That makes the implementation super complicated.

Do your experiments need that kind of behavior?
If not, it might actually be simpler to just start with some very simple
object-based hash table.

If you want to actually work with these classes, I would suggest to first
change the MutableHashTable a bit. It is probably easier to remove the
logic to pull from iterators into build and probe side, but rather do this
outside the class.
(I am assuming you plan to work with a fork for research here).

Best,
Stephan




On Thu, Jul 25, 2019 at 11:58 AM Benjamin Burkhardt <
pri...@benjaminburkhardt.de> wrote:

> Hi,
>
> while doing a join, a MutableHashTable is created and filled while
> building. After building it is closed and the probing can begin.
> I would like to start probing while building the hash table still runs.
> (ignoring the fact that this would lead to join misses...)
>
> Anyone having an idea how one could do that?
>
> Am 24. Juli 2019, 04:14 +0200 schrieb Caizhi Weng :
> > Hi Benjamin,
> >
> > As you mentioned hash join I assume that you are referring to
> > `HashJoinOperator` in blink planner.
> >
> > The input is selected by `nextSelection` method. As you can see, it will
> > first read all records in the build side then read all records in the
> probe
> > side. So the probing will only start after the build side has been fully
> > received.
> >
> > Apart from this specific question, I'm interested in how you're going to
> > implement a hash join where any of the two sides can be read. Could you
> > share your ideas or give some hints about this? Thanks a lot.
> >
> > Benjamin Burkhardt  于2019年7月24日周三 上午1:24写道：
> >
> > > Hi all,
> > >
> > > Let’s imagine a simple repartition hash Join oft two tables.
> > >
> > > As soon as the first table is hashed completely (all EndOfPartition
> Events
> > > sent) the shipping and probing of the second table starts.
> > >
> > > What I can’t find:
> > >
> > > 1. What triggers to start the probing exactly?
> > > 2. Where can I find it in the code?
> > >
> > >
> > > My final goal is to change the 2-phase join mechanism to a mixed
> > > implementation where probing for finished subpartitions begins earlier.
> > >
> > > I appreciate any help.
> > >
> > > Thanks.
> > >
> > > Benjamin
> > >
>

Re: [DISCUSS] FLIP-27: Refactor Source Interface

[Stephan Ewen]

Hi Biao!

Thanks for reviving this. I would like to join this discussion, but am
quite occupied with the 1.9 release, so can we maybe pause this discussion
for a week or so?

In the meantime I can share some suggestion based on prior experiments:

How to do watermarks / timestamp extractors in a simpler and more flexible
way. I think that part is quite promising should be part of the new source
interface.
https://github.com/StephanEwen/flink/tree/source_interface/flink-core/src/main/java/org/apache/flink/api/common/eventtime

https://github.com/StephanEwen/flink/blob/source_interface/flink-core/src/main/java/org/apache/flink/api/common/src/SourceOutput.java



Some experiments on how to build the source reader and its library for
common threading/split patterns:
https://github.com/StephanEwen/flink/tree/source_interface/flink-core/src/main/java/org/apache/flink/api/common/src


Best,
Stephan


On Thu, Jul 25, 2019 at 10:03 AM Biao Liu  wrote:

> Hi devs,
>
> Since 1.9 is nearly released, I think we could get back to FLIP-27. I
> believe it should be included in 1.10.
>
> There are so many things mentioned in document of FLIP-27. [1] I think
> we'd better discuss them separately. However the wiki is not a good place
> to discuss. I wrote google doc about SplitReader API which misses some
> details in the document. [2]
>
> 1.
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-27:+Refactor+Source+Interface
> 2.
> https://docs.google.com/document/d/1R1s_89T4S3CZwq7Tf31DciaMCqZwrLHGZFqPASu66oE/edit?usp=sharing
>
> CC Stephan, Aljoscha, Piotrek, Becket
>
>
> On Thu, Mar 28, 2019 at 4:38 PM Biao Liu  wrote:
>
>> Hi Steven,
>> Thank you for the feedback. Please take a look at the document FLIP-27
>> 
>>  which
>> is updated recently. A lot of details of enumerator were added in this
>> document. I think it would help.
>>
>> Steven Wu  于2019年3月28日周四 下午12:52写道：
>>
>>> This proposal mentioned that SplitEnumerator might run on the JobManager
>>> or
>>> in a single task on a TaskManager.
>>>
>>> if enumerator is a single task on a taskmanager, then the job DAG can
>>> never
>>> been embarrassingly parallel anymore. That will nullify the leverage of
>>> fine-grained recovery for embarrassingly parallel jobs.
>>>
>>> It's not clear to me what's the implication of running enumerator on the
>>> jobmanager. So I will leave that out for now.
>>>
>>> On Mon, Jan 28, 2019 at 3:05 AM Biao Liu  wrote:
>>>
>>> > Hi Stephan & Piotrek,
>>> >
>>> > Thank you for feedback.
>>> >
>>> > It seems that there are a lot of things to do in community. I am just
>>> > afraid that this discussion may be forgotten since there so many
>>> proposals
>>> > recently.
>>> > Anyway, wish to see the split topics soon :)
>>> >
>>> > Piotr Nowojski  于2019年1月24日周四 下午8:21写道：
>>> >
>>> > > Hi Biao!
>>> > >
>>> > > This discussion was stalled because of preparations for the open
>>> sourcing
>>> > > & merging Blink. I think before creating the tickets we should split
>>> this
>>> > > discussion into topics/areas outlined by Stephan and create Flips for
>>> > that.
>>> > >
>>> > > I think there is no chance for this to be completed in couple of
>>> > remaining
>>> > > weeks/1 month before 1.8 feature freeze, however it would be good to
>>> aim
>>> > > with those changes for 1.9.
>>> > >
>>> > > Piotrek
>>> > >
>>> > > > On 20 Jan 2019, at 16:08, Biao Liu  wrote:
>>> > > >
>>> > > > Hi community,
>>> > > > The summary of Stephan makes a lot sense to me. It is much clearer
>>> > indeed
>>> > > > after splitting the complex topic into small ones.
>>> > > > I was wondering is there any detail plan for next step? If not, I
>>> would
>>> > > > like to push this thing forward by creating some JIRA issues.
>>> > > > Another question is that should version 1.8 include these features?
>>> > > >
>>> > > > Stephan Ewen  于2018年12月1日周六 上午4:20写道：
>>> > > >
>>> > > >> Thanks everyone for the lively discussion. Let me try to summarize
>>> > > where I
>>> > > >> see convergence in the discussion and open issues.
>>> > > >> I'll try to group this by design aspect of the source. Please let
>>> me
>>> > > know
>>> > > >> if I got things wrong or missed something crucial here.
>>> > > >>
>>> > > >> For issues 1-3, if the below reflects the state of the
>>> discussion, I
>>> > > would
>>> > > >> try and update the FLIP in the next days.
>>> > > >> For the remaining ones we need more discussion.
>>> > > >>
>>> > > >> I would suggest to fork each of these aspects into a separate mail
>>> > > thread,
>>> > > >> or will loose sight of the individual aspects.
>>> > > >>
>>> > > >> *(1) Separation of Split Enumerator and Split Reader*
>>> > > >>
>>> > > >>  - All seem to agree this is a good thing
>>> > > >>  - Split Enumerator could in the end live on JobManager (and
>>> assign
>>> > > splits
>>> > > >> via RPC) or in a task (and assign splits via data streams)
>>> > > >>  - this discussion is

REST API / JarRunHandler: More flexibility for launching jobs

[Thomas Weise]

Hi,

While considering different options to launch Beam jobs through the Flink
REST API, I noticed that the implementation of JarRunHandler places quite a
few restrictions on how the entry point shall construct a Flink job, by
extracting and manipulating the job graph.

That's normally not a problem for Flink Java programs, but in the scenario
I'm looking at, the job graph would be constructed by a different process
and isn't available to the REST handler. Instead, I would like to be able
to just respond with the job ID of the already launched job.

For context, please see:

https://docs.google.com/document/d/1z3LNrRtr8kkiFHonZ5JJM_L4NWNBBNcqRc_yAf6G0VI/edit#heading=h.fh2f571kms4d

The current JarRunHandler code is here:

https://github.com/apache/flink/blob/f3c5dd960ff81a022ece2391ed3aee86080a352a/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/JarRunHandler.java#L82

It would be nice if there was an option to delegate the responsibility for
job submission to the user code / entry point. That would be useful for
Beam and other frameworks built on top of Flink that dynamically create a
job graph from a different representation.

Possible ways to get there:

* an interface that the main class can be implement end when present, the
jar run handler calls instead of main.

* an annotated method

Either way query parameters like savepoint path and parallelism would be
forwarded to the user code and the result would be the ID of the launched
job.

Thougths?

Thanks,
Thomas

Re: Fwd: Kafka Summit listing on ASF events site

[Austin Bennett]

Hi Melissa,

Is there a rate for Apache Members/Committers?

I recently saw Flink is offering that for their forthcoming EU Summit:
https://lists.apache.org/thread.html/%3CCAAdrtT2geJjp

Also, @sha...@apache.org  and @dev
 , that one (Flink EU conference) might be good
to get on the Apache Calendar?

Cheers,
Austin



On Wed, Jul 24, 2019 at 1:26 PM Sharan Foga  wrote:

> Hi Melissa
>
> I will add it.
>
> Thanks
> Sharan
>
> On 2019/06/20 20:39:05, Melissa Almgren  wrote:
> > Just circling back on this request, thank you!
> > Best,
> > Melissa
> >
> > -- Forwarded message -
> > From: Melissa Almgren 
> > Date: Thu, Jun 13, 2019 at 10:09 AM
> > Subject: Kafka Summit listing on ASF events site
> > To: 
> >
> >
> > Greetings!
> >
> > We would like to add Kafka Summit San Francisco to the events calendar
> here:
> > http://community.apache.org/calendars/index.html
> >
> > Here is the relevant info:
> >
> > https://kafka-summit.org/events/kafka-summit-san-francisco-2019/
> > *2019-09-30 to 2019-10-01*
> > Hilton San Francisco Union Square
> > 333 O’Farrell Street
> > San Francisco,
> > CA 94102
> > 415-771-1400
> >
> > Please let me know if you have any questions or need any further
> > information.
> > Best,
> > Melissa
> >
> > --
> > *Melissa Almgren*
> > Senior Manager, Events Marketing | Confluent
> > 415.378.7191
> > Follow us: Twitter  | blog
> >
> > 
> > [image: https://kafka-summit.org/events/kafka-summit-san-francisco-2019]
> > 
> >
> >
> >
> >
> > 
> >
> >
> >
> >
> > --
> > *Melissa Almgren*
> > Senior Manager, Events Marketing | Confluent
> > 415.378.7191
> > Follow us: Twitter  | blog
> >
> > 
> > [image: https://kafka-summit.org/events/kafka-summit-san-francisco-2019]
> > 
> >
> >
> >
> >
> > 
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@community.apache.org
> For additional commands, e-mail: dev-h...@community.apache.org
>
>

Fine Grained Recovery / FLIP-1

[Thomas Weise]

Hi,

We are using Flink for streaming and find the "stop-the-world" recovery
behavior of Flink prohibitive for use cases that prioritize availability.
Partial recovery as outlined in FLIP-1 would probably alleviate these
concerns.

https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures

Looking at the subtasks in https://issues.apache.org/jira/browse/FLINK-4256 it
appears that much of the work was already done but not much recent
progress? What is missing (for streaming)? How close is version 2 (recovery
from limited intermediate results)?

Thanks!
Thomas

Re: Fine Grained Recovery / FLIP-1

[Guowei Ma]

Hi,
1. Currently, much work in FLINK-4256 is about failover improvements in the
bouded dataset scenario.
2. For the streaming scenario,  a new shuffle plugin + proper failover
strategy could avoid the "stop-the-word" recovery.
3. We have already done many works about the new shuffle in the old Flink
shuffle architectures because many of our customers have the concern. We
have a plan to move the work to the new Flink pluggable shuffle
architecture.

Best,
Guowei


Thomas Weise  于2019年7月26日周五 上午8:54写道：

> Hi,
>
> We are using Flink for streaming and find the "stop-the-world" recovery
> behavior of Flink prohibitive for use cases that prioritize availability.
> Partial recovery as outlined in FLIP-1 would probably alleviate these
> concerns.
>
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures
>
> Looking at the subtasks in
> https://issues.apache.org/jira/browse/FLINK-4256 it
> appears that much of the work was already done but not much recent
> progress? What is missing (for streaming)? How close is version 2 (recovery
> from limited intermediate results)?
>
> Thanks!
> Thomas
>